Navigating RAG Optimization with an Evaluation-Driven Compass

[on-hold music] Hello, everyone. Welcome to our talk.

Uh, my name is Atita, and I work as a solution architect, uh, at Q- uh, Qdrant. And, um-

I am-

Together with me, I have, uh, Deanna.

I'm Deanna Emery, and I'm founding AI researcher at Quotient AI.

Cool. So we would be talking about navigating RAG optimization with an evaluation-driven compass. So I think the track is about RAG, so let's, uh, talk and extend what we already know and what we've already seen so far.

So in this talk today, we will be discussing some of the key essential topics for anyone interested in building or productionizing the most popular implementation of generative AI, that is RAG, Retrieval-Augmented Generation.

Uh, in simple terms, RAG combines the capabilities of, uh, searching and retrieving through the vast amount of information stored in knowledge source, um, usually a vector database. Um, and then we use this information to generate relevant and coherent responses, leveraging the capabilities of a large language model.

We will be talking through the known challenges of this approach, and, uh, how can you combat, uh, them by adopting an evaluation-based, uh, optimization techniques to get the desired results.

So without further ado, let's get started. So yeah, as, uh, simple as it's, uh, defined, RAG can be implemented in many different ways, as we can see on the slide.

Apologies for the busy slide, as I wanted to cover all the aspects of how, um, RAG can be implemented. And, uh, we will be starting with the simplest one, the naive RAG.

This, uh, involves three key steps. First, we split the documents, uh, using the specific chunking strategy. Next, we process the document embedding and store them into a vector database.

For each user query, we then retrieve the most relevant document chunks based on our retrieval strategy. Finally, we add these retrieved chunks to our prompt to generate an answer using a chosen LLM.

Advanced versions involve enhancement to user queries, such as, um, query expansion or rewriting, and, uh, post-retrieval treatments like, um, result re-ranking or fusion. The further advanced version of, uh, RAG includes, uh, query routing, um, to task-based, uh, agents and self-improvement modules like DSPy.

However, there's one tool that is common in all these implementations if you notice, no matter if you want to build a naive or agentic RAG, and that is a vector database.

One of the most common use cases for building RAG is for knowledge management. There are aspects like retrieval performance, scalability to support large data volume, and resource optimization are paramount.

Speaking of vector database, Qdrant is open source, uh, vector search database built on Rust and, uh, is purpose-built to support your generative AI applications built on large scale of data.

If you haven't already checked out, uh, please do check out.

So coming back to our topic, it is, uh, worth to understand and acknowledge, um, the challenges that come along with all the goodness of RAG. I'm repurposing our naive RAG, uh, architecture to highlight common possible issues on each level.

After all, the first step to solving any problem is to recognize there is one. To begin with, during the data processing stage, we could have, um, issues with, uh, information missing from our dataset or information that fails to get extracted from our source of information.

This would result in incorrect and incomplete responses. On data ingestion, there is a constant battle to determine the optimum chunking strategy, along with determining a suitable embedding model that basically understands the specificity and jargons, uh, used in your dataset.

Information retrieval itself is quite interesting and a very evol-- ever-evolving, uh, field. Having spent seventeen years in, uh, the space myself, I can probably say that, um, relevancy is an unsolved problem.

So the challenge with determining relevant documents, retrieval size or retrieval window, if you may call it, and the order of documents is unskippable. Response generation can face challenges such as incorrect and incomplete answers to all the previously mentioned issues, and the issues of straying from the provided context.

To add, our query is also vulnerable to ambiguous or vague, uh, questions. There are certainly other challenges like generating coherent responses, maintaining user conversations, scaling a RAG for hundred or thousands of concurrent users, along with data security and compliance issues.

So looks like RAG isn't really a piece of cake after all. Fortunately, as for, like, the challenges, we have plenty of improvement techniques as well for the RAG. Let's look at them next.

So we saw challenges zooming into data quality, data that missed to get generated. After all, the foundation of great responses is, uh, it lies in the richness and, uh, accuracy of its, uh, information or context in case of RAG, which can be controlled through adopting data cleaning and advanced data extraction methodologies.

It is not a bad idea to use a general purpose embedding model to begin with, but for added improvements, it would be a good idea to use an embedding model that comprehends the terminologies of your domain.

Metadata. It is a very versatile feature that can help you retrieve, um, um, uh, your, um, added, um, understanding of your documents and, uh, improving your retrieval. Plus, leveraging the metadata filtering during the retrieval, it can also help you out with filtering out the irrelevant documents.

So it is also probably a good idea to invest your, uh, time into determining and improving your chunking strategy, which we spoke about earlier. Sometimes just by reducing the chunk size or adding semantic chunking can work wonders.

We would be seeing that also when Deanna walks us through all the experimentation. So we have been aware of Lost in the middle problem, and this is where it may be a good idea to determine an apt context size needed for your RAG to generate a helpful response.

Also, using suitable indexing algorithms like HNSW, BM25, or even graphs, they can do wonders.

Talking about suitable context size also prioritizes the document re-ranking as one of the key improvement parameters to ensure the most relevant documents, uh, that are provided in the context for LLM to generate a helpful response.

With LLMs in the picture, we cannot, um, overlook the difference a good prompt or thinking about questions as a chain of thought can make to the progress of-- process of, uh, response generation.

Semantic understanding is clearly desired. We know that, uh, text search alone doesn't works. But if the dataset has the requirement for the exact matches, it may be worth exploring dense as well as sparse vectors, and you can do that on one or many fields.

Similar to the embedding model, it may be worth switching and experimenting with different LLMs as well to ensure that, uh, the desired response is generated. And lastly, for better handling of task-driven, uh, user queries, such as fetching specific information or using custom data, it is beneficial to address these, uh, challenges using agents, which are well-suited for the

job. So with so many different levers to tweak in RAG pipeline, it's hard to know what's going wrong, what to change, where to start. This is why evaluation is very important.

Without an evaluation-based guided flow, it is difficult to accurately measure progress and ensure optimal performance. Evaluation also helps you to iteratively refine applications, making informed decisions, and ultimately achieve goals more effectively.

So on that topic, I would like to welcome Deanna, who's gonna basically walk us through how we did this, uh, experimentation.

Uh, thanks, Atita, for the perfect setup. [laughs] [audience applauding] Um, so this is where Quotient comes in, and I'm going to steal that from you. Thank you. Um, so because the quality of RAG is so dependent on the underlying documents, a thorough evaluation of a RAG system has to be customized to suit that specific domain and dataset.

Um, so Quotient's evaluation solution fills this need by enabling developers to measure the effectiveness of their LLM products accurately. Quotient's platform accelerates the experimentation process. With an evaluation dataset that contains realistic examples of inputs and expected outputs for your AI solution, you can quickly experiment and iterate to optimize

your RAG solutions. And if you don't have an evaluation dataset, don't worry. Uh, Quotient can help you get started by generating one for you, and you can hear more from us on this at the AI, SIISL, and Waves meetup on Friday.

So how does this work in practice? Once you have your Qdrant vector database set up, you can populate your evaluation dataset by submitting queries to return the contexts for the LLM.

You can then submit your evaluation dataset to Quotient, which handles the full orchestration, including the prompt formatting, execution of LLMs, and the metric computations.

So to see it in action, we've put together a demo walkthrough where we'll show you a workflow for making evaluation-informed changes to optimize your RAG system using Quotient and Qdrant.

For the sake of time, we've executed the notebook ahead of time, and we'll be walking you through the code outputs. And if you scare-- scan the QR code here, you can find the notebook on GitHub.

In this demo, we are building a RAG solution for question answering on Qdrant's documentation, and this will help enable Qdrant users to get help quickly.

So before we can begin evaluation, it's important to take a step back and consider what we're optimizing for. Given this use case, it's generally important to get helpful answers to the questions, but it's perhaps more important that the answers do not contain any inaccurate information that could misguide users.

And so in other words, we want to minimize hallucinations. And with that in mind, we will be looking at the following metrics shown here with a focus on faithfulness.

Uh, the first two metrics are both focused on measuring the quality of the retrieval side of RAG. Context relevance tells us whether the necessary information to answer the question is in the retrieved documents.

Chunk relevance tells us how much of the information retrieved is actually useful for answering the question versus just noise. Faithfulness is our hallucination metric. And, and then because be-- the focus of this talk is going to be optimizing the retrieval side of RAG, we're sticking to some of the more general text quality metrics here.

So when we're first getting started, we want to consider a simple naive RAG implementation to help us better optimize the data processing and vector database setup. So we start off by choosing a reasonable embedding model and chunking parameters, retrieval window, and the Mistral instruct model.

And then to see if we require additional context to answer the questions, we set up a second experiment where we increase the chunk parameters.

And so here are the results of those first two experiments. You can see that by increasing the chunk size in experiment two, we had some minor improvements in our text quality metrics.

That said, we had a considerable drop in our faithfulness, which is the metric we're optimizing for. Of note, you can see that the context relevance increased, meaning that we retrieved more of the necessary information to answer the question.

But the chunk relevance dropped considerably, meaning that a smaller portion of the retrieved documents was actually relevant. And so what this implies is that if we simply retrieve more documents but use the smaller chunk size from before, we might get better results.

And so we try this out. As expected, the smaller chunk size, uh, with the larger w- retrieval window achieved the highest relevance scores and the best faithfulness score, uh, meaning that we have a lower occurrence of hallucinations.

In our next two iterations, we test out a new embedding model as well as a different LLM, switching to GPT 3.5. And while the embedding model experiment in the light gray didn't quite work out, for experiment five in the dark gray, we found that using the same RAG configuration but changing the LLM improved performance across all our

metrics. So here we're looking at the aggregated metrics for our top-performing RAG configuration, and the large variance in the context relevance, which is highlighted in red, implies that some of the questions are having a harder time retrieving the right documents.

So to better understand what's going on, we have to look into the data. And here we've shown the two worst-performing data points in terms of hallucination. This third column here shows us the retrieved documents, and it's a lot of text, so I'll just summarize.

Um, so it seems like we're returning a lot of unrelated documents that are likely over-indexing on specific words in the query like Qdrant, support, and search, and we're returning documents that repeat these terms many times.

So to address this issue, one possible solution could be expanding our retrieval window to capture more documents and thereby making it more likely we get the right information. But in doing so, we'd also be adding a lot of noise to our context.

So by now, we likely need to expand out of naive RAG and start to add in some more advanced techniques, and this is where re-ranking comes in. So we could try using a different embedding model to re-rank the retrieved documents and then return only the top few, thus weeding out some of the more unrelated ones.

So we try this out. We perform three different re-ranking experiments, trying re-rankers from Mixed Bread, Cohere, and Gina's Colbert model, and we plot them here against our top naive RAG approach in blue.

And you can see that, in general, our context relevances, and with it our faithfulness, have improved with this strategy, with Cohere achieving the best relevance and faithfulness scores.

If we return to those same two worst-performing data points from experiment five now and look at how our re-ranking implementation did, you can see that in the second example, we're now retrieving documents that contain the desired answer, and with it, our context relevance and faithfulness are close to one.

That said, in the first example, we're still getting poor results, and this suggests that even after expanding our retrieval windows, we're still unable to identify the relevant documents. So if we think about the Qdrant documentation and the text within it, it contains a lot of special terminology, jargon, acronyms, and so it's unsurprising that these generally trained embedding

models are going to be limited in performance. So we could try fine-tuning or training our own model to address this issue, but this could be time-consuming and costly. Um, so another option to try is hybrid search, which combines sparse and dense vectors, and these sparse vectors help us capture documents that share similar terminology.

So we tried two implementations of this, one where we incorporate hybrid search with a re-ranker and one without. And you can see that, uh, using the Cohere re-ranker with hybrid search gives us the best performance across all metrics except for chunk relevance.

So looking at this, uh, hybrid search re-ranking experiment now on these same two data points, you can see that the context relevance and faithfulness scores are both close to one now, a significant improvement over our prior ones.

And we're also better to retrieve, better able to retrieve the information necessary to answer these questions. And so this suggests that domain-specific terminology has a big effect on our overall performance.

So to summarize, the table here shows what gains in performance we were able to make over just ten experiments. Starting from a faithfulness score of .76, we worked our way up to a score of just under .85, and notably, all of these gains were made without changing from a generic question answering prompt.

So there's certainly many more experiments to run, and we have plenty of room for improvement. But you can see how starting from scratch you can improve your RAG system by making incremental changes, evaluating using a combination of metrics that together can help you identify underlying issues, then observing patterns in your data, forming a hypothesis, and repeating the

process.

Thank you, Deanna. So to summarize, um, this talk and the experimentation, in this talk, we covered several key aspects of improving RAG. The baseline takeaway is that, uh, there is no substitute for evaluation-based or data-driven improvements.

We emphasize leveraging domain understanding to achieve significant, uh, wins and, uh, outlined various techniques for enhancement in our experiments. To ensure continuous improvement, it is crucial to keep your evaluation dataset up to date.

Lastly, avoid over-engineering your RAG application without considering a combination of carefully chosen metrics. If this talk piqued your interest and you are interested to get in touch with us, before that, some key references and the QR code.

Please feel free to scan them, get in touch with us. We're going to be around and looking forward to all your questions. [audience applauding] [upbeat music]