# Navigating RAG Optimization with an Evaluation-Driven Compass

Atita Arora, Qdrant · Deanna Emery, Quotient AI

AI Engineer World's Fair 2024

[Talk page](https://ai.engineer/talks/DId2KP8Ykz4-navigating-rag-optimization-evaluation-driven) · [Video](https://www.youtube.com/watch?v=DId2KP8Ykz4)

## Navigating RAG Optimization with an Evaluation-Driven Compass

A documentation question-answering system improves through ten experiments that separate missing evidence from noisy context, then use reranking and hybrid search to address distinct failures.

Familiarity with embeddings, vector search, and language-model prompts will help you follow the experiments.

### Start with the simplest retrieval pipeline

[0:53](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=53s)

Split documents into chunks, embed those chunks, and store their vectors. When a user asks a question, retrieve the relevant chunks and add them to a prompt for a language model. This is the starting point for **retrieval-augmented generation**, or RAG: the model answers with information drawn from an external knowledge source. The implementation is straightforward to describe, but each step introduces a choice that can change the answer.

More advanced pipelines add query expansion or rewriting before retrieval, then reranking or fusion afterward. Agentic versions can route questions to task-specific agents and incorporate optimization systems such as [DSPy](https://dspy.ai/). These additions create more opportunities to improve results—and more components whose contribution needs to be measured.

![Three pipeline diagrams connect documents, a vector database, retrieval, and a language model, with query and result treatments in advanced RAG and tools in agentic RAG.](https://ai.engineer/api/talk-essay-media/d2fa09958b396927e346da3b29a2d44381d01ec938558b34a0759e432bb72456)

Three flavours of RAG: naive, advanced, and agentic.

The architectures Atita Arora presents share a vector database. For knowledge management, that layer must support relevant retrieval, large document collections, and efficient resource use. The demonstration uses [Qdrant](https://github.com/qdrant/qdrant), an open-source vector search database written in Rust.

### An incorrect answer can begin upstream

[3:10](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=190s)

Before changing the model, trace the evidence through the pipeline. Information might be absent from the source collection or lost during extraction. Either gap can produce an incomplete answer even when retrieval and generation behave as intended. During ingestion, chunk boundaries determine which facts remain together, while the embedding model determines how well the system represents the terminology in those chunks.

Retrieval adds three closely related decisions: which documents are relevant, how many to retrieve, and in what order to present them. Generation can inherit the resulting omissions or distractions, then introduce another failure by straying from the supplied context. A fluent answer therefore does not tell you which stage succeeded.

The query itself can also be vague or ambiguous. Beyond an individual answer, an application must maintain coherent conversations, handle hundreds or thousands of concurrent users, and meet security and compliance requirements. These are separate concerns from whether one retrieved passage contains the right fact, but they remain part of productionizing RAG.

![RAG pipeline surrounded by challenge boxes for data processing, ingestion, retrieval, response generation, query processing, and production, with sad-face markers beside all six.](https://ai.engineer/api/talk-essay-media/aeaa2e586a8357ecbe8e2c0c773fd8b858f518bb99764a8a0680607a19aca16e)

Challenges across the RAG pipeline, including production concerns.

### Improve what enters the context

[4:58](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=298s)

The first interventions follow the path of the data.

- **Extraction and cleaning:** Recover information that was missed and remove defects before indexing it.
- **Embedding choice:** Start with a general-purpose model, then test whether a model that understands the domain's terminology improves retrieval.
- **Metadata:** Attach useful document attributes and filter on them to exclude irrelevant material.
- **Chunking:** Test smaller chunks or semantic boundaries instead of assuming the initial split is appropriate.

These changes affect what evidence is available and how precisely it can be retrieved.

Context size requires its own experiment. [Lost in the Middle](https://arxiv.org/abs/2307.03172) motivates examining where useful evidence appears within a long context; simply supplying more text need not make that evidence easier for the model to use. Arora also points to HNSW, BM25, and graph-based retrieval approaches as options to explore. Once candidates have been retrieved, **reranking** can prioritize the documents that deserve space in the final context.

### Choose changes with evidence

[6:34](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=394s)

Generation offers additional levers, including prompt design and chain-of-thought prompting. Retrieval can combine dense vectors for semantic matching with sparse vectors when exact terms matter, over one field or several. Different language models may produce different answers from the same context, while agents can handle task-specific information fetching or access to custom data.

**A menu of techniques is not a sequence of required upgrades.** With so many possible changes, the practical problem is deciding which failure to address next. Evaluation supplies the feedback: establish how the system behaves, make a targeted change, and measure whether it moves the application toward its goal.

### Build an evaluation around the documents

[7:55](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=475s)

Because a RAG system depends on its underlying documents, Deanna Emery starts with evaluation data tailored to the application and domain. The dataset should contain realistic inputs and expected outputs. Quotient supports running these experiments and can help generate an initial evaluation dataset when one is not already available.

The evaluation workflow connects retrieval to answer generation:

1. Submit the evaluation questions to the populated Qdrant database.
2. Attach the returned contexts to the evaluation examples.
3. Submit those examples to Quotient, which orchestrates prompt formatting, language-model execution, and metric computation.

This makes the retrieved context part of the experiment rather than treating the final answer as the only observable result.

The [companion walkthrough](https://github.com/qdrant/qdrant-rag-eval) uses notebook outputs executed before the presentation. Its application is question answering over Qdrant's documentation, intended to help users find answers quickly. That concrete use case determines what the experiments should optimize.

### Distinguish missing evidence from noisy evidence

[9:59](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=599s)

A documentation assistant should be helpful, but inaccurate instructions can actively misguide a user. Emery therefore makes **faithfulness** the principal optimization metric, with the aim of minimizing hallucinations.

The metrics answer different questions in this demonstration:

| Metric | Diagnostic question |
|---|---|
| Context relevance | Does the retrieved context contain the information needed to answer? |
| Chunk relevance | How much retrieved information is useful rather than noise? |
| Faithfulness | Does the answer remain faithful to the supplied context? |
| General text quality | How does the generated text perform on the accompanying quality measures? |

The distinction between the first two is crucial. A context can contain the necessary answer while surrounding it with a large amount of irrelevant material. The experiment needs to detect both conditions. These are the speakers' working metric definitions; no numerical formula is needed to follow the subsequent diagnosis.

### Separate chunk size from retrieval window

[11:01](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=661s)

The initial configuration is deliberately simple: an embedding model, a chunking strategy, a retrieval window, and **Mistral Instruct** for generation. The first question is whether the model needs more context. Experiment two increases the chunking parameters to supply larger chunks.

Emery reports that larger chunks slightly improve text quality and increase context relevance, but substantially reduce faithfulness and chunk relevance. More of the necessary information is now present, yet a smaller portion of the retrieved material is useful. The additional context has not produced a more faithful answer.

That combination suggests a narrower intervention: restore the smaller chunks, but retrieve more of them. This changes the retrieval window without retaining the larger chunk size. Among the first three experiments, smaller chunks with a larger retrieval window achieve the best relevance and faithfulness scores. The useful lesson is the distinction between **larger chunks** and **more chunks**: both add context, but they change its composition differently.

### Change models, then inspect individual failures

[12:31](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=751s)

The next two experiments test a different embedding model and a different generation model. The alternative embedding model does not improve the result in this experiment. In experiment five, switching to **GPT-3.5** while retaining the same RAG configuration improves all reported metrics. This establishes a stronger naive baseline before adding another retrieval stage.

The aggregate scores still conceal an uneven experience. A large variance in context relevance indicates that some questions retrieve useful evidence much less reliably than others. Emery opens the two examples with the worst hallucination performance and inspects their retrieved documents.

Those documents appear to overemphasize recurring words such as `Qdrant`, `support`, and `search`. They repeat terms from the questions but contain unrelated information. This is a diagnosis drawn from the retrieved text: matching familiar vocabulary is not sufficient to recover the passage that answers a particular question.

### Retrieve broadly, then rerank

[13:39](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=819s)

Expanding the retrieval window could improve the chance of finding the missing evidence, but passing every additional document to the generator would also add noise. Reranking separates those decisions: retrieve a broader candidate set, score those candidates again, and send only the top few onward. The candidate window can grow without forcing the final context to grow with it.

The selection step can be expressed directly in Python. Here, `scores` contains one relevance score per retrieved candidate, with larger scores indicating greater relevance; scoring happens before this function is called.

```python
from collections.abc import Sequence


def select_context(
    candidates: Sequence[str],
    scores: Sequence[float],
    final_count: int,
) -> list[str]:
    if len(candidates) != len(scores):
        raise ValueError("Each candidate needs one score")
    if final_count < 1:
        raise ValueError("final_count must be positive")

    ranked = sorted(
        zip(candidates, scores),
        key=lambda item: item[1],
        reverse=True,
    )
    return [text for text, _ in ranked[:final_count]]
```

The separate `final_count` makes the boundary explicit: retrieval decides which documents are eligible, while reranking decides which eligible documents reach generation.

The next three experiments compare rerankers from **mixedbread**, **Cohere**, and **Jina ColBERT** against the best naive configuration. Emery reports that reranking generally improves context relevance and faithfulness, with Cohere achieving the best scores on those two metrics in this comparison. The chart also displays chunk relevance and the accompanying text-quality measures: ROUGE-L, BERTScore, and BERT sentence similarity.

![Grouped bar chart compares a baseline with Mixedbread, Cohere, and Jina-ColBERT across chunk relevance, context relevance, faithfulness, ROUGE-L, BERTScore, and BERT sentence similarity.](https://ai.engineer/api/talk-essay-media/f1daca5e8bd446cbba80af87a9cb0453c53459498640c84f65dd1584ea0340b2)

Reranker comparison across six evaluation metrics.

Returning to the same two difficult examples reveals where that improvement occurred. For the second previously failing example, reranking retrieves documents containing the answer, and context relevance and faithfulness are close to one. Looking at the documents alongside the scores connects the metric movement to a concrete change in available evidence.

### Recover evidence that reranking cannot find

[14:53](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=893s)

The first difficult example still performs poorly. Even after expanding the candidate window and reranking it, the system has not identified the relevant documents. **Reranking cannot promote evidence that retrieval never supplies.** Qdrant's documentation contains specialized terminology, jargon, and acronyms, making the limitations of a general-purpose embedding model a plausible next explanation.

Fine-tuning or training an embedding model could address that mismatch, but requires time and money. The next experiment instead changes how candidates are found: **hybrid search** combines dense vectors with sparse vectors that help capture documents sharing the query's terminology. Dense retrieval supplies semantic matches; sparse retrieval adds a route to evidence whose exact technical vocabulary matters.

Emery tests hybrid retrieval both with and without reranking. Hybrid search with Cohere reranking achieves the best reported performance across all metrics except chunk relevance. That exception preserves an important distinction: the configuration that performs best overall does not necessarily produce the highest proportion of useful retrieved material.

On both inspected difficult examples, hybrid search with reranking brings context relevance and faithfulness close to one. These are results for those two examples, not the dataset aggregate. The retrieved material now better contains the information needed to answer, supporting the diagnosis that domain-specific terminology was a significant retrieval obstacle.

### Keep the experiment loop tied to the application

[16:18](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=978s)

Across ten experiments on Qdrant documentation question answering, Emery reports that faithfulness rises from 0.76 to just under 0.85, while the generic question-answering prompt remains unchanged. Retrieval settings, retrieval methods, and the generation model do change. The scores describe this demonstration's evaluation, rather than answer-accuracy percentages.

The progression is an experimental method: make incremental changes, evaluate a combination of metrics, inspect patterns in the underlying examples, form a hypothesis, and repeat. Aggregate scores identify a direction; individual retrieved documents help explain why the system is failing and what to test next.

Continued improvement depends on keeping that evidence relevant. Domain understanding helps identify useful interventions, and an up-to-date evaluation dataset keeps the tests aligned with the application. The closing constraint is to avoid over-engineering: add complexity when a carefully chosen combination of metrics and observed failures gives it a purpose.

![Summary slide lists domain understanding, avoiding premature optimization, data complexity, updated evaluation datasets, carefully chosen metrics, scalable evaluation, and combined LLM and human evaluation, alongside a notebook QR code.](https://ai.engineer/api/talk-essay-media/77692fe1eebe3b991e0464cfdc4b2c5e195abbe5c3b3fc3d5dd5712604f05a66)

Recommendations for continued evaluation and RAG improvement.

## Resources

- [Qdrant–Quotient evaluation examples](https://github.com/qdrant/qdrant-rag-eval) — Companion repository linking the talk's hybrid retrieval and reranking example, alongside other RAG evaluation workshops.
- [Evaluation notebook and saved experiments](https://github.com/qdrant/qdrant-rag-eval/tree/master/workshop-rag-eval-qdrant-quotient/notebook) — Notebook and saved experiment files covering chunking, embedding models, generation models, rerankers and hybrid retrieval.
- [Lost in the Middle: How Language Models Use Long Contexts](https://arxiv.org/abs/2307.03172) — Experiments showing how the position of relevant evidence affects multi-document question answering and key-value retrieval.
- [Qdrant](https://github.com/qdrant/qdrant) — Rust-based vector database with vector search, payload filtering and client libraries.
- [Hybrid and Multi-Stage Queries in Qdrant](https://qdrant.tech/documentation/search/hybrid-queries/) — Current instructions for combining dense and sparse searches and composing retrieval stages through Qdrant's Query API.
- [DSPy documentation](https://dspy.ai/) — Current guides to composing language-model programs and optimizing them against evaluation metrics.

## Slides

### Presenting sponsor

Microsoft is the presenting sponsor.

[0:05](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=5s) [Microsoft presenting sponsor slide.](https://assets.ai.engineer/sites/aiecode2025/sha256/1eea0eb78fa7f6b8cb7d6897d1f0d3f0def0576d5131a4487df21c939aa73487.webp)

PRESENTING SPONSOR

Microsoft

### Innovation and platinum sponsors

AWS is the innovation sponsor; MongoDB, Google Cloud, and Neo4j are platinum sponsors.

[0:10](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=10s) [AWS is the innovation sponsor; MongoDB, Google Cloud, and Neo4j are platinum sponsors.](https://assets.ai.engineer/sites/aiecode2025/sha256/5e4db98a32051696455ddda6fd8b233613ba9fc25208f7185cd6606793dd056b.webp)

INNOVATION SPONSOR
aws

PLATINUM SPONSORS
MongoDB®
Google Cloud
neo4j

### Key discussion points

The talk examines RAG challenges and improvement techniques, including evaluation-based optimization, before presenting results and conclusions.

[0:41](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=41s) [Agenda covering RAG challenges, improvement techniques, evaluation-based optimization, and results and conclusions.](https://assets.ai.engineer/sites/aiecode2025/sha256/9029dc5ce9f19c82db3919843c66b9a80f955bc2912e73caee4c2a884a3c492e.webp)

Key discussion points :

• Core challenges of RAG
• Improvement Techniques
• Evaluation based RAG Optimization
• Results and Conclusions

Qdrant | QuotientAI

### Flavours of RAG

Naive RAG retrieves document chunks from a vector database. Advanced RAG adds query and result treatment; agentic RAG introduces tools and a self-improvement module, DSPy.

[1:25](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=85s) [Diagrams compare naive, advanced, and agentic self-improving RAG, shown with the presenter and event branding.](https://assets.ai.engineer/sites/aiecode2025/sha256/efe773546de1175f7a1e05a02648424cf64d69b87eb5ceb933c8ff4f0a12cb3e.webp)

Flavours of RAG
Qdrant | QuotientAI

Simple / Naive RAG
Document
split
Query?
encoding + indexing
Qdrant
vector search engine or vector database
Doc chunks
Knowledge Retrieval + Answer Processing
Large Language Model (LLM)

Advance RAG
Document
split
Query?
Query Treatment
- Routing
- Rewriting
- Expansion
encoding + indexing
Qdrant
vector search engine or vector database
Doc chunks
Results Treatment
- Rerank
- Fusion
- Summary
Knowledge Retrieval + Answer Processing
Large Language Model (LLM)

Agentic + Self Improving / RAG
Document
split
Query?
Agent
Tool
Tool
Tool
encoding + indexing
Qdrant
vector search engine or vector database
Doc chunks
Knowledge Retrieval + Answer Processing
DSPy
Large Language Model (LLM)

Interface text:
AIE
Microsoft
smol ai
aws

### Flavours of RAG

All three designs use a vector database. Naive RAG retrieves chunks for an LLM; advanced RAG treats queries and results; agentic RAG adds tools and self-improvement with DSPy.

[1:32](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=92s) [Diagrams compare naive, advanced, and agentic self-improving RAG, including query treatment, result treatment, tools, and DSPy.](https://assets.ai.engineer/sites/aiecode2025/sha256/88a1949007644e68ba3bb696abba5282234cf68478af2ed1c80db4320a207710.webp)

Qdrant | QuotientAI

Flavours of RAG

Simple / Naive RAG
Document
split
Query?
encoding + indexing
Qdrant
vector search engine or vector database
Doc chunks
Knowledge Retrieval + Answer Processing
Large Language Model (LLM)

Advance RAG
Document
split
Query?
Query Treatment
- Routing
- Rewriting
- Expansion
encoding + indexing
Qdrant
vector search engine or vector database
Doc chunks
Results Treatment
- Rerank
- Fusion
- Summary
Knowledge Retrieval + Answer Processing
Large Language Model (LLM)

Agentic + Self Improving / RAG
Document
split
Query?
Agent
Tool
Tool
Tool
encoding + indexing
Qdrant
vector search engine or vector database
Doc chunks
Knowledge Retrieval + Answer Processing
DSPy
Large Language Model (LLM)

### Qdrant vector search

Qdrant presents its open-source vector database as performance- and scalability-oriented. The slide reports over 10K adopters worldwide, 5M downloads and 18K GitHub stars.

[2:58](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=178s) [Qdrant overview listing performance, scalability, ease of use, resource optimization, open source and embedding support, alongside adoption figures.](https://assets.ai.engineer/sites/aiecode2025/sha256/6ba10f9338db7643bdcf20d21260c1ccdd9d3c397ba4355e55d76da19b7ce0a7.webp)

Qdrant

Enterprise-Ready, Massive-Scale Vector Search Technology for the Next AI Generation

Performance Centric
Scalability Oriented
Quick and easy to start
Resource optimization focussed
Fully Open Source project
All embeddings supported OOTB

Most Loved
open-source vector search database

> 10 K+
Adopters Worldwide

> 5 M+
Downloads

> 18 K+
Github Stars

### Challenges of RAG

Documents are split and indexed for retrieval before an LLM generates a response. Missing or unextracted information can lead to incorrect or incomplete answers.

[3:09](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=189s) [RAG workflow diagram labels challenges in data processing, ingestion, retrieval, response generation, query processing and production.](https://assets.ai.engineer/sites/aiecode2025/sha256/d0a6ab601f30dac9591001f1509f688c7f0bce29a58e7598aea618461fa47722.webp)

Qdrant | QuotientAI

Challenges of RAG

Principle challenges with RAG

Data Processing
- Missing Information
- Information Extraction

Data Ingestion
- Determine Document Chunk Parameters
- Choosing Embedding model

Information Retrieval
- Missing Relevant Documents
- Determining Retrieval Window
- Order of documents

Response Generation
- Incorrect Response Format
- Incomplete response
- Unfaithful response

Document
split
encoding + indexing
Query?
vector search engine or vector database
Qdrant
Doc chunks
Knowledge Retrieval + Answer Processing
Large Language Model (LLM)

Query Processing
- Vulnerability to Ambiguous or Vague queries

Additional Others
Production Challenges
- Coherent Responses
- Scalable performance
- Fallback Mechanism
- Security and Compliance

Extended: “Seven Failure Points When Engineering a Retrieval Augmented Generation System” by Barnett et al.

### Challenges of RAG: data processing and ingestion

Missing or unextracted information can lead to incomplete answers. Ingestion adds two choices: how to chunk documents and which embedding model understands the dataset’s terminology.

[3:43](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=223s) [RAG challenges diagram with sad-face icons beside data processing and ingestion.](https://assets.ai.engineer/sites/aiecode2025/sha256/41e0e29a9731a137ce136f66c6e039cc6b4f0d46b3c96c35238451613ef08894.webp)

Qdrant | QuotientAI

Challenges of RAG

Principle challenges with RAG

Data Processing
- Missing Information
- Information Extraction

Data Ingestion
- Determine Document Chunk Parameters
- Choosing Embedding model

Document
split
encoding + indexing
Query?
Qdrant
vector search engine or vector database
Doc chunks
Knowledge Retrieval + Answer Processing
Large Language Model (LLM)

Query Processing
- Vulnerability to Ambiguous or Vague queries

Information Retrieval
- Missing Relevant Documents
- Determining Retrieval Window
- Order of documents

Response Generation
- Incorrect Response Format
- Incomplete response
- Unfaithful response

Additional Others
Production Challenges
- Coherent Responses
- Scalable performance
- Fallback Mechanism
- Security and Compliance

Extended: “Seven Failure Points When Engineering a Retrieval Augmented Generation System” by Barnett et al.

AIE
Microsoft
smol ai
aws

### RAG improvement techniques: indexing algorithms

Choosing an indexing algorithm can improve information retrieval in RAG. The speaker suggests HNSW, BM25, and graphs as options.

[5:00](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=300s) [Grid of RAG improvement techniques with indexing algorithms outlined in red.](https://assets.ai.engineer/sites/aiecode2025/sha256/e1599ea0ca49c0c5d684c663d0a6a7a15334c434d4174d1383d8bc0f6ed5d39c.webp)

Qdrant | QuotientAI

RAG Improvement techniques

Data Cleaning
Leverage Metadata
Indexing Algorithms
Multi Vector Indexing

Advanced data extraction
Data Chunking
Document Reranking
LLM

Embedding Model
Retrieval Window
Prompt Engineering
Agents

Prompt

### Evaluation is Paramount

Evaluation helps teams measure progress and decide which RAG changes to make. The slide also connects evaluation to trust, go/no-go decisions, improvements, and compliance.

[7:25](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=445s) [Reasons to evaluate include trust, use-case outcomes, avoiding pitfalls, decisions, improvements, and compliance and ethics.](https://assets.ai.engineer/sites/aiecode2025/sha256/2ab3ee6584da4cf905dbbd63e685980bece990b3658affef93b4f6a952fc4e61.webp)

Qdrant | QuotientAI

Evaluation is Paramount !!

Why should you evaluate ?

• Establish Trust (Reputation and Confidence)
• Correlate outcomes wrt your use case
• Validation that your application avoids common pitfalls
• Criterion to make a go / no-go decisions
• Roadmap for improvements
• Compliance and Ethics

Image text: EVALUATION

### Domain-specific evaluation for RAG

Quotient combines a RAG configuration with an evaluation dataset, scores the generated completions, and feeds the results back into an iteration loop tailored to the use case.

[7:50](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=470s) [Diagram showing a RAG configuration and evaluation dataset feeding metrics and results in an iteration loop.](https://assets.ai.engineer/sites/aiecode2025/sha256/193dff1c82e101933f2489db9aad534962322bb8d2d209cc6b2e34c5eff92a4f.webp)

Domain-specific eval is essential for high-quality RAG apps

RAG quality is inherently use-case-dependent. It depends on the database and its contents.

With Quotient we can evaluate and iterate on LLM products using domain-specific, data-backed experimentation.

Iterate

Configuration
LLM
Prompt template
Question
Context
RAG Configuration

+
Eval dataset
Question | Answer | Context

QuotientAI

Eval dataset
Question | Answer | Context | Completion

Metrics

Results
score: 0.7

### Getting started with Qdrant and Quotient

Qdrant queries populate evaluation contexts. Quotient then formats prompts, runs LLMs, and computes metrics on the resulting completions.

[8:58](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=538s) [Workflow linking documents and a vector database to an evaluation dataset, prompt, language model, metrics, and results.](https://assets.ai.engineer/sites/aiecode2025/sha256/a5c155d46e48521601c187cebdebebacb483bf5aa21fdfc239d6e691e57eef93.webp)

Qdrant | QuotientAI

Let’s get started with Qdrant and Quotient

Documents → Vector Database

Eval Dataset
Question | Answer | Context
query → Vector Database
response → Eval Dataset

QuotientAI
Prompt Template
Question
Context

LLM Provider
together.ai
OpenAI

Eval Dataset
Question | Answer | Context | Completion

Metrics
Results
Score: 0.7

Interface text:
AIE
Microsoft
smol.ai
aws

### Demo Walkthrough

The pre-executed notebook sets up Qdrant and Quotient clients for a RAG question-answering demo using Qdrant’s documentation. The slide links to the notebook via a QR code.

[9:22](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=562s) [Python client setup for Qdrant and Quotient, alongside a QR code for the notebook.](https://assets.ai.engineer/sites/aiecode2025/sha256/d0d46f8df339ecb3fd711725054d1de504837a20dbaf46e4bb6c2f0617635bb2.webp)

Demo Walkthrough

# Connect to Qdrant Cloud
import qdrant_client

client = qdrant_client.QdrantClient(
    os.environ.get("QDRANT_URL"),
    api_key=os.environ.get("QDRANT_API_KEY"),
)

# Import QuotientAI client
# IMPORTANT: set your API key as an environment variable called QUOTIENT_API_KEY
from quotientai.client import QuotientClient

quotient = QuotientClient()

Python

Find our notebook here:
Qdrant | QuotientAI

AIE
Microsoft
smol ai
aws

### Evaluation Metrics

The evaluation prioritizes faithfulness to minimize hallucinations. Context relevance checks for needed information in retrieved documents; chunk relevance measures useful content versus noise.

[9:57](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=597s) [Evaluation metrics for RAG, emphasizing faithfulness and listing relevance and text similarity measures.](https://assets.ai.engineer/sites/aiecode2025/sha256/fd66c2ca16086f0282d8d5352071964ba4a0a3dcffb365fffdb13b4fd8c4af98.webp)

Qdrant | QuotientAI

Evaluation Metrics

• We want to minimize hallucinations in the LLM outputs.
• For our evaluation, we will be considering the following metrics, with a focus on Faithfulness:
  ○ Context Relevance
  ○ Chunk Relevance
  ○ Faithfulness
  ○ ROUGE-L
  ○ BERTScore
  ○ BERT Sentence Similarity

AIE
Microsoft
smol ai
aws

### Changing Chunk Parameters

To test whether more context helps, experiment 2 doubles chunk size from 512 to 1024 and overlap from 64 to 128, keeping the embedding, document count, and language model fixed.

[11:03](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=663s) [Notebook compares baseline RAG settings with an experiment using larger chunks and overlap.](https://assets.ai.engineer/sites/aiecode2025/sha256/ad27226b3d66679caff1a6e3efcc4e9aab5cb6ea9933bad96e51113c06544f88.webp)

8. Optimizing RAG through experimentation

Experiment 1:

We will be starting off with the following specifications:
• Embeddings: bge-small-en
• Chunk size: 512
• Chunk overlap: 64
• Number of docs: 3
• Language model: Mistral-7B-Instruct

9 cells hidden

Experiment 2 - Changing the chunk parameters

Keeping all other parameters constant, we changed the chunk size and chunk overlap to see if we can improve performance by providing more context.

This time, our RAG setup looks like:
• Embeddings: bge-small-en
• Chunk size: 1024
• Chunk overlap: 128
• Number of docs: 3
• Language model: Mistral-7B-Instruct

Data preprocessing based on configuration above

## for iteration 2 - lets modify chunk configuration
## We will start with creating seperate collection to store vectors

chunk_size = 1024
chunk_overlap = 128
embedding_model_name = "BAAI/bge-small-en"
num_docs = 3

COLLECTION_NAME = f"experiment_{chunk_size}_{chunk_overlap}_{embedding_model_name.split('/')[1]}"

add_documents(client,
              collection_name=COLLECTION_NAME,

Interface text:
notebook > quotient_qdrant_rag_eval_with_reranker.ipynb > Retrieval-Augmented Generation (RAG) with QuotientAI and Qdrant > Hybrid Search > Experiment 1:
Code · Markdown · Run All · Restart · Clear All Outputs · Variables · Outline
workshop-rag-eval-qdrant-quotient (Python 3.10.13)
quotient_with_reranker* · Live Share · Spaces: 4 · LF · Cell 33 of 106

### Chunk Parameter Results

Larger chunks and overlap improved context relevance and text-quality scores, but chunk relevance fell and faithfulness dropped from 0.767 to 0.692—the metric being optimized.

[11:30](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=690s) [Bar chart compares six evaluation metrics for the baseline and larger-chunk experiments.](https://assets.ai.engineer/sites/aiecode2025/sha256/4ad72fa7314f10b96b18fd6f71ffbc2446421e4d6c448b0a75cc48f6fd3e2fc3.webp)

Qdrant Documentation Question Answering

Legend:
Experiment 1: 512, 64, 3, bge-small, Mistral
Experiment 2: 1024, 128, 3, bge-small, Mistral

Y-axis: Score
Y-axis ticks: 0.0, 0.2, 0.4, 0.6, 0.8, 1.0
X-axis: Metric
Chunk Relevance: 0.469, 0.426
Context Relevance: 0.721, 0.744
Faithfulness: 0.767, 0.692
ROUGE-L: 0.232, 0.257
BERTScore: 0.622, 0.635
BERT Sentence Similarity: 0.818, 0.828

Interface text:
notebook > quotient_qdrant_rag_eval_with_reranker.ipynb > Retrieval-Augmented Generation (RAG) with QuotientAI and Qdrant > Hybrid Search > Experiment 3: Increasing number of documents retrieved
Code | Markdown | Run All | Restart | Clear All Outputs | Variables | Outline
workshop-rag-eval-qdrant-quotient (Python 3.10.13)
[46] Python
quotient_with_reranker* | Live Share | Spaces: 4 | LF | Cell 53 of 106

### Increasing Retrieved Documents

With the same chunk settings, retrieving five documents instead of three raises chunk relevance from 0.469 to 0.495 and faithfulness from 0.767 to 0.789.

[12:15](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=735s) [Notebook shows a five-document retrieval configuration and compares its metrics with the first two experiments.](https://assets.ai.engineer/sites/aiecode2025/sha256/7717c84f128f0cbfdae302cce27fb873fb7953e822dad6af02e1fd6f88291629.webp)

Experiment 3: Increasing number of documents retrieved

This time, we are using the same RAG setup as Experiment 1, but increasing the number of retrieved documents from 3 to 5.

• Embeddings: bge-small-en
• Chunk size: 512
• Chunk overlap: 64
• Number of docs: 5
• Language model: Mistral-7B-Instruct

4 cells hidden

Result Comparison

Qdrant Documentation Question Answering

Legend:
Experiment 1: 512, 64, 3, bge-small, Mistral
Experiment 2: 1024, 128, 3, bge-small, Mistral
Experiment 3: 512, 64, 5, bge-small, Mistral

Y-axis: Score
Y-axis ticks: 0.0, 0.2, 0.4, 0.6, 0.8, 1.0

Values by metric, in experiment order:
Chunk Relevance: 0.469, 0.426, 0.495
Context Relevance: 0.721, 0.744, 0.756
Faithfulness: 0.767, 0.692, 0.789
ROUGE-L: 0.232, 0.257, 0.242
BERTScore: 0.622, 0.635, 0.628
BERT Sentence Similarity: 0.818, 0.828, 0.828

Interface text:
notebook > quotient_qdrant_rag_eval_with_reranker.ipynb > Retrieval-Augmented Generation (RAG) with QuotientAI and Qdrant > Hybrid Search > Result Comparison > Observation
Code · Markdown · Run All · Restart · Clear All Outputs · Variables · Outline
workshop-rag-eval-qdrant-quotient (Python 3.10.13)
[59]
quotient_with_reranker* · Live Share · LF · Cell 52 of 106

### Changing the Language Model

Replacing Mistral with GPT-3.5 while keeping Experiment 3’s retrieval settings raised four scores and left chunk and context relevance unchanged at the chart’s displayed precision.

[12:29](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=749s) [Notebook shows the fifth experiment using GPT-3.5 and compares its scores with four earlier configurations.](https://assets.ai.engineer/sites/aiecode2025/sha256/096efcd8101bcfa1886ee109f8d28ee85658b2173559513d213330cd6fa66e78.webp)

Experiment 5: Changing the LLM

So far, we've only experimented with the retrieval side of our RAG pipeline. Let's try changing the LLM, keeping the best parameters from Experiment 3:

• Embeddings: bge-small-en
• Chunk size: 512
• Chunk overlap: 64
• Number of docs: 5
• Language model: GPT-3.5-turbo

7 cells hidden

Result Comparison

Qdrant Documentation Question Answering
Y-axis: Score
Y-axis ticks: 0.0, 0.2, 0.4, 0.6, 0.8, 1.0

Legend:
Experiment 1: 512, 64, 3, bge-small, Mistral
Experiment 2: 1024, 128, 3, bge-small, Mistral
Experiment 3: 512, 64, 5, bge-small, Mistral
Experiment 4: 512, 64, 5, all-MiniLM, Mistral
Experiment 5: 512, 64, 5, bge-small, GPT-3.5

Values in experiment order:
Chunk Relevance: 0.469, 0.426, 0.495, 0.463, 0.495
Context Relevance: 0.721, 0.744, 0.756, 0.661, 0.756
Faithfulness: 0.767, 0.692, 0.789, 0.729, 0.810
ROUGE-L: 0.232, 0.257, 0.242, 0.244, 0.292
BERTScore: 0.622, 0.635, 0.628, 0.628, 0.653
BERT Sentence Similarity: 0.818, 0.828, 0.828, 0.804, 0.842

Interface text:
notebook > quotient_qdrant_rag_eval_with_reranker.ipynb > Retrieval-Augmented Generation (RAG) with QuotientAI and Qdrant > Hybrid Search > Experiment 5: Changing the LLM
Code  Markdown  Run All  Restart  Clear All Outputs  Variables  Outline
workshop-rag-eval-qdrant-quotient (Python 3.10.13)
quotient_with_reranker*  Live Share  Spaces: 2  LF  Cell 69 of 106

### Reranker comparison

Against the naive RAG baseline, Cohere raises context relevance from 0.756 to 0.794 and faithfulness from 0.810 to 0.835. It scores highest on both among the three rerankers tested.

[14:11](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=851s) [Bar chart comparing RAG metrics across a baseline and three rerankers.](https://assets.ai.engineer/sites/aiecode2025/sha256/62e7326ec5979234e35e800f07ebb0c272ad7fbb85a9bc215ef0efdd94fd6c96.webp)

Comparing the Reranker effect on the RAG Pipeline

Qdrant Documentation Question Answering

Y-axis: Score
Y-axis ticks: 0.0, 0.2, 0.4, 0.6, 0.8, 1.0
X-axis: Metric

Legend:
Experiment 5: 512, 64, 5, bge-small, GPT-3.5
Experiment 6: 512, 64, 5, bge-small, GPT-3.5, Mixedbread
Experiment 7: 512, 64, 5, bge-small, GPT-3.5, Cohere
Experiment 8: 512, 64, 5, bge-small, GPT-3.5, Jina-Colbert

Scores in experiment order 5, 6, 7, 8:
Chunk Relevance: 0.495, 0.491, 0.502, 0.481
Context Relevance: 0.756, 0.786, 0.794, 0.776
Faithfulness: 0.810, 0.824, 0.835, 0.803
ROUGE-L: 0.292, 0.288, 0.293, 0.287
BERTScore: 0.653, 0.650, 0.651, 0.650
BERT Sentence Similarity: 0.842, 0.849, 0.846, 0.842

Interface text:
notebook > quotient_qdrant_rag_eval_with_reranker.ipynb > ...
Code  Markdown  Run All  Restart  Clear All Outputs  Variables  Outline
workshop-rag-eval-qdrant-quotient (Python 3.10.13)
quotient_with_reranker*  Live Share  Ln 7, Col 43  Spaces: 4  LF  Cell 92 of 111

### Hybrid search comparison

Hybrid search with Cohere reranking scores highest on five of six metrics. Chunk relevance is the exception: the combined approach scores 0.483, below the other three experiments.

[15:36](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=936s) [Bar chart comparing baseline, reranking, hybrid search, and combined approaches across RAG metrics.](https://assets.ai.engineer/sites/aiecode2025/sha256/1775d2801bd0962d5467b871579ce388caee54615611d19f11eb4697b33a42c9.webp)

Comparing the effect of Hybrid Search on the RAG Pipeline

Qdrant Documentation Question Answering

Y-axis: Score
Y-axis ticks: 0.0, 0.2, 0.4, 0.6, 0.8, 1.0
X-axis: Metric

Legend:
Experiment 5: 512, 64, 5, bge-small, GPT-3.5
Experiment 7: 512, 64, 5, bge-small, GPT-3.5, Cohere
Experiment 9: 512, 64, 5, bge-small, GPT-3.5, Hybrid
Experiment 10: 512, 64, 5, bge-small, GPT-3.5, Cohere + Hybrid

Values in legend order:
Chunk Relevance: 0.495, 0.502, 0.500, 0.483
Context Relevance: 0.756, 0.794, 0.781, 0.804
Faithfulness: 0.810, 0.835, 0.811, 0.849
ROUGE-L: 0.292, 0.293, 0.302, 0.317
BERTScore: 0.653, 0.651, 0.659, 0.663
BERT Sentence Similarity: 0.842, 0.846, 0.849, 0.850

Interface text:
notebook > quotient_qdrant_rag_eval_with_reranker.ipynb > Experiment 10: Hybrid Search + Reranking > Comparing the effect of Hybrid Search on the RAG Pipeline
Code | Markdown | Run All | Restart | Clear All Outputs | Variables | Outline
workshop-rag-eval-qdrant-quotient (Python 3.10.13)
quotient_with_reranker* | Live Share | Spaces: 7 | LF | Cell 104 of 112

### Experiment conclusions

Across ten RAG experiments, faithfulness rose from 0.766502 to 0.849271 without changing the prompt. The speaker describes an iterative process of evaluating metrics, observing patterns, and testing hypotheses.

[16:16](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=976s) [Notebook conclusions comparing initial and final RAG scores and summarizing six experiments.](https://assets.ai.engineer/sites/aiecode2025/sha256/d84016817259be499947195699d3e0e46f740f64c8bfc8549ce2a9b57b4c0037.webp)

Conclusions

The table below illustrates the advancements achieved through 10 experiments. It showcases the score improvements between our initial and final experiment, all attained without altering the prompt. These enhancements predominantly stem from adjustments in the RAG implementation.

Iteration | Chunk Relevance | Context Relevance | Faithfulness | ROUGE-L | BERTScore | BERT Sentence Similarity
Experiment 1 | 0.469500 | 0.721464 | 0.766502 | 0.232352 | 0.621800 | 0.817773
Experiment 10 | 0.482720 | 0.804072 | 0.849271 | 0.316757 | 0.662770 | 0.850435

The table below provides an overview of the key experiments conducted, along with their average performance. This includes the top-performing Naive RAG approach (Experiment 5), as well as four advanced RAG implementations, each testing different rerankers and a multivector hybrid search approach.

results = pd.DataFrame([exp5, exp6, exp7, exp8, exp9, exp10], columns=['experiment']+main_metrics)
results['iteration'] = results.experiment.str.split(':').str[0]
results[['chunk size', 'chunk overlap', 'num docs', 'embedding model', 'LLM', 'Advanced RAG']] = results.experiment.str.split(':').str[1].str.split(',', expand=True)

display(results[['iteration', "Advanced RAG", 'chunk size', 'chunk overlap', 'num docs', 'embedding model', 'LLM']+main_metrics])

iteration | Advanced RAG | chunk size | chunk overlap | num docs | embedding model | LLM | Context Relevance | Chunk Relevance | Faithfulness | ROUGE-L | BERT Sentence Similarity | BERTScore
0 | Experiment 5 | None | 512 | 64 | 5 | bge-small | GPT-3.5 | 0.755696 | 0.495340 | 0.809529 | 0.291575 | 0.841584 | 0.652713
1 | Experiment 6 | Mixedbread | 512 | 64 | 5 | bge-small | GPT-3.5 | 0.785544 | 0.490597 | 0.824394 | 0.288392 | 0.849143 | 0.649738
2 | Experiment 7 | Cohere | 512 | 64 | 5 | bge-small | GPT-3.5 | 0.793897 | 0.502324 | 0.834533 | 0.292739 | 0.846161 | 0.650726
3 | Experiment 8 | Jina-Colbert | 512 | 64 | 5 | bge-small | GPT-3.5 | 0.776272 | 0.480759 | 0.803262 | 0.287293 | 0.841771 | 0.650255
4 | Experiment 9 | Hybrid | 512 | 64 | 5 | bge-small | GPT-3.5 | 0.780795 | 0.500447 | 0.810670 | 0.301904 | 0.849098 | 0.658590
5 | Experiment 10 | Cohere + Hybrid | 512 | 64 | 5 | bge-small | GPT-3.5 | 0.804072 | 0.482720 | 0.849271 | 0.316757 | 0.850435 | 0.662770

You can see how starting from zero, you can build an improved RAG system by making evaluation-informed changes throughout your experimentation.

If you're interested, you can request access to Quotient by joining the waitlist on quotientai.co

Interface text:
notebook > quotient_qdrant_rag_eval_with_reranker.ipynb > Retrieval-Augmented Generation (RAG) with QuotientAI and Qdrant
Code | Markdown | Run All | Restart | Clear All Outputs | Variables | Outline
workshop-rag-eval-qdrant-quotient (Python 3.10.13)
[359] Python
master* | Live Share

### To sum up

Improve RAG through evaluation-based changes: use domain knowledge, keep the evaluation dataset current, and combine chosen metrics with LLM and human evaluations.

[17:05](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=1025s) [Summary of RAG evaluation practices, with a notebook QR code.](https://assets.ai.engineer/sites/aiecode2025/sha256/06536039d54dc8e75deaafc8abcdefee1958cfe197560e0b33d863311adcc749.webp)

Qdrant | QuotientAI

To sum up

• Domain understanding helps address challenges like document order, terminology, and chain of thought.
• Avoid over-optimizing your first run; there's no substitute for evaluation-based improvements.
• Your data is a crucial determinant of the complexity of RAG.
• Regularly update your evaluation dataset to keep it aligned with the latest challenges in your LLM application.
• Evaluate with a combination of carefully chosen metrics to effectively diagnose issues.
• Ensure the scalability of your evaluation process to accommodate future expansions and refinements.
• LLMs, combined with human evaluations, are among the most effective methods for assessing LLM-based applications.

Find our notebook here:

## Full transcript

[0:00](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=0s) [on-hold music] Hello, everyone. Welcome to our talk.

[0:15](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=15s) Uh, my name is Atita, and I work as a solution architect, uh, at Q- uh, Qdrant. And, um-

[0:21](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=21s) I am-

[0:21](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=21s) Together with me, I have, uh, Deanna.

[0:23](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=23s) I'm Deanna Emery, and I'm founding AI researcher at Quotient AI.

[0:26](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=26s) Cool. So we would be talking about navigating RAG optimization with an evaluation-driven compass. So I think the track is about RAG, so let's, uh, talk and extend what we already know and what we've already seen so far.

[0:40](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=40s) So in this talk today, we will be discussing some of the key essential topics for anyone interested in building or productionizing the most popular implementation of generative AI, that is RAG, Retrieval-Augmented Generation.

[0:53](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=53s) Uh, in simple terms, RAG combines the capabilities of, uh, searching and retrieving through the vast amount of information stored in knowledge source, um, usually a vector database. Um, and then we use this information to generate relevant and coherent responses, leveraging the capabilities of a large language model.

[1:13](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=73s) We will be talking through the known challenges of this approach, and, uh, how can you combat, uh, them by adopting an evaluation-based, uh, optimization techniques to get the desired results.

[1:23](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=83s) So without further ado, let's get started. So yeah, as, uh, simple as it's, uh, defined, RAG can be implemented in many different ways, as we can see on the slide.

[1:34](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=94s) Apologies for the busy slide, as I wanted to cover all the aspects of how, um, RAG can be implemented. And, uh, we will be starting with the simplest one, the naive RAG.

[1:43](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=103s) This, uh, involves three key steps. First, we split the documents, uh, using the specific chunking strategy. Next, we process the document embedding and store them into a vector database.

[1:55](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=115s) For each user query, we then retrieve the most relevant document chunks based on our retrieval strategy. Finally, we add these retrieved chunks to our prompt to generate an answer using a chosen LLM.

[2:08](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=128s) Advanced versions involve enhancement to user queries, such as, um, query expansion or rewriting, and, uh, post-retrieval treatments like, um, result re-ranking or fusion. The further advanced version of, uh, RAG includes, uh, query routing, um, to task-based, uh, agents and self-improvement modules like DSPy.

[2:31](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=151s) However, there's one tool that is common in all these implementations if you notice, no matter if you want to build a naive or agentic RAG, and that is a vector database.

[2:40](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=160s) One of the most common use cases for building RAG is for knowledge management. There are aspects like retrieval performance, scalability to support large data volume, and resource optimization are paramount.

[2:54](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=174s) Speaking of vector database, Qdrant is open source, uh, vector search database built on Rust and, uh, is purpose-built to support your generative AI applications built on large scale of data.

[3:06](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=186s) If you haven't already checked out, uh, please do check out.

[3:10](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=190s) So coming back to our topic, it is, uh, worth to understand and acknowledge, um, the challenges that come along with all the goodness of RAG. I'm repurposing our naive RAG, uh, architecture to highlight common possible issues on each level.

[3:22](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=202s) After all, the first step to solving any problem is to recognize there is one. To begin with, during the data processing stage, we could have, um, issues with, uh, information missing from our dataset or information that fails to get extracted from our source of information.

[3:38](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=218s) This would result in incorrect and incomplete responses. On data ingestion, there is a constant battle to determine the optimum chunking strategy, along with determining a suitable embedding model that basically understands the specificity and jargons, uh, used in your dataset.

[3:55](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=235s) Information retrieval itself is quite interesting and a very evol-- ever-evolving, uh, field. Having spent seventeen years in, uh, the space myself, I can probably say that, um, relevancy is an unsolved problem.

[4:08](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=248s) So the challenge with determining relevant documents, retrieval size or retrieval window, if you may call it, and the order of documents is unskippable. Response generation can face challenges such as incorrect and incomplete answers due to all the previously mentioned issues, and the issues of straying from the provided context.

[4:29](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=269s) To add, our query is also vulnerable to ambiguous or vague, uh, questions. There are certainly other challenges like generating coherent responses, maintaining user conversations, scaling a RAG for hundred or thousands of concurrent users, along with data security and compliance issues.

[4:47](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=287s) So looks like RAG isn't really a piece of cake after all. Fortunately, as for, like, the challenges, we have plenty of improvement techniques as well for the RAG. Let's look at them next.

[4:58](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=298s) So we saw challenges zooming into data quality, data that missed to get extracted. After all, the foundation of great responses is, uh, it lies in the richness and, uh, accuracy of its, uh, information or context in case of RAG, which can be controlled through adopting data cleaning and advanced data extraction methodologies.

[5:17](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=317s) It is not a bad idea to use a general purpose embedding model to begin with, but for added improvements, it would be a good idea to use an embedding model that comprehends the terminologies of your domain.

[5:28](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=328s) Metadata. It is a very versatile feature that can help you retrieve, um, um, uh, your, um, added, um, understanding of your documents and, uh, improving your retrieval. Plus, leveraging the metadata filtering during the retrieval, it can also help you out with filtering out the irrelevant documents.

[5:45](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=345s) So it is also probably a good idea to invest your, uh, time into determining and improving your chunking strategy, which we spoke about earlier. Sometimes just by reducing the chunk size or adding semantic chunking can work wonders.

[5:57](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=357s) We would be seeing that also when Deanna walks us through all the experimentation. So we have been aware of Lost in the Middle problem, and this is where it may be a good idea to determine an apt context size needed for your RAG to generate a helpful response.

[6:12](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=372s) Also, using suitable indexing algorithms like HNSW, BM25, or even graphs, they can do wonders.

[6:20](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=380s) Talking about suitable context size also prioritizes the document re-ranking as one of the key improvement parameters to ensure the most relevant documents, uh, that are provided in the context for LLM to generate a helpful response.

[6:34](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=394s) With LLMs in the picture, we cannot, um, overlook the difference a good prompt or thinking about questions as a chain of thought can make to the progress of-- process of, uh, response generation.

[6:46](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=406s) Semantic understanding is clearly desired. We know that, uh, text search alone doesn't works. But if the dataset has the requirement for the exact matches, it may be worth exploring dense as well as sparse vectors, and you can do that on one or many fields.

[7:01](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=421s) Similar to the embedding model, it may be worth switching and experimenting with different LLMs as well to ensure that, uh, the desired response is generated. And lastly, for better handling of task-driven, uh, user queries, such as fetching specific information or using custom data, it is beneficial to address these, uh, challenges using agents, which are well-suited for the

[7:21](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=441s) job. So with so many different levers to tweak in RAG pipeline, it's hard to know what's going wrong, what to change, where to start. This is why evaluation is very important.

[7:33](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=453s) Without an evaluation-based guided flow, it is difficult to accurately measure progress and ensure optimal performance. Evaluation also helps you to iteratively refine applications, making informed decisions, and ultimately achieve goals more effectively.

[7:48](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=468s) So on that topic, I would like to welcome Deanna, who's gonna basically walk us through how we did this, uh, experimentation.

[7:55](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=475s) Uh, thanks, Atita, for the perfect setup. [laughs] [audience applauding] Um, so this is where Quotient comes in, and I'm going to steal that from you. Thank you. Um, so because the quality of RAG is so dependent on the underlying documents, a thorough evaluation of a RAG system has to be customized to suit that specific domain and dataset.

[8:19](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=499s) Um, so Quotient's evaluation solution fills this need by enabling developers to measure the effectiveness of their LLM products accurately. Quotient's platform accelerates the experimentation process. With an evaluation dataset that contains realistic examples of inputs and expected outputs for your AI solution, you can quickly experiment and iterate to optimize

[8:44](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=524s) your RAG solutions. And if you don't have an evaluation dataset, don't worry. Uh, Quotient can help you get started by generating one for you, and you can hear more from us on this at the AI, SIISL, and Waves meetup on Friday.

[9:00](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=540s) So how does this work in practice? Once you have your Qdrant vector database set up, you can populate your evaluation dataset by submitting queries to return the contexts for the LLM.

[9:12](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=552s) You can then submit your evaluation dataset to Quotient, which handles the full orchestration, including the prompt formatting, execution of LLMs, and the metric computations.

[9:23](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=563s) So to see it in action, we've put together a demo walkthrough where we'll show you a workflow for making evaluation-informed changes to optimize your RAG system using Quotient and Qdrant.

[9:36](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=576s) For the sake of time, we've executed the notebook ahead of time, and we'll be walking you through the code outputs. And if you scare-- scan the QR code here, you can find the notebook on GitHub.

[9:46](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=586s) In this demo, we are building a RAG solution for question answering on Qdrant's documentation, and this will help enable Qdrant users to get help quickly.

[9:59](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=599s) So before we can begin evaluation, it's important to take a step back and consider what we're optimizing for. Given this use case, it's generally important to get helpful answers to the questions, but it's perhaps more important that the answers do not contain any inaccurate information that could misguide users.

[10:17](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=617s) And so in other words, we want to minimize hallucinations. And with that in mind, we will be looking at the following metrics shown here with a focus on faithfulness.

[10:27](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=627s) Uh, the first two metrics are both focused on measuring the quality of the retrieval side of RAG. Context relevance tells us whether the necessary information to answer the question is in the retrieved documents.

[10:39](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=639s) Chunk relevance tells us how much of the information retrieved is actually useful for answering the question versus just noise. Faithfulness is our hallucination metric. And, and then because be-- the focus of this talk is going to be optimizing the retrieval side of RAG, we're sticking to some of the more general text quality metrics here.

[11:01](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=661s) So when we're first getting started, we want to consider a simple naive RAG implementation to help us better optimize the data processing and vector database setup. So we start off by choosing a reasonable embedding model and chunking parameters, retrieval window, and the Mistral Instruct model.

[11:21](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=681s) And then to see if we require additional context to answer the questions, we set up a second experiment where we increase the chunk parameters.

[11:31](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=691s) And so here are the results of those first two experiments. You can see that by increasing the chunk size in experiment two, we had some minor improvements in our text quality metrics.

[11:42](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=702s) That said, we had a considerable drop in our faithfulness, which is the metric we're optimizing for. Of note, you can see that the context relevance increased, meaning that we retrieved more of the necessary information to answer the question.

[11:57](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=717s) But the chunk relevance dropped considerably, meaning that a smaller portion of the retrieved documents was actually relevant. And so what this implies is that if we simply retrieve more documents but use the smaller chunk size from before, we might get better results.

[12:15](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=735s) And so we try this out. As expected, the smaller chunk size, uh, with the larger w- retrieval window achieved the highest relevance scores and the best faithfulness score, uh, meaning that we have a lower occurrence of hallucinations.

[12:31](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=751s) In our next two iterations, we test out a new embedding model as well as a different LLM, switching to GPT-3.5. And while the embedding model experiment in the light gray didn't quite work out, for experiment five in the dark gray, we found that using the same RAG configuration but changing the LLM improved performance across all our

[12:51](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=771s) metrics. So here we're looking at the aggregated metrics for our top-performing RAG configuration, and the large variance in the context relevance, which is highlighted in red, implies that some of the questions are having a harder time retrieving the right documents.

[13:07](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=787s) So to better understand what's going on, we have to look into the data. And here we've shown the two worst-performing data points in terms of hallucination. This third column here shows us the retrieved documents, and it's a lot of text, so I'll just summarize.

[13:23](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=803s) Um, so it seems like we're returning a lot of unrelated documents that are likely over-indexing on specific words in the query like Qdrant, support, and search, and we're returning documents that repeat these terms many times.

[13:39](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=819s) So to address this issue, one possible solution could be expanding our retrieval window to capture more documents and thereby making it more likely we get the right information. But in doing so, we'd also be adding a lot of noise to our context.

[13:53](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=833s) So by now, we likely need to expand out of naive RAG and start to add in some more advanced techniques, and this is where re-ranking comes in. So we could try using a different embedding model to re-rank the retrieved documents and then return only the top few, thus weeding out some of the more unrelated ones.

[14:11](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=851s) So we try this out. We perform three different re-ranking experiments, trying re-rankers from mixedbread, Cohere, and Jina's ColBERT model, and we plot them here against our top naive RAG approach in blue.

[14:23](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=863s) And you can see that, in general, our context relevances, and with it our faithfulness, have improved with this strategy, with Cohere achieving the best relevance and faithfulness scores.

[14:36](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=876s) If we return to those same two worst-performing data points from experiment five now and look at how our re-ranking implementation did, you can see that in the second example, we're now retrieving documents that contain the desired answer, and with it, our context relevance and faithfulness are close to one.

[14:53](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=893s) That said, in the first example, we're still getting poor results, and this suggests that even after expanding our retrieval windows, we're still unable to identify the relevant documents. So if we think about the Qdrant documentation and the text within it, it contains a lot of special terminology, jargon, acronyms, and so it's unsurprising that these generally trained embedding

[15:14](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=914s) models are going to be limited in performance. So we could try fine-tuning or training our own model to address this issue, but this could be time-consuming and costly. Um, so another option to try is hybrid search, which combines sparse and dense vectors, and these sparse vectors help us capture documents that share similar terminology.

[15:37](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=937s) So we tried two implementations of this, one where we incorporate hybrid search with a re-ranker and one without. And you can see that, uh, using the Cohere re-ranker with hybrid search gives us the best performance across all metrics except for chunk relevance.

[15:53](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=953s) So looking at this, uh, hybrid search re-ranking experiment now on these same two data points, you can see that the context relevance and faithfulness scores are both close to one now, a significant improvement over our prior ones.

[16:05](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=965s) And we're also better to retrieve, better able to retrieve the information necessary to answer these questions. And so this suggests that domain-specific terminology has a big effect on our overall performance.

[16:18](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=978s) So to summarize, the table here shows what gains in performance we were able to make over just ten experiments. Starting from a faithfulness score of .76, we worked our way up to a score of just under .85, and notably, all of these gains were made without changing from a generic question answering prompt.

[16:37](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=997s) So there's certainly many more experiments to run, and we have plenty of room for improvement. But you can see how starting from scratch you can improve your RAG system by making incremental changes, evaluating using a combination of metrics that together can help you identify underlying issues, then observing patterns in your data, forming a hypothesis, and repeating the

[16:57](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=1017s) process.

[17:00](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=1020s) Thank you, Deanna. So to summarize, um, this talk and the experimentation, in this talk, we covered several key aspects of improving RAG. The baseline takeaway is that, uh, there is no substitute for evaluation-based or data-driven improvements.

[17:17](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=1037s) We emphasize leveraging domain understanding to achieve significant, uh, wins and, uh, outlined various techniques for enhancement in our experiments. To ensure continuous improvement, it is crucial to keep your evaluation dataset up to date.

[17:31](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=1051s) Lastly, avoid over-engineering your RAG application without considering a combination of carefully chosen metrics. If this talk piqued your interest and you are interested to get in touch with us, before that, some key references and the QR code.

[17:48](https://www.youtube.com/watch?v=DId2KP8Ykz4&t=1068s) Please feel free to scan them, get in touch with us. We're going to be around and looking forward to all your questions. [audience applauding] [upbeat music]
