AI Engineer World's Fair 2025
The RAG Stack We Landed On After 37 Fails
Read the talk
Building a RAG Stack That Finds the Right Help
A railway assistance query exposes the limits of basic retrieval, then shows how source inspection, reranking, evaluation, and container deployment fit together.
From a talk by Jonathan Fernandes
Before you start: Basic Python familiarity helps with the examples; retrieval-augmented generation and encoder architectures are introduced as they appear.
From a convenient prototype to an on-premise system
How do you turn a convenient RAG prototype into a system whose data and processing can stay inside a financial institution? That constraint shapes Jonathan Fernandes’s stack. An independent AI engineer working with financial clients, he prototypes in Google Colab for its free hardware accelerator, then uses Docker when deployment must support on-premise processing or a cloud environment. The talk’s “37 fails” framing introduces accumulated implementation lessons, rather than a numbered account of failed experiments.
The first choices concern orchestration, embeddings, storage, and generation. APIs make experiments easy; downloadable models make it possible to keep processing within the deployment boundary.
| Component | Prototype | Production preference |
|---|---|---|
| Orchestration | LlamaIndex or LangGraph | LlamaIndex |
| Embeddings | Closed APIs or open NVIDIA/BAAI models | Open NVIDIA/BAAI models |
| Vector database | Qdrant | Qdrant |
| Generation | Closed APIs or open models | Open models served within Docker |
Fernandes reports that Qdrant works for his workloads from a few documents to hundreds of thousands; he provides no accompanying latency, recall, or hardware measurements. For generation, his open-model examples include Llama 3.2 and Qwen3-4B, with Ollama or Hugging Face’s Text Generation Inference as serving options.
The remaining components make the pipeline inspectable and its answers assessable:
- Tracing: LangSmith or Arize Phoenix during prototyping, with Phoenix preferred in production because it can run in a Docker container. Component timings help identify whether generation or another stage consumes most of a request’s time.
- Reranking: A closed Cohere model is convenient for experiments; an open NVIDIA model fits the Docker deployment.
- Evaluation: Ragas supplies the framework for checking answer quality.
These are separate responsibilities: tracing locates execution problems, reranking changes the evidence selected for an answer, and evaluation checks whether the resulting behavior is useful.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Where can I get help in London?
The working example is a railway operator’s knowledge base: a collection of HTML help articles, queried with a deceptively simple question: “Where can I get help in London?” The answer must come from the operator’s information, so retrieval determines which facts generation can use.
The basic RAG path has three stages:
- Retrieve: Embed the query and search the vector database for semantically related documents.
- Augment: Combine those documents with the original question as context for the language model.
- Generate: Ask the model to produce an answer from that context.
The model receives the selected evidence, not the whole knowledge base. Fernandes takes this minimal pipeline into Colab so each component can be inspected and replaced.
Before indexing anything, inspect what the corpus actually contains. The first article asks whether London has wheelchair-friendly taxis and explains that most black cabs have ramps. Its Paris counterpart describes an accessible taxi firm and provides a telephone number. Both are plausible matches for words associated with travel, assistance, and accessibility, but neither necessarily answers where someone should go for help inside a station. Fernandes copies the HTML files into the notebook’s data folder and views the articles there.
LlamaIndex’s SimpleDirectoryReader loads the directory into document objects. The demonstrated corpus contains 32 documents. Each object exposes a document ID, metadata such as its file path, and its text—which in the inspected examples contains HTML. Looking at these fields connects an indexed object back to the original article:
python
from llama_index.core import SimpleDirectoryReader
documents = SimpleDirectoryReader("data").load_data()
The useful inspection goes beyond whether loading succeeded: check the text the index will receive and retain the file metadata needed to trace a later answer back to its source.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The baseline sends a station passenger to black cabs
The initial implementation loads the documents, builds an in-memory vector index, and creates a query engine. With the notebook’s model credentials configured, the core Python workflow is compact:
python
from llama_index.core import SimpleDirectoryReader, VectorStoreIndex
documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
question = "Where can I get help in London? I'm at the station."
response = query_engine.query(question)
print(response)
The answer recommends approaching London black cabs because most have ramps. It draws on relevant-looking material, but does not give the passenger a useful assistance location at the station. A functioning retrieval-and-generation chain has produced an unsatisfactory answer.
Two places in the pipeline allow more targeted intervention. Query processing happens before retrieval; for example, personally identifiable information can be removed before a question enters the RAG system. Post-retrieval processing happens after the vector search and can improve which retrieved documents reach generation. The next part of the demonstration develops that second intervention.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Retrieve broadly, then score query–document pairs
A cross-encoder takes the query and a document together. In the architecture Fernandes presents, BERT—a pretrained bidirectional Transformer encoder—processes the pair, and a classifier produces a relevance score. Because the model sees both texts together, it can judge their relationship directly. The slide depicts an output between zero and one; cross-encoder implementations can also emit logits, with a sigmoid optionally mapping them into that range.
Joint processing is expensive when repeated across many documents, and longer documents increase the work for each pair. Fernandes therefore positions the cross-encoder as a way to gain additional accuracy over a small candidate set, rather than as the first search over the entire corpus.
A bi-encoder separates the two encoding paths. The query passes through an encoder and pooling step to become a vector; a document passes through its own encoding path to become another vector. A similarity function such as cosine similarity then compares them. This separation lets document vectors be prepared for retrieval and compared with a query vector without jointly processing every query–document pair.
| Architecture | What it compares | Pipeline role |
|---|---|---|
| Bi-encoder | Separately computed vectors | Fast retrieval across many documents |
| Cross-encoder | Query and document jointly | Reranking a small candidate set |
The two architectures fit consecutive stages. Vector search finds a manageable set of candidates; the cross-encoder then spends more computation deciding which candidates best answer this particular question. In that post-retrieval role, the cross-encoder is the reranker.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
New embeddings and generation change the answer
The first replacement is Qdrant. Although Qdrant is also the production database choice, this notebook uses its in-memory local mode. Switching the vector-store integration does not by itself turn the notebook into a persistent server deployment.
Next comes the embedder. The notebook shows text-embedding-ada-002 as its existing OpenAI embedding model and mentions text-embedding-3-large as another API option. Fernandes instead downloads BAAI BGE small from Hugging Face into Colab and makes it LlamaIndex’s default embedding model. That moves embedding computation from a closed API to a locally available model.
Generation changes from GPT-3.5-turbo at temperature 0.1 to GPT-4o at temperature 0. These and the earlier embedding default are settings shown in the recorded notebook, not a statement of current library defaults. The generation change is expressed through LlamaIndex’s settings:
python
from llama_index.core import Settings
from llama_index.llms.openai import OpenAI
Settings.llm = OpenAI(model="gpt-4o", temperature=0)
Fernandes then queries the pipeline using GPT-4o, the open embedder, and Qdrant.
The answer now tells the passenger to arrive at least 75 minutes before departure and notify a team member. This is closer to station assistance than the black-cab answer, but still does not supply the requested place to get help. Changing the models has changed the response without resolving the missing location.
To investigate, inspect LlamaIndex’s source_nodes. They expose the documents used for the response, allowing the answer to be checked against its supporting HTML files:
python
for source in response.source_nodes:
print(source.node.metadata.get("file_path"))
print(source.node.get_content())
In the revised notebook response, two HTML files supplied the evidence. One is titled “Can I get assistance for a connecting UK train journey?” Following those source nodes makes the problem concrete: inspect the retrieved material before deciding that a different generator will fix the answer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reranking finds the assistance booth
The next change adds Cohere reranking through LlamaIndex’s CohereRerank node postprocessor. Qdrant retrieves five candidates, and Fernandes describes a top-two reranking setting. Those counts serve different purposes: similarity_top_k controls the candidates retrieved, while top_n controls how many the reranker retains. Expressing the five-candidate, two-retained interpretation gives this query-engine configuration:
python
import os
from llama_index.postprocessor.cohere_rerank import CohereRerank
reranker = CohereRerank(
api_key=os.environ["COHERE_API_KEY"],
top_n=2,
)
query_engine = index.as_query_engine(
similarity_top_k=5,
node_postprocessors=[reranker],
)
response = query_engine.query(
"Where can I get help in London? I'm at the station."
)
print(response)
The integration’s parameter meanings distinguish retrieval breadth from the final context size; the narration alone does not establish the exact historical configuration.
The answer now names London St. Pancras International and directs the passenger to booth number five beside the Eurostar ticket gates. It is the best response in this demonstration because it supplies an actionable location, rather than taxi information or advance-arrival instructions. That location also appears in Eurostar’s current assistance guidance; for travel planning, the current guidance recommends 60 minutes before departure, rather than the earlier generated answer’s 75 minutes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
One useful answer is the start of evaluation
The demonstration has tested one recurring question, not measured aggregate answer quality. Fernandes returns to tracing and evaluation at this point because a good answer does not explain how the pipeline will behave across the rest of the knowledge base. Phoenix or LangSmith can show the time spent in each component; Ragas supports broader evaluation across documents and different dimensions of quality.
The production choices follow the earlier deployment constraints: open BAAI or NVIDIA embeddings, Qdrant for storage and retrieval, a deployable Phoenix Docker image for tracing, and probably an NVIDIA reranker instead of the prototype’s Cohere API. Ragas uses a language model to help assess quality across multiple dimensions. Evaluation belongs alongside the pipeline as its corpus and component choices change, rather than being replaced by confidence in one successful response.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Assembling the production containers
The deployment walkthrough ends with a compose.yml file coordinating Docker images as running containers. An ingestion service connects to the knowledge base and pulls the HTML files. A Qdrant image, available from Docker Hub, provides the vector database, while a frontend application exposes the solution to users.
Ollama usually serves Fernandes’s models, with Text Generation Inference presented as an alternative. Phoenix supplies tracing, and Ragas supplies evaluation. The completed diagram puts ingestion, Qdrant, the application, model serving, tracing, and evaluation under Docker Compose, with separate selections for embedding, reranking, and generation models. The notebook’s retrieval experiment has become a deployment of distinct services: the knowledge base can be ingested, answers can be served, and both execution and answer quality can be inspected within the chosen environment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Examples integrating CohereRerank as a node postprocessor, with separate retrieval and reranking limits.
Python client with in-memory prototyping, local persistence and connections to a Qdrant server.
Evaluation dimensions for retrieval quality, faithfulness and response quality.
Instructions for self-hosting Phoenix with Docker and Compose, including persistent storage.
Official model card and inference examples for Qwen3-4B.
Further reading
- Eurostar assistance meeting pointsDocumentation
Current official station assistance locations and recommended arrival times, including booth 5 at London St Pancras.
Model card and usage instructions for an English BGE small embedding checkpoint.
Explains pairwise scoring, reranking costs and the distinction between logits and sigmoid scores.
Original research introducing BERT's bidirectional pretraining approach.
Updates since the talk
Source and deployment documentation for TGI, now in maintenance mode.
Read the complete timestamped transcript
- 0:00
So this is the RAG stack we landed on after thirty-seven fails. Hi, I'm Jonathan Fernandes, and I've been working with language models way before ChatGPT appeared on the scene.
- 0:10
So I work as an independent AI engineer, and I help companies build and ship production-ready generative AI solutions. Now, if you're new to RAG, I'll give you a bit of an introduction, but this is-- my objective is for this to be the most ROI-packed RAG guide per minute on the internet.
- 0:29
Now, if you're pretty familiar with RAG, uh, then I want to give you the details straight away, right? And this is, these are the headlines for the thirty-seven fails and the learnings from those thirty-seven fails.
- 0:39
So, uh, I've broken them down into the components that, uh, I would use in a RAG stack. So you've got the orchestration, you've got the embedding models, the vector database, the large language model, and so on.
- 0:50
Now, typically, I'll break them down into two components. Uh, a prototyping piece, which I normally do in Google Colab. Uh, one of the reasons I use Google Colab is because I get access to a free hardware accelerator, so that just makes it very easy to prototype.
- 1:03
And then normally when I'm working in production, because I tend to work with financial institutions, there's a requirement often to have a lot of my data and also all of the processing taking place on-premise.
- 1:14
And so I'll often use Docker as my solution because I can run it either on-premise or sometimes, uh, in the cloud if required. So, uh, for the orchestration layer, uh, there are a couple of options, right?
- 1:25
So prototyping, I'd use [REDACTED:username] or LangGraph, uh, and then when using it in production, I'll use [REDACTED:username]. Uh, with the embedding models, there are a couple of options. So you can u-go with closed models, and this is often helpful, especially when you can just use APIs and it makes it pretty simple to use an embedding model.
- 1:40
Or you can use open models, uh, such as the ones from NVIDIA or Baai. And again, when I move over to production, then I'll use an open model, uh, such as the one from Baai and NVIDIA.
- 1:51
Uh, vector databases, Qdrant is an excellent solution because it scales really well. So I could be working with just a couple of documents to hundreds of thousands of documents, and it just works.
- 2:01
And so Qdrant is my choice for a vector database. And then in terms of language models, uh, again, I like to use, uh, a closed models here just because of the simplicity of using APIs.
- 2:11
If I want to go ahead with an open model, I could use the likes of, uh, any of the models from, uh, Meta, or I'll use a Qwen, uh, 3 model.
- 2:19
Uh, and I'll do exactly the same thing, uh, when working in production with a, within a Docker environment. I normally use Ollama or Hugging Face, uh, Text Generation Inference to allow me to, um, serve the, uh, Llama, uh, 3.2 models or the, uh, Qwen 3.4, uh, billion models from, uh, Alibaba Cloud.
- 2:40
Now, it's really important to be able to monitor and trace your RAG solution. This just helps in terms of troubleshooting. Uh, it helps you to know, uh, you know, where the majority of the time is, for example, in terms of the, uh, language model and other components.
- 2:53
And so the two solutions here are LangSmith and, uh, Arize Phoenix, uh, for my prototype. And in production, I'll, uh, often just use, uh, Arize Phoenix, uh, because I can just very easily use that in a Docker container.
- 3:07
In terms of re-ranking and improving the accuracy of your RAG solution, uh, again, my, my go-to choice is a closed model such as, uh, the one from Cohere, or I might use an open solution, uh, such as the one from NVIDIA.
- 3:19
Uh, and again, in the Docker solution, I'd use the open solution from NVIDIA. And then finally, uh, it's really important to be able to evaluate how good your RAG solution is doing, uh, and I'll use the Ragas, uh, framework there.
- 3:30
Now, in our time together, I'll be looking at just a single knowledge base. And if you want to get the most of your time in terms of understanding RAG, I suggest you just use the next three to five minutes, pause this video and take a look at this knowledge base.
- 3:46
Uh, because we'll be answering a very simple question, which is: Where can I get help in London? And so this knowledge base is for a train or a railway company operating in London, and this is the information in terms of HTML files, uh, where you can go to each of the links and you'll be able to get,
- 4:03
uh, inform-- for that train operator. Now, if you're new to RAG, uh, let me just provide a very quick overview. So RAG is, uh, Retrieval-Augmented Generation. And so you might have the initial user query, which might be something like, "Where can I get help in London?"
- 4:17
And the first and most important step is the retrieval step. This is where you do a semantic search, wh-where you look through a vector database and you retrieve relevant, uh, documents based on that query.
- 4:29
The next piece is the augmented piece of RAG, and this is where the original query is combined with the information that's been picked up from the vector database, and this is provided as context to the language model.
- 4:44
This then allows the language model to produce the generation piece because it now has the required context and the original query, and so it can provide a response to that original question, "Where can I get help in London?"
- 4:57
So a naive RAG solution looks something like this. You have the query. That query is then embedded, and then you want to then compare that to the documents that are available in the vector database.
- 5:10
You then retrieve the relevant documents from the vector database, uh, and you pass that along with the query to the language model. The language model now has that as context and is able to generate a response to that original query.
- 5:22
Now, as I said, I like to prototype in Google Colab. So what we'll do over the next few minutes is to take a look at each of these different components in Google Colab, uh, and see how they work.
- 5:32
So the first thing you'll want to do is to get and view the knowledge, uh, base. So what I'm gonna do is I'm gonna show you the original knowledge base.
- 5:39
And so let's look at the first two articles. So the first one is, "Are there any wheelchair-friendly taxis in London?" And you've got the associated content over here. So most black, uh, cabs in London are equipped with ramps and so on.
- 5:53
And let me take a look at the next one. So are there any wheelchair-friendly taxis in Paris? And then it gives you the number for a, a taxi company, and there's a reliable taxi firm dedicated to providing accessible taxi services and so on.
- 6:06
So if you haven't had a chance to look at this knowledge base, please do take some time to look at it and, uh, w-with that one question in mind, "I'm now in London.
- 6:15
Where can I get help?" So let's go ahead and grab all of the files that we need, and let's copy it over to Google Colab. And so we have all of this in our data folder, and these are the relevant HTML files.
- 6:28
Now, I can look at those HTML files either here, or I can view them in Google Colab like this. And so these are the two files that we looked at, "Are there any wheelchair friendly taxis in London?"
- 6:39
and, "Are there any wheelchair friendly taxis in Paris?" Now let's dig a little bit deeper and look at what exactly are in the documents. So with [REDACTED:username], I can use the simple directory reader, and that'll allow me to read through all of the files in a directory, which are my HTML files, right?
- 6:54
And you can see that, um, I've been able to pull them through over here. Now let's look at some of the details of some of these documents. So I've got, uh, thirty-two documents in total.
- 7:04
If I look at the first document, you can see that I've got a document ID over here, and then you can see I've got metadata such as the file path and where the information is taken from, and so on.
- 7:15
If I wanted to see the actual text of the document, I'm able to do that over here, and you can see that this is the HTML text for that first document.
- 7:24
I can then go ahead and move to the second document, and in the same way, I've got the document ID and metadata such as the file path and the, uh, HTML file, and so on.
- 7:34
If I want to take a look at the contents of that second file, that's available over here, and this is what that HTML file looks like. Now, um, i- as part of the metadata, I'm able to determine the, the actual location of that file.
- 7:47
So this is taken from this HTML file. And for that second document, this is from the second HTML file over here. Now let's take a look at the naive RAG solution.
- 7:57
Now, what's really nice about [REDACTED:username] is that just in a couple of lines of code, I have a basic RAG solution. So in that first line, I'm going to be able to retrieve all of the HTML files from, from that directory and store them in documents.
- 8:11
I then take all of those documents and store them in an in-memory vector database, and I'm then in a position to be able to query that, uh, vector database with my question, which is, "Where can I get help in London?
- 8:25
I'm at the station." And you can see that the first response using a naive solution is not very satisfactory. You can see that you can, you can get help in London at the station by approaching London black cabs, as most of them have ramps, and so on.
- 8:40
Now, if we want to get a, a more, uh, sophisticated, uh, RAG solution, uh, then there are a couple of components we'll want to add. We'll want to add a query processing step.
- 8:51
And what happens in the query processing is that we might want to remove, uh, certain bits of information. So for example, if there's, uh, personal identifi- uh, personally identifiable, uh, information, then we'll want to remove that before, um, passing that to our RAG, uh, system.
- 9:06
And the second thing we might want to do is you might want to improve on the accuracy of some of the documents that have been retrieved from the vector database, and we can do that in the post-retrieval, uh, uh, step after the information has been retrieved from that vector database.
- 9:19
Let's look at some of the building blocks o-of these systems. So what you have is a cross-encoder. Now, your objective of having a cross-encoder is to be able to semantically compare a query with a document.
- 9:35
Uh, you want to send both your query and your document to a BERT model. Now, a BERT model is the encoder from the original transformer model, right? You then pass that on to a classifier, and then you're gonna get a result between zero and one, and that will indicate how semantically similar the query is to the document.
- 9:53
Now, as you can imagine, uh, the larger your document gets, right, the less scalable the solution gets, because now you've got a, a, a rather short query and a large document, right?
- 10:02
So this solution is excellent for additional accuracy, but it's slow and not scalable, right? So let's think about where we can put that as part of our RAG pipeline.
- 10:14
The next solution is where you can actually s-split things out, right? So one of the challenges we had in the original with the cross-encoder was the lack of scalability because we had everything just running with one model.
- 10:24
Now, what if we instead have two models, two encoders. So we have a separate, uh, encoder and a BERT model for the query, right, which then passes on to a pooling and an embedding layer.
- 10:33
And then we have a s- another model, right? Again, a BERT layer and a pooling model and embedding layer, right? And we have those as two separate models, and then we can compare how closely these, uh, the, the query is related to the document by using something like the cosine similarity.
- 10:49
Now, what's great about this is that by separating out these two models, right, this is a fast and scalable solution, and this is excellent for information retrieval. Now, given that we now have a better understanding of these cross-encoders and bi-encoders, let's try and figure out where we might want to put this in our RAG pipeline, right?
- 11:10
So given that we have a fast and scalable solution and we're having to retrieve information and compare that, that query with multiple documents, it would make sense for that bi-encoder piece to be where our vector database.
- 11:23
And similarly, because we're looking to get additional accuracy, but we can't scale and we are only able to work with a few documents, then the cross-encoder would be a really good solution post the vector database, and so post retrieval.
- 11:39
And so this is where you normally have your cross-encoder, uh, which is als- uh, also often known as a re-ranker. So let's head back to our code, and let's look at swapping out some of the, uh, components of our solution.
- 11:52
Now, previously, we had an in-memory, uh, vector database. Let's go ahead and use a Qdrant solution. Now, this Qdrant solution also happens to be an in-memory solution. Um, and so as you can see, we've now got our simple directory reader, and we're using the Qdrant client.
- 12:08
So by default, uh, we'll have a built-in embedding system, right? And so you can see that by default, the, uh, embedding solution that we have is the text-embedding-ada-002 from OpenAI.
- 12:20
Uh, if we want to, we can swap that out and use a text-embedding-3-large, again from OpenAI But what's, what we have as an option is to use an open solution.
- 12:30
So now we're not gonna be using a closed model with APIs, right? We can use an open solution and actually download the embedding model, uh, onto Google Colab, right?
- 12:40
And so what we're gonna do over here is we're gonna use the model from BaAI, uh, which is the BGE small model. Now, if you want to take a look at that, I've provided a link where you can go to Hugging Face and get a little bit more information about this embedding model.
- 12:54
And so we're using this, uh, embedding model from Hugging Face, and we can go ahead and set that as our default embedding model, uh, in [REDACTED:username]. And you can see that, uh, over the next few lines, we've actually downloaded that model onto Google Colab.
- 13:09
And now we have, uh, we've now set that up as our embedding model. Let's go ahead and swap out a, a couple of the other components, um, in our RAG pipeline, right?
- 13:19
So this is our default settings. You can see that by default, we're using, uh, the GPT-3.5-turbo model as our large language model. It has a temperature of zero point one, and you can see that we've got a couple of other components.
- 13:33
So let's go ahead and swap out that large language model, and let's just use another model from OpenAI, which is the GPT-4o model, and this time let's set the temperature to zero.
- 13:44
Now, let's go ahead and make a couple of changes to the bi-encoder, right? So we know that our vector database is going to be, uh, the Qdrant, uh, vector database, right?
- 13:54
We have our storage solution, and now let's go ahead and query using this new language model that we have, which is the GPT-4o model, right, and our new embedder, which is open, um, embedding solution.
- 14:07
Let's go ahead and query that, and let's see what sort of response we now get from our RAG solution, right? And you can see that the response has changed.
- 14:16
You can get help. So the, in response to the question, "Where can I get help in London? I'm at the station," the response has now changed to, "You can get help at the station in London by arriving at least seventy-five minutes before departure and letting a member of the team know," and so on.
- 14:30
Now, this again is not a particularly helpful response, especially if you've taken a good look at the knowledge base. But let's try and see w- how we can improve on that.
- 14:39
Before we do that, let's try and see where the, the RAG solution was able to retrieve this information from. Now, in [REDACTED:username], this is known as the source nodes, and it provides information about, uh, which, uh, files were used to provide that response.
- 14:53
And we wanted to take a, a look at those files. We can do so by looking at the source nodes, right? And so it's used these two HTML files in order for it to provide that response.
- 15:03
So, "Can I get assistance for a connecting UK train journey?" And it's taken a look at, uh, some of the information available here and also, uh, information from the second file over here.
- 15:13
Now let's make our next step, which is to use our second improvement, which is where we're gonna use a cross-encoder or our re-ranker solution. So this time, we're gonna use a closed model, so that's the one from Cohere.
- 15:28
And what we're gonna do is re-rank the top two results. So again, back to the, the five lime- lines of, uh, [REDACTED:username] code, right? We want to go ahead and pull all of the information from the directory.
- 15:40
We want to then go ahead and use our vector database, uh, which is Qdrant. We then want to go ahead and take those five results from the vector database based on that query, and we want to then go ahead and use co- the Cohere re-rank as our node post-processor.
- 15:58
And in response to that original query, which is, "Where can I get help in London? I'm at the station," we get the best response so far. So, "At London St.
- 16:09
Pancras International, you can get help by going to booth number five next to the Eurostar ticket gates." Now, one thing that we've not looked at, uh, but is an in- a, a very important part of monitoring and tracing a solution, especially when you deploy it into production, is, uh, something like the Phoenix, Arize or the LangSmith solution.
- 16:28
Right now, what this allows you to do is it allows you to determine how long each of the different components takes. So another thing that's really important is, so far, I've just looked at this one question, and I've only asked this one question to the RAG solution, which is, "Where can I get help in London?"
- 16:42
What you'll want to do when, uh, when working with a RAG solution is to have a RAG eval- evaluation framework that will allow you to test on a whole load more documents.
- 16:51
Now, one really good solution is the Ragas solution. In production, I'm actually gonna be using open model, uh, the likes of BaAI or the ones from NVIDIA. Vector database Qdrant scales incredibly well, few documents to hundreds of thousands of documents.
- 17:06
Monitoring and tracing, um, again, very important to be able to track, uh, and troubleshoot your RAG, uh, pipeline. Um, so LangSmith or Arize Phoenix, uh, in prototyping, and then there is, uh, a very good Docker solution available from Arize Phoenix that you can just pull.
- 17:22
Re-rankers, um, I'll use the closed Cohere solution for prototyping, but in production I'll probably use the NVIDIA, um, solution that allows you to, to do re-ranking. And finally, uh, RAG evaluation.
- 17:34
Ragas is great, and it allows you to, to check the, the quality of your RAG solutions across a couple of different, uh, ways of looking at that, and it, uh, works with the large language model, so it makes that task pretty painless.
- 17:46
Now, just a quick look at the production environment. I, I have a comp YAML file. This will work together with Docker Compose, and then I'll have a couple of the, uh, Docker images that I'll use here.
- 17:56
So for example, I'll have an image for ingesting the data that'll be connected to the knowledge base where I'll pull in all of the HTML files. I'll have an image for Qdrant.
- 18:05
That, that's gonna be my vector database that I can just pull from Docker Hub. I'll have a front-end app for my solution. Uh, often I'll use Ollama to, uh, serve my models, but you can also use Hugging Face is Text Generation Inference, uh, Engine here.
- 18:19
Phoenix for my tracing, right, and Ragas, uh, to be able to evaluate my models. Uh, and then I can run each of these images and have them as containers, uh, within Docker Compose, and then I've got then details about the different models that I'd use for in the embeddings and re-ranking and the large language models over here.
- 18:36
So these are my results. These are my lessons from the RAG stack we landed on after thirty-seven fails. I hope that this has been helpful to you. I'm always happy to answer any questions, so you're very welcome to connect with me and, uh, chat with me on LinkedIn.
- 18:50
Thank you for your time.