The RAG Stack We Landed On After 37 Fails

Talk transcript

1. 0:00

   So this is the RAG stack we landed on after thirty-seven fails. Hi, I'm Jonathan Fernandes, and I've been working with language models way before ChatGPT appeared on the scene.

2. 0:10

   So I work as an independent AI engineer, and I help companies build and ship production-ready generative AI solutions. Now, if you're new to RAG, I'll give you a bit of an introduction, but this is-- my objective is for this to be the most ROI-packed RAG guide per minute on the internet.

3. 0:29

   Now, if you're pretty familiar with RAG, uh, then I want to give you the details straight away, right? And this is, these are the headlines for the thirty-seven fails and the learnings from those thirty-seven fails.

4. 0:39

   So, uh, I've broken them down into the components that, uh, I would use in a RAG stack. So you've got the orchestration, you've got the embedding models, the vector database, the large language model, and so on.

5. 0:50

   Now, typically, I'll break them down into two components. Uh, a prototyping piece, which I normally do in Google Colab. Uh, one of the reasons I use Google Colab is because I get access to a free hardware accelerator, so that just makes it very easy to prototype.

6. 1:03

   And then normally when I'm working in production, because I tend to work with financial institutions, there's a requirement often to have a lot of my data and also all of the processing taking place on-premise.

7. 1:14

   And so I'll often use Docker as my solution because I can run it either on-premise or sometimes, uh, in the cloud if required. So, uh, for the orchestration layer, uh, there are a couple of options, right?

8. 1:25

   So prototyping, I'd use [REDACTED:username] or LangGraph, uh, and then when using it in production, I'll use [REDACTED:username]. Uh, with the embedding models, there are a couple of options. So you can u-go with closed models, and this is often helpful, especially when you can just use APIs and it makes it pretty simple to use an embedding model.

9. 1:40

   Or you can use open models, uh, such as the ones from NVIDIA or Baai. And again, when I move over to production, then I'll use an open model, uh, such as the one from Baai and NVIDIA.

10. 1:51

    Uh, vector databases, Qdrant is an excellent solution because it scales really well. So I could be working with just a couple of documents to hundreds of thousands of documents, and it just works.

11. 2:01

    And so Qdrant is my choice for a vector database. And then in terms of language models, uh, again, I like to use, uh, a closed models here just because of the simplicity of using APIs.

12. 2:11

    If I want to go ahead with an open model, I could use the likes of, uh, any of the models from, uh, Meta, or I'll use a Qwen, uh, 3 model.

13. 2:19

    Uh, and I'll do exactly the same thing, uh, when working in production with a, within a Docker environment. I normally use Ollama or Hugging Face, uh, Text Generation Inference to allow me to, um, serve the, uh, Llama, uh, 3.2 models or the, uh, Qwen 3.4, uh, billion models from, uh, Alibaba Cloud.

14. 2:40

    Now, it's really important to be able to monitor and trace your RAG solution. This just helps in terms of troubleshooting. Uh, it helps you to know, uh, you know, where the majority of the time is, for example, in terms of the, uh, language model and other components.

15. 2:53

    And so the two solutions here are LangSmith and, uh, Arize Phoenix, uh, for my prototype. And in production, I'll, uh, often just use, uh, Arize Phoenix, uh, because I can just very easily use that in a Docker container.

16. 3:07

    In terms of re-ranking and improving the accuracy of your RAG solution, uh, again, my, my go-to choice is a closed model such as, uh, the one from Cohere, or I might use an open solution, uh, such as the one from NVIDIA.

17. 3:19

    Uh, and again, in the Docker solution, I'd use the open solution from NVIDIA. And then finally, uh, it's really important to be able to evaluate how good your RAG solution is doing, uh, and I'll use the Ragas, uh, framework there.

18. 3:30

    Now, in our time together, I'll be looking at just a single knowledge base. And if you want to get the most of your time in terms of understanding RAG, I suggest you just use the next three to five minutes, pause this video and take a look at this knowledge base.

19. 3:46

    Uh, because we'll be answering a very simple question, which is: Where can I get help in London? And so this knowledge base is for a train or a railway company operating in London, and this is the information in terms of HTML files, uh, where you can go to each of the links and you'll be able to get,

20. 4:03

    uh, inform-- for that train operator. Now, if you're new to RAG, uh, let me just provide a very quick overview. So RAG is, uh, Retrieval-Augmented Generation. And so you might have the initial user query, which might be something like, "Where can I get help in London?"

21. 4:17

    And the first and most important step is the retrieval step. This is where you do a semantic search, wh-where you look through a vector database and you retrieve relevant, uh, documents based on that query.

22. 4:29

    The next piece is the augmented piece of RAG, and this is where the original query is combined with the information that's been picked up from the vector database, and this is provided as context to the language model.

23. 4:44

    This then allows the language model to produce the generation piece because it now has the required context and the original query, and so it can provide a response to that original question, "Where can I get help in London?"

24. 4:57

    So a naive RAG solution looks something like this. You have the query. That query is then embedded, and then you want to then compare that to the documents that are available in the vector database.

25. 5:10

    You then retrieve the relevant documents from the vector database, uh, and you pass that along with the query to the language model. The language model now has that as context and is able to generate a response to that original query.

26. 5:22

    Now, as I said, I like to prototype in Google Colab. So what we'll do over the next few minutes is to take a look at each of these different components in Google Colab, uh, and see how they work.

27. 5:32

    So the first thing you'll want to do is to get and view the knowledge, uh, base. So what I'm gonna do is I'm gonna show you the original knowledge base.

28. 5:39

    And so let's look at the first two articles. So the first one is, "Are there any wheelchair-friendly taxis in London?" And you've got the associated content over here. So most black, uh, cabs in London are equipped with ramps and so on.

29. 5:53

    And let me take a look at the next one. So are there any wheelchair-friendly taxis in Paris? And then it gives you the number for a, a taxi company, and there's a reliable taxi firm dedicated to providing accessible taxi services and so on.

30. 6:06

    So if you haven't had a chance to look at this knowledge base, please do take some time to look at it and, uh, w-with that one question in mind, "I'm now in London.

31. 6:15

    Where can I get help?" So let's go ahead and grab all of the files that we need, and let's copy it over to Google Colab. And so we have all of this in our data folder, and these are the relevant HTML files.

32. 6:28

    Now, I can look at those HTML files either here, or I can view them in Google Colab like this. And so these are the two files that we looked at, "Are there any wheelchair friendly taxis in London?"

33. 6:39

    and, "Are there any wheelchair friendly taxis in Paris?" Now let's dig a little bit deeper and look at what exactly are in the documents. So with [REDACTED:username], I can use the simple directory reader, and that'll allow me to read through all of the files in a directory, which are my HTML files, right?

34. 6:54

    And you can see that, um, I've been able to pull them through over here. Now let's look at some of the details of some of these documents. So I've got, uh, thirty-two documents in total.

35. 7:04

    If I look at the first document, you can see that I've got a document ID over here, and then you can see I've got metadata such as the file path and where the information is taken from, and so on.

36. 7:15

    If I wanted to see the actual text of the document, I'm able to do that over here, and you can see that this is the HTML text for that first document.

37. 7:24

    I can then go ahead and move to the second document, and in the same way, I've got the document ID and metadata such as the file path and the, uh, HTML file, and so on.

38. 7:34

    If I want to take a look at the contents of that second file, that's available over here, and this is what that HTML file looks like. Now, um, i- as part of the metadata, I'm able to determine the, the actual location of that file.

39. 7:47

    So this is taken from this HTML file. And for that second document, this is from the second HTML file over here. Now let's take a look at the naive RAG solution.

40. 7:57

    Now, what's really nice about [REDACTED:username] is that just in a couple of lines of code, I have a basic RAG solution. So in that first line, I'm going to be able to retrieve all of the HTML files from, from that directory and store them in documents.

41. 8:11

    I then take all of those documents and store them in an in-memory vector database, and I'm then in a position to be able to query that, uh, vector database with my question, which is, "Where can I get help in London?

42. 8:25

    I'm at the station." And you can see that the first response using a naive solution is not very satisfactory. You can see that you can, you can get help in London at the station by approaching London black cabs, as most of them have ramps, and so on.

43. 8:40

    Now, if we want to get a, a more, uh, sophisticated, uh, RAG solution, uh, then there are a couple of components we'll want to add. We'll want to add a query processing step.

44. 8:51

    And what happens in the query processing is that we might want to remove, uh, certain bits of information. So for example, if there's, uh, personal identifi- uh, personally identifiable, uh, information, then we'll want to remove that before, um, passing that to our RAG, uh, system.

45. 9:06

    And the second thing we might want to do is you might want to improve on the accuracy of some of the documents that have been retrieved from the vector database, and we can do that in the post-retrieval, uh, uh, step after the information has been retrieved from that vector database.

46. 9:19

    Let's look at some of the building blocks o-of these systems. So what you have is a cross-encoder. Now, your objective of having a cross-encoder is to be able to semantically compare a query with a document.

47. 9:35

    Uh, you want to send both your query and your document to a BERT model. Now, a BERT model is the encoder from the original transformer model, right? You then pass that on to a classifier, and then you're gonna get a result between zero and one, and that will indicate how semantically similar the query is to the document.

48. 9:53

    Now, as you can imagine, uh, the larger your document gets, right, the less scalable the solution gets, because now you've got a, a, a rather short query and a large document, right?

49. 10:02

    So this solution is excellent for additional accuracy, but it's slow and not scalable, right? So let's think about where we can put that as part of our RAG pipeline.

50. 10:14

    The next solution is where you can actually s-split things out, right? So one of the challenges we had in the original with the cross-encoder was the lack of scalability because we had everything just running with one model.

51. 10:24

    Now, what if we instead have two models, two encoders. So we have a separate, uh, encoder and a BERT model for the query, right, which then passes on to a pooling and an embedding layer.

52. 10:33

    And then we have a s- another model, right? Again, a BERT layer and a pooling model and embedding layer, right? And we have those as two separate models, and then we can compare how closely these, uh, the, the query is related to the document by using something like the cosine similarity.

53. 10:49

    Now, what's great about this is that by separating out these two models, right, this is a fast and scalable solution, and this is excellent for information retrieval. Now, given that we now have a better understanding of these cross-encoders and bi-encoders, let's try and figure out where we might want to put this in our RAG pipeline, right?

54. 11:10

    So given that we have a fast and scalable solution and we're having to retrieve information and compare that, that query with multiple documents, it would make sense for that bi-encoder piece to be where our vector database.

55. 11:23

    And similarly, because we're looking to get additional accuracy, but we can't scale and we are only able to work with a few documents, then the cross-encoder would be a really good solution post the vector database, and so post retrieval.

56. 11:39

    And so this is where you normally have your cross-encoder, uh, which is als- uh, also often known as a re-ranker. So let's head back to our code, and let's look at swapping out some of the, uh, components of our solution.

57. 11:52

    Now, previously, we had an in-memory, uh, vector database. Let's go ahead and use a Qdrant solution. Now, this Qdrant solution also happens to be an in-memory solution. Um, and so as you can see, we've now got our simple directory reader, and we're using the Qdrant client.

58. 12:08

    So by default, uh, we'll have a built-in embedding system, right? And so you can see that by default, the, uh, embedding solution that we have is the text-embedding-ada-002 from OpenAI.

59. 12:20

    Uh, if we want to, we can swap that out and use a text-embedding-3-large, again from OpenAI But what's, what we have as an option is to use an open solution.

60. 12:30

    So now we're not gonna be using a closed model with APIs, right? We can use an open solution and actually download the embedding model, uh, onto Google Colab, right?

61. 12:40

    And so what we're gonna do over here is we're gonna use the model from BaAI, uh, which is the BGE small model. Now, if you want to take a look at that, I've provided a link where you can go to Hugging Face and get a little bit more information about this embedding model.

62. 12:54

    And so we're using this, uh, embedding model from Hugging Face, and we can go ahead and set that as our default embedding model, uh, in [REDACTED:username]. And you can see that, uh, over the next few lines, we've actually downloaded that model onto Google Colab.

63. 13:09

    And now we have, uh, we've now set that up as our embedding model. Let's go ahead and swap out a, a couple of the other components, um, in our RAG pipeline, right?

64. 13:19

    So this is our default settings. You can see that by default, we're using, uh, the GPT-3.5-turbo model as our large language model. It has a temperature of zero point one, and you can see that we've got a couple of other components.

65. 13:33

    So let's go ahead and swap out that large language model, and let's just use another model from OpenAI, which is the GPT-4o model, and this time let's set the temperature to zero.

66. 13:44

    Now, let's go ahead and make a couple of changes to the bi-encoder, right? So we know that our vector database is going to be, uh, the Qdrant, uh, vector database, right?

67. 13:54

    We have our storage solution, and now let's go ahead and query using this new language model that we have, which is the GPT-4o model, right, and our new embedder, which is open, um, embedding solution.

68. 14:07

    Let's go ahead and query that, and let's see what sort of response we now get from our RAG solution, right? And you can see that the response has changed.

69. 14:16

    You can get help. So the, in response to the question, "Where can I get help in London? I'm at the station," the response has now changed to, "You can get help at the station in London by arriving at least seventy-five minutes before departure and letting a member of the team know," and so on.

70. 14:30

    Now, this again is not a particularly helpful response, especially if you've taken a good look at the knowledge base. But let's try and see w- how we can improve on that.

71. 14:39

    Before we do that, let's try and see where the, the RAG solution was able to retrieve this information from. Now, in [REDACTED:username], this is known as the source nodes, and it provides information about, uh, which, uh, files were used to provide that response.

72. 14:53

    And we wanted to take a, a look at those files. We can do so by looking at the source nodes, right? And so it's used these two HTML files in order for it to provide that response.

73. 15:03

    So, "Can I get assistance for a connecting UK train journey?" And it's taken a look at, uh, some of the information available here and also, uh, information from the second file over here.

74. 15:13

    Now let's make our next step, which is to use our second improvement, which is where we're gonna use a cross-encoder or our re-ranker solution. So this time, we're gonna use a closed model, so that's the one from Cohere.

75. 15:28

    And what we're gonna do is re-rank the top two results. So again, back to the, the five lime- lines of, uh, [REDACTED:username] code, right? We want to go ahead and pull all of the information from the directory.

76. 15:40

    We want to then go ahead and use our vector database, uh, which is Qdrant. We then want to go ahead and take those five results from the vector database based on that query, and we want to then go ahead and use co- the Cohere re-rank as our node post-processor.

77. 15:58

    And in response to that original query, which is, "Where can I get help in London? I'm at the station," we get the best response so far. So, "At London St.

78. 16:09

    Pancras International, you can get help by going to booth number five next to the Eurostar ticket gates." Now, one thing that we've not looked at, uh, but is an in- a, a very important part of monitoring and tracing a solution, especially when you deploy it into production, is, uh, something like the Phoenix, Arize or the LangSmith solution.

79. 16:28

    Right now, what this allows you to do is it allows you to determine how long each of the different components takes. So another thing that's really important is, so far, I've just looked at this one question, and I've only asked this one question to the RAG solution, which is, "Where can I get help in London?"

80. 16:42

    What you'll want to do when, uh, when working with a RAG solution is to have a RAG eval- evaluation framework that will allow you to test on a whole load more documents.

81. 16:51

    Now, one really good solution is the Ragas solution. In production, I'm actually gonna be using open model, uh, the likes of BaAI or the ones from NVIDIA. Vector database Qdrant scales incredibly well, few documents to hundreds of thousands of documents.

82. 17:06

    Monitoring and tracing, um, again, very important to be able to track, uh, and troubleshoot your RAG, uh, pipeline. Um, so LangSmith or Arize Phoenix, uh, in prototyping, and then there is, uh, a very good Docker solution available from Arize Phoenix that you can just pull.

83. 17:22

    Re-rankers, um, I'll use the closed Cohere solution for prototyping, but in production I'll probably use the NVIDIA, um, solution that allows you to, to do re-ranking. And finally, uh, RAG evaluation.

84. 17:34

    Ragas is great, and it allows you to, to check the, the quality of your RAG solutions across a couple of different, uh, ways of looking at that, and it, uh, works with the large language model, so it makes that task pretty painless.

85. 17:46

    Now, just a quick look at the production environment. I, I have a comp YAML file. This will work together with Docker Compose, and then I'll have a couple of the, uh, Docker images that I'll use here.

86. 17:56

    So for example, I'll have an image for ingesting the data that'll be connected to the knowledge base where I'll pull in all of the HTML files. I'll have an image for Qdrant.

87. 18:05

    That, that's gonna be my vector database that I can just pull from Docker Hub. I'll have a front-end app for my solution. Uh, often I'll use Ollama to, uh, serve my models, but you can also use Hugging Face is Text Generation Inference, uh, Engine here.

88. 18:19

    Phoenix for my tracing, right, and Ragas, uh, to be able to evaluate my models. Uh, and then I can run each of these images and have them as containers, uh, within Docker Compose, and then I've got then details about the different models that I'd use for in the embeddings and re-ranking and the large language models over here.

89. 18:36

    So these are my results. These are my lessons from the RAG stack we landed on after thirty-seven fails. I hope that this has been helpful to you. I'm always happy to answer any questions, so you're very welcome to connect with me and, uh, chat with me on LinkedIn.

90. 18:50

    Thank you for your time.