HybridRAG: A Fusion of Graph and Vector Retrieval to Enhance Data Interpretation

Talk transcript

1. 0:00

   [on hold music] To quickly introduce myself, my name is Mitesh.

2. 0:16

   I lead the developer advocate team at NVIDIA, and the goal of my team is to, uh, create technical workflows, notebooks, uh, for different applications, and then we release that code base, uh, on GitHub.

3. 0:27

   So developers in general, which is me and you, all of us together, we can harness that, uh, that knowledge and take it further for the application or use case that you're working on.

4. 0:38

   So that is what my, uh, my team does, including myself. In today's talk, um, I'm, I'm gonna talk about this project that we did with one of our partners, um, um, and, um, some of my colleagues at NVIDIA and our partner about how can we create a GraphRAG system, what are the advantages of it, and if we

5. 0:55

   add the hybrid nature to it, how it is helpful. So that's what my, uh, my talk is going to be on. I will not give-- I will not be able to give you a ten-feet view where you can-- I can dive with you in the code base, but there is a GitHub link at the end of this

6. 1:11

   talk which you can, um, scan, and all these notebooks, whatever I'm gonna talk about, is available for you to take home.

7. 1:19

   But I'll give you a ten-thousand-feet view or if you are trying to build your own GraphRAG system, how can you build it?

8. 1:25

   So, uh, a quick refresher, uh, what is knowledge graph? Um, and why are they important? So, um, it is a network that represents relationship between different entities, and those entities can be anything.

9. 1:38

   It can be people, places, uh, concept, events. A, a simple example would be me being here, what is my relationship to AI World Fair Conference, AI Engineers World Fair Conference?

10. 1:49

    And my relationship is I'm a speaker at this conference. What is my relationship to anyone who's attending here? Well, uh, our relationship is you attended my session. So, uh, this edge of relationship, uh, between the two entities becomes very important, uh, to, um, which only graph-based network can exploit or knowledge graphs can exploit.

11. 2:09

    And that is the reason why, uh, there's a lot of active research happening in this domain of how you can harness GraphRAG, um, uh, how can, how can harness knowledge graph and put it into a RAG-based system.

12. 2:22

    So the goal is three things. How can you create a triplet, which is the-- which defines the relationship between these entities that GraphRAG or, um, graph-based system or knowledge graph is really good at exploiting.

13. 2:38

    And that's what is unique about this knowledge graph. So if you think about, um, um, um, why can they work better than semantic, uh, uh, RAG system? Well, it captures the information between entities in much more detail.

14. 2:53

    So those connections can, um, can provide a very comprehensive view, um, um, of the knowledge that you, that you are creating in your RAG system, and that will become very important to exploit when you are retrieving some of that information and cr-- and converting that into, into a response for the user who's asking that question.

15. 3:13

    And it, and it has the ability, um, um, to organize your data from multiple sources. I mean, that's a given no matter, uh, what kind of RAG system you're building.

16. 3:24

    So how do we create a GraphRAG or a hybrid system? So this is the high-level diagram of what it entails. So I broke it down into four components. The very first thing is your data.

17. 3:33

    You need to process your data. The better you process your data, the better is the knowledge graph. The better is the knowledge graph, the better is the retrieval. So four components: data, data processing, your graph creation, or your semantic, uh, um, um, embedding, uh, vector database creation.

18. 3:49

    Those are the three, uh, steps. And then the last step is, of course, inferencing when you're asking questions, uh, uh, to your, uh, RAG pipeline.

19. 3:59

    And at a higher level, this can be broken down into two big pieces: offline, online. So all your data processing, uh, um, um, work, which is a one-time process, is offline.

20. 4:10

    And, and once you have created your, uh, uh, knowledge graph, uh, which is your triplet entity relationship, entity two, um, or your semantic, uh, vector database, once you have it, then it's all about querying it and converting that information into, um, um, a response that is readable to the user.

21. 4:28

    It cannot be something that here are the three re-relationship, and then, uh, we as the user have to go figure out what does this exactly mean.

22. 4:39

    So the top, um, uh, part of this, um, um, flow diagram is where you build your semantic, uh, um, vector database, which is you, you pick your, uh, um, documents, and then you convert them into vector embeddings, and you store it into a vector database.

23. 4:56

    So that piece is, uh, is how you create your semantic, uh, uh, vector database. And then the piece below is, um, how you create your knowledge graph. And it is much more, uh, um...

24. 5:08

    There are much more steps that you have to follow, a care that you have to take when you're creating your knowledge graph.

25. 5:18

    So diving in the first step, creating your knowledge graph. How can you create those triplets out of documents that are, that are not that structured? So, um, creating triplets which, uh, which exposes the information between two entities and picking up those entities, uh, so that that information becomes helpful is very important.

26. 5:36

    Uh, here's a simple example. This document is of ExxonMobil's, uh, um, results, I think, uh, their, their quarterly results, and we, we try to pick up, uh, um, the relationship or create the, the knowledge graph using an LLM.

27. 5:50

    And if you see at the first line, uh, it's ExxonMobil, which is a company, that's the entity. Uh, um, cut is the feature of, uh, um, of that entity, spending oil and gas exploration, um, and activity.

28. 6:03

    Uh, um, um, my apologies. Cut is the relationship between ExxonMobil and spending on oil and gas exploration. An activity is the, the, um, um, the name of the entity spending on oil and gas exploration.

29. 6:15

    So this is how the relationship needs to be exploited. Now, the question that comes to our mind is that sounds very difficult to do. And exactly it is difficult to do, and that is the reason why we need to harness, uh, or we need to use LLMs to figure out a way to extract this information and structure

30. 6:33

    it for us so that we can save it in, uh, um, um, in a triplet format. And how can we do that?

31. 6:41

    Prompt engineering. But we need to be much more, uh, uh, defined about it. So you-- Based on the use case that you are trying to work on, you can define your ontology.

32. 6:52

    And once you have defined your ontology, you can put it in your prompt and then ask the LLM to go extract this information that is ontology-specific from the documents, and then structure it in that way so that that can be stored in the form of a triplet.

33. 7:08

    This step is very important. You might be spending a lot of time here to make sure your prompt is doing the right thing and it is creating the right ontology for you.

34. 7:18

    If your ontology is not right, uh, if your triplets are not right, if they are noisy, your retrieval will be noisy. So this is where you will be go-going back and forth figuring out how to get a better ontology.

35. 7:32

    So thi-this is where you will spend-- My take is this is where you will spend, uh, eighty percent of your time to make sure you get the ontology right, and you'll be going back and forth in an iterative manner to see how you can make it better over time.

36. 7:46

    And then the next vector database for a hybrid RAG system is to create, uh, um, the semantic vector database. And that is very reasonably straight-straight-straightforward or it is well-studied.

37. 7:56

    So you pick your document. This is the first page of Attention Is All You Need research paper, and you, you break it into chunk sizes. And you, you have another factor called overlap.

38. 8:05

    And chunk sizes are important because what semantic vector database does is it will, it will, um, pick up that chunk and, um, convert that into-- use the embedding model and convert them into a, a, a embedding vector and store into the vector database.

39. 8:19

    And it will... If you don't have an overlap, then the context between the previous and the, and the next chunk will be lost if there is any relationship. So you try to be smart on how much overlap do I need between my previous chunk and the, and the next chunk, and what is the size of the chunk

40. 8:34

    that I should, uh, um, I should use when I'm chunking my documents into different paragraphs. That is where the, the advantage of graph RAG comes into play. Because, uh, if you think about it, the important information, which is, uh, the relationship between different entities, are not exploited by, uh, by your semantic, uh, uh, vector database, but they

41. 8:53

    are exploited really well when you're trying to, um, um, use a knowledge graph or create a knowledge graph-based system. So once you have created this, uh, um, this knowledge graph, what is the next step?

42. 9:05

    Now, now comes the retrieval piece, which is, um, um, you, you ask a question, uh, "What is, uh, ExxonMobil's, um, um, uh, cut, uh, this quarter that, that it is looking like?"

43. 9:16

    And knowledge graph, uh, um, will, will help you figure out how to retrieve those nodes or those entities and the relationship between them. But if you do, uh, a very flat retrieval, which is a single hop, you are missing, uh, the, the most important, um, um, piece that graph allows you, which is exploitation through multiple nodes that

44. 9:40

    you can think about. And that becomes very, very, very important. I, I cannot stress how important that becomes. So think of different strategies. Again, you will spend a lot of time to optimize this, whether you should look at, um, single hop, double hop, how much deep you want to go so that nodes, um, the relationship between your

45. 9:56

    first node to the second node, your second node to the third node is exploited pretty well. And, and the, the more deeper you go, the better context you'll get.

46. 10:04

    But there's a disadvantage of that. The more deeper you go, the more time you're gonna spend on retrieving that information. So then, uh, uh, latency becomes a factor as well, especially when you're working in a production environment.

47. 10:15

    So there is a sweet spot that you'll have to hit when you're trying to, um, um, go, um, how deep you want to go, how mu- how many hops you want to go into your graph versus how many, uh, um, what is the latency that you can, uh, um, you can survive.

48. 10:29

    So, so that becomes very, uh, very important.

49. 10:33

    And those-- Some of those searches can be accelerated. So, um, um, um, we created a library called CooGraph, um, which, which is ava- which is available or integrated in a lot of, um, libraries out there like NetworkX and whatnot.

50. 10:45

    But that acceleration becomes important so that it gives you the flexibility to get deeper into your graph, go through multiple hops, but at the same time, you can reduce the latency.

51. 10:55

    So your performance of your graph improves, uh, a lot.

52. 11:02

    So this is the-- where the retrieval piece comes into play, where you can have different strategies defined so that when you're querying, uh, your data, um, and get-getting the responses, you can have better responses.

53. 11:17

    And the other important piece, uh, I personally worked on this piece, so I, I can talk at length on this, but, uh, I'm, I'm gonna give you a very high level, um, is evaluating the performance.

54. 11:26

    And there are multiple factors that you can evaluate on: faithfulness, um, answer relevancy, uh, precision recall. Um, um, if you try to use an LLM model, uh, helpfulness, correctiveness, coherence, complexity, verbosity, all these factors becomes very important.

55. 11:41

    So there is a library, a PyPI store library called RAGAS. Um, it is meant to evaluate your RAG workflow end-to-end. Anyone who've used RAGAS for evaluating your graph RAG?

56. 11:52

    All right. A few of them. Thank you. But it is, it is an amazing library that you can, uh, uh, use to evaluate your, uh, your RAG pipeline end-to-end because it evaluates the response, it evaluates the retrieval, and it evaluates what the query is.

57. 12:06

    So eval-- Uh, it, it will evaluate your, your pipeline end-to-end, which becomes very handy When you're, when you're trying to test whether my retrieval is doing the right thing or whether my, uh, the questions that I'm asking, is the LLM interpreting it in, in the right way or not?

58. 12:19

    So you can break down your responses in, uh, the RAGAS pipeline will evaluate all those pieces and see what your eventual score is. So it is a pip install library.

59. 12:29

    The other is LLM-- Uh, and RAGAS under the hood, uh, uses an LLM, um, um, no surprises there. Uh, by default it is integrated with, uh, um, GPT, uh, but it provides, um, um, you the flexibility that if you have your own, um, uh, model, you can bring it in as well, and you can, uh, uh, wire

60. 12:48

    it up with your API, and you can use that LLM to figure out, uh, on these four, uh, four, uh, evaluation parameters that RAGAS offers. So, so it's, uh, it's, it's, it's quite compl-- I wouldn't say it's comprehensive, but it's really good in terms of giving you that flexibility.

61. 13:02

    The other part is, uh, um, uh, using a model that is meant to evaluate specifically the response coming out of LLM. And that is where this model LlamaNimotron, uh, three hundred and forty billion reward model that we released, I think a few years ago.

62. 13:15

    At that time, it was a really good response model. It's, it's a three hundred and forty billion parameter model, so reasonably big. But, uh, it evaluates, uh, um, um, it's a reward model, so it will go and evaluate the response of another LLM and, um, judge it in terms of, um, um, how the responses are looking, looking

63. 13:32

    like on these five parameters, but it is meant to go and judge other LLMs. That is how it was trained.

64. 13:42

    So moving further, I would like to use this analogy that for, uh,

65. 13:47

    to create a graph-RAG system, it will take you, uh, which is eighty percent of the job, it will take you twenty percent of your time. But then to, uh, make it better, which is the last twenty percent-- Uh, sorry, which is the, um, the eighty-twenty rule.

66. 14:02

    The last twenty percent will take eighty percent of your time, because now you are in the process of optimizing it further to make it make sure it works for the use case good enough, uh, um, um, um, for, uh, for the application that you're working on.

67. 14:16

    And there are some strategies there which I would like to, uh, walk you through. So one, as I said before, which I couldn't stress enough, the way you are creating your knowledge graph out of your unstructured data becomes very important.

68. 14:28

    The better your knowledge graph, the better results you're gonna get. And something that we did as experimentation through this use case that we were exploring with one of our partners, uh, was can we fine-tune an LLM model to get the quality of the, um, of the triplets that we are creating better?

69. 14:48

    And does that improve results? Can we do a better job at data processing, like removing regex, apostrophes, uh, brackets, words that, uh, um, characters that don't matter? If we remove them, does it give you better results?

70. 15:01

    So these are like small things that, um, that you can think about, but it gives you-- it, it improves the performance of your overall system. So that is where you-- I'm talking about eighty percent of your time.

71. 15:11

    Small nitty-gritty, other things that you are-- the knobs that you are, um, um, fine-tuning wi- um, slowly and steadily to make sure your performance gets better and better. And I would like to share a few strategies that we did, which we got, uh, um, which led us to, uh, uh, which led us to, uh, um, uh, get

72. 15:28

    better results. So the very first thing is, uh, um, regex or just cleaning out your data. Um, we, we removed, uh, apostrophe S, other, other, other re-- uh, characters that are not that important, if you think about, uh, a triple- triplet generation.

73. 15:44

    That led us to, uh, um, um, to better, uh, better re- um, results. We, we then implemented another strategy of, um, reducing the, um, um, not, not missing out of longer output, making it smaller.

74. 15:57

    That got us, uh, uh, better results. And we also fine-tuned the La- um, the Llama three point three model or three point two model, and that got us better, uh, better results.

75. 16:05

    So if you look at the last three columns, you'll see that by using Llama three point three as is, we got seventy per- one percent accuracy. So this was tested on hundred, uh, triplets to see how it is performing.

76. 16:19

    And as it got-- Sorry, hundred documents. So as it got better and, uh, a-as we introduced LoRA, we fine-tuned the Llama three point one model, our, our accuracy or, uh, performance went up from seventy-one to eighty-seven percent.

77. 16:31

    And then we did those small tweaks, uh, it improved the performance better. Again, remember, this is on hundred documents, so the accuracy is looking high. But if your document pool increases, that will come down a bit.

78. 16:40

    But in comparison to where we were before, we saw improvement. And, and that is where the small, uh, uh, tweaks come into play, which will be very, very, very helpful to you when you're putting a, a system, um, a graph RAG or a RAG system into production.

79. 16:56

    The other is from a latency standpoint. Um, so if your graph gets bigger and bigger, now you're talking about a network which, which goes into millions or billions of parameter and, uh, or millions and billions of nodes.

80. 17:08

    Now how do you, how do you do search in, uh, um, in those, uh, millions and billions, uh, um, um, um, in the graph that has got millions or billions of nodes?

81. 17:18

    And that is where acceleration comes into play. So with, with, with WhoGraph, which is now available through NetworkX. So NetworkX is also, al-also a, uh, a pip install library.

82. 17:27

    Uh, anyone who use Net-NetworkX here? All right. A few. Okay. Um, so NetworkX is also a pip install library. Uh, under the hood, um, it uses, um, acceleration. And if you see a few of the algorithms, uh, we, um, we, we did a, uh, performance test on that.

83. 17:42

    And, um, um, you can see the amount of latency in terms of overall execution reducing drastically. So that is where you can, again, small tweaks which will lead you to better results.

84. 17:53

    So these are two things that we experimented, which led us to, um, to better results in terms of accuracy, as well as reducing the overall latency. And these are small tweaks, and it, it leads us to better results.

85. 18:07

    So then the question obviously is, should I, uh, use graph or should I use semantic, uh, um, based RAG system or should I use hybrid? And I'm gonna give you the diplomatic answer.

86. 18:17

    It depends. [laughs] But, but there are a few things I would like you, you guys to take home to, uh, to, um, um, uh, which will help you to come up to a decision so that you can make an educated guess that for this use case that I'm working on, a RAG system would solve the problem, I don't

87. 18:31

    need a graph, and vice versa, or I need a hybrid approach. So it, it depends on two f- two factors. One is your data. Um, um, traditionally, if you look at retail data, if you look at FSI data, if you look at, uh, employee database of companies, those have a really good structure defined.

88. 18:48

    So those, uh, um, kind of dataset becomes really good use cases for a graph-based system.

89. 18:55

    And the other thing you think about is even if you have unstructured data, can you create a good, uh, graph, a knowledge graph out of it? If the answer is yes, then it's worthwhile experimenting, uh, um, with, uh, um, um, to go the graph path.

90. 19:09

    And it depend, it will depend on the application and use case. So if your use case requires to, um, to understand the complex relationship and then extract that information, uh, um, to, for the response that you're, um, um, for the questions that you are asking, only then it makes sense, uh, to use GraphRAG.

91. 19:25

    Because remember, these are compute-heavy s- uh, um, heavy systems, so you need to make sure that these things are taken care of. I am running out of time, I think.

92. 19:32

    But, uh, as I said before, all these things that I talked about, I gave you a 10,000 feet view, but if you want to get a hundred feet view where you are coding into, um, into things, all these things is available on GitHub, even the fine-tuning of the Llama 1.1, uh, LoRA model.

93. 19:46

    And we had a workshop, a two-hour workshop. So I gave you a twenty-minute talk, but this whole workshop is covered, uh, in two hours as well. And lastly, um, join our developer programs.

94. 19:55

    We do release all these things on a regular basis. You, if you join the mailing list, you get this information based on your interest. And as, um, uh, my colleague mentioned, I will be across, uh, the hall at Neo4j booth, uh, to answer questions, if any.

95. 20:09

    I would love to interact with you and see if you have any quest- uh, any questions, and I can answer those questions. Thank you for your time. [audience applauding] [upbeat music]