Practical GraphRAG: Making LLMs Smarter with Knowledge Graphs

Talk transcript

1. 0:00

   [upbeat music] We are talking about GraphRAG today.

2. 0:16

   That's the GraphRAG track, of course. Uh, and we want to look at patterns for successful GraphRAG applications, uh, for, um, making LLMs a little bit smarter by putting knowledge graphs into picture.

3. 0:27

   My name is Michael Hunger. I'm VP at-- of Product Innovation at Neo4j.

4. 0:32

   My name is Stephen Chin. I lead the Developer Relations at Neo4j, and, um, actually, we're, we're both co-authoring. This is fun-

5. 0:40

   Yeah

6. 0:40

   ... 'cause we're both already authors, and finally, we've been friends for years, and we-

7. 0:43

   Yeah

8. 0:43

   ... finally get to co-author a book. We're co-authoring GraphRAG: The Definitive Guide for O'Reilly. So-

9. 0:49

   Yeah

10. 0:50

    ... basically, we didn't sleep this past weekend- [laughs] ... 'cause we have a book deadline.

11. 0:53

    Mm-hmm. Yep.

12. 0:55

    So, um, I'm gonna talk a little bit about kind of at a high level what GraphRAG is, why it's important, what we're seeing in the media, and then Michael's gonna drill down into all of the details and patterns, and give you a bunch of takeaways and things you can do.

13. 1:09

    This is probably if, if you wanna know how to do GraphRAG, Michael's quick- [laughs] ... dive, deep dive on this is the best introduction you can get, so I'm also excited.

14. 1:19

    Awesome. Let's get going.

15. 1:20

    Okay. So the case for GraphRAG is where we're gonna start, and the challenge with using LLMs and using other patterns for this is basically they, they don't have the enterprise domain knowledge.

16. 1:33

    They don't verify or explain the answers, they're subject to hallucinations, um, and they've ethical and data bias concerns. And you can see that very much like our, our friendly parrot here, um, they are all the things which parrots behave and act like, except a cute bird.

17. 1:51

    So we wanna do better than this with GraphRAG and figure out how we can use domain-specific knowledge, accurate, contextual, and explainable answers. And really, I think, like, what a lot of companies and what the industry is figuring out is it's really a data problem.

18. 2:06

    You, you need good data, you need to have data you can power your system with. Um, one of the patterns you can do this with is RAG. So you can stick your external data into a, into a RAG system.

19. 2:16

    You can get stuff back from a, um, a database for the pattern. But vector databases in RAG fall short because it's, it's lacking kind of your full data set.

20. 2:28

    It's l- it's only pulling back a fraction of the information by vector similarity algorithms. Typically, a lot of the, especially modern vector databases which everyone's using, they're, they're easy to get started with, but they're not robust, they're not mature.

21. 2:40

    They're not something which has scalability and fallback, and gives you that, what you need to get into build a strong, robust, um, enterprise system. And vector similarity is not the same as relevance.

22. 2:52

    So results you get back from using a basic RAG system, they're-- they give you back things which are related to the topic, but it's not complete, and it's typically also not very relevant.

23. 3:04

    And then it's very hard to explain what's coming out of the system. So we need an answer. Lifeline.

24. 3:11

    Yeah. GraphRAG.

25. 3:12

    GraphRAG. [laughs] And what GraphRAG is, is we're bringing the re- we're bringing the knowledge c- and the context and the environments to what LLMs are good at. So you can think of this kinda like the human brain.

26. 3:23

    Our, our, um, left brain is, um, our right brain is more creative, it does more like a, uh, like building things, it does more, um, extrapolation of information. Whereas our left brain is the logical part.

27. 3:35

    That's what actually has reasoning, has facts, and can enrich data, and it's built off of knowledge graphs. So a knowledge graph is a collection of nodes, relationships, and properties.

28. 3:47

    Here's a really simple example of a knowledge graph where you have two people, they, they live together, you have a car. But when you look into the details, it's actually like a little bit more complex than it seems at first because s- they, they both have a car, but the owner of the car is not the person

29. 4:03

    who drives it.

30. 4:03

    Yeah.

31. 4:04

    This, this is kinda like my family. [laughs] [laughs] My, my wife does all the bills, but then she hands me the keys whenever we get on the freeway. She, she hates driving.

32. 4:12

    So knowledge graphs also are a great way of getting really rich data. Um, here's an example of the Stack Overflow graph built into a knowledge graph, where you can see all of the rich metadata and the complexity of the results.

33. 4:24

    And we can use this to evolve RAG into a more complex system, basically GraphRAG, where we get better relevancy. We're getting b- more relevant results. We get more context because now we can actually pull back all of the related information by graph closeness algorithms.

34. 4:39

    We can explain what's going on because it's no longer just, um, vectors, it's no longer statistical probabilities coming out of a vector database. We actually have nodes, we have structure, we have semantics we can look at, and we can add in security and role-based access on top of this.

35. 4:54

    So it's context rich, it's grounded. This gives us a lot of power, and it gives us the ability to start explaining what we're doing, where now we can, we can visualize it, we can analyze it, and we can log all of this.

36. 5:06

    Now, um, this is one of the, the initial papers, the, the GraphRAG paper from Microsoft Research, where they went through this and they did, they showed that you could actually get not only better results, but less token cost.

37. 5:17

    It was actually less expensive to do a GraphRAG algorithm. Um, there have been a lot of papers since then which show all of the different research and interesting work which is going on in the GraphRAG area.

38. 5:30

    And, um, this is just a quick view of the different studies and results which are coming out. But even from the early Data.World study, where they showed a three times improvement in GraphRAG capabilities, and the analysts are even showing how GraphRAG is trending up.

39. 5:46

    So this is the, um, Gartner, um, kinda hype cycle from, from twenty twenty-four, and you can see generic AI is kind of, you know, on the downtrends. RAG is getting over the hump, but GraphRAG and a bunch of these things actually are providing and breathing more life into the AI ecosystem.

40. 6:03

    So a lot of great reports from, from Gartner showing that it's grounded in facts, it resolves hallucinations. Together, knowledge graphs and AI are solving these problems, and it's getting a lot of adoption by different industry leaders, by big organizations, um, who are taking advantage of this and actually producing production applications and making it work.

41. 6:22

    Like LinkedIn customer support, where they actually wrote this great research paper where they showed that using a knowledge graph for customer support scenarios actually gave them better results and allowed them to improve the, um, quality and reduce the response time for getting back to customers.

42. 6:40

    Um, median per issue resolution time was reduced by twenty-eight point six percent. I mentioned the Data.World study, which basically was a comparison of doing, um, RAG on SQL versus RAG on graph databases, and they showed a three times improvement in accuracy of LLM responses.

43. 6:56

    And let's chat about patterns, Michael, 'cause I think everyone's here to learn how to do this.

44. 7:01

    Exactly. So let's, let's look at how to do this actually, right? So, and, um, if you look at GraphRAG, uh, there are actually two sides to the coin. So one, of course, you don't start in a vacuum.

45. 7:12

    You have to create your knowledge graph, right? So-- And we see basically multiple steps to get there. Initially, you get unstructured information, you substructure it, you put it into a lexical graph, which represents documents, chunks, and their relationships.

46. 7:25

    In a second step, you can then extract entities using, for instance, LLMs with this graph schema to extract entities and their relationships from that graph. And in a third phase, you would enrich this graph, for instance, with graph algorithms doing things like, you know, page rank, uh, community summarization, and, and so on.

47. 7:42

    And then when you have this, uh, built up knowledge graph, then you do GraphRAG as the, as the search mechanism, um, either with local search or global search and, and, um, other ways, right?

48. 7:54

    So let's first look at the first phase of, like, knowledge graph construction a little bit. Um, so like always in data engineering, there's-- if you want to have higher quality outputs, you have to put in more effort at the beginning, right?

49. 8:06

    So it's basically nothing comes for free. There's no free lunch after all. But what you do at the beginning is basically paying off multiple times because what you get out of your unstructured documents is actually highly-- high quality, high structured information, which you then can use to extract contextual information for your, for your queries, which allows rich

50. 8:24

    retrieval at the end. Okay. And so a-after seeing, uh, GraphRAG being used, uh, by a number of users, customers, we've seen, uh, we looked at research papers. We, we saw that a number of patterns emerging, uh, in terms of like how we structure our graphs, how we query these graphs, and so on.

51. 8:43

    And so we started to collect these patterns and put them on graphrag.com. Um, and we want to-- I wanted to show what, what this looks like. So we have basically, uh, example graphs, uh, in the pattern.

52. 8:55

    The pattern has a name, description, uh, context, and we see also queries that are used for extracting this information, right? So for instance, here's an, uh, mix of a lexical graph and a domain graph, and then we can have the query that fetches, uh, this, uh, information.

53. 9:09

    Let's look at the three steps in a little bit more detail on the, um, on the graph model side. So on one side, we have, uh, for lexical graphs, you represent documents and their elements.

54. 9:19

    So that could be something as simple as a chunk. But if you have structured element, uh, documents, you can also do something like, okay, I have a book which has chapters, which have, uh, sections, which have paragraphs, where the paragraph is a semantically cohesive unit that you would use to, for instance, create a vector embedding of-- that

55. 9:35

    you can use, uh, later for vector search. But what's really inter-interesting in the graph is basically you can connect these things all up, right? So you know exa-exactly who's the predecessor, who's the successor to a chunk, who's the parent of an element.

56. 9:47

    And using something like, uh, vector or text similarity, you can also connect these, uh, chunks as well by, uh, an k-nearest neighbor or similarity graph, where you basically store similarities, uh, between chunks, and then you put on the relationship between them and, and, and weighted score, basically how similarity two chunks.

57. 10:04

    And then you can use all these relationships when you extract the context in the retrieval phase to, for instance, find what are related chunks by document, by, uh, temporal sequence, by similarity and other things, right?

58. 10:16

    So that's on the, on the lexical side. Um, this looks like this. So for instance, you have an, uh, RFP and you want to break it up in a structured way.

59. 10:24

    Then you basically create the relationships between, uh, these chunks, uh, or the, the, these, uh, subsections, uh, add the text, do the vector embeddings, and then you do it at scale, and then you get a full, uh, lexical, uh, gra-graph out of that.

60. 10:38

    Next phase is entity extraction, uh, which is also something that has been around for quite some time with NLP. But LLMs actually take this to the next level with their multi-language understanding, with their high flexibility, good language skills for extraction.

61. 10:52

    So you basically provide an, uh, graph schema and an, um, instruction prompt to the LLM, plus your pieces of information, pieces of text. Now with large context windows, you can then p-put in ten thousand, a hundred thousand tokens for extraction.

62. 11:09

    If you have, you can also put in already existing ground truth. So for instance, if you have exis-existing structured data where your entities, let's say products or genes or partners or clients are already existing, then you can also pass this in as part of the prompt so that the LLM doesn't do an extraction, but more an recognition

63. 11:28

    and, and finding, um, approach. There you find your entities, and then you extract relationships, uh, from them, and then you can store additional facts and, and, uh, additional information into store, uh, as part of relationships and entities as well.

64. 11:41

    So basically, in the first part, you have the lexical graph, which is representing document structure. But in the second part, you extract the relevant entities and their relationships. If you have already an existing knowledge graph, you can also connect this to an existing knowledge graph.

65. 11:53

    So imagine you have an, um, CRM where you already have customer clients, uh, and, and leads in your knowledge graph, but then you want to enrich this with, for instance, uh, protocol from call transcripts, and then you basically connect this to your existing structured data as well.

66. 12:08

    So that's also a possibility. And then in the next phase, what you can do is you can run graph algorithms for enrichment, which then, for instance, can do clustering on the entity graph, and then you generate, uh, something like, uh, communities where, uh, an LLM can generate summaries, uh, across them as such.

67. 12:26

    Right. And, uh, for especially last one, it's interesting because what you identify is actually cross-document, uh, topics, right? So because it's basically each sh- document is in basically temporal, uh, v- vertical representation of information.

68. 12:40

    But what this is, is actually it looks at which topics are reoccurring across many d- different documents. So you find these kind of topic clusters a- across, uh, documents as well.

69. 12:50

    Cool. So if you look at the, the second phase, the search phase, which is basically the retrieval, uh, um, part of RAG, what we see here is basically that in a graph retriever, you don't just do a simple vector lookup, uh, to get, uh, returns, uh, results return.

70. 13:06

    But what you do, you do an i- initial index search. It could be vector search, full text search, hy- hybrid search, spatial search, rather kinds of searches to find the entry points in your graph.

71. 13:15

    And then you basically, uh, can take, as you can see here, um, starting from these entry points, you then follow the relationships up to a certain degree or up to a certain relevancy to f- f- fetch in, uh, additional context.

72. 13:28

    And this context b- can be coming from the user question, it can be external user context that comes in. For instance, when someone from, let's say, your, uh, finance department is looking at your data, you return different information than if someone from the, let's say, engineering department is, is looking at your data, right?

73. 13:44

    So it also takes this external context into account, how much and which context you retrieve. And then you return to the LLM to generate the answer, not just basically text fragments like you would do in vector search, but you also create, uh, return these, um, more complete, uh, subset of the, of the contextual graph, uh, to the

74. 14:04

    LLM as well. And modern LLMs are actually more trained on, uh, graph processing as well, so they can actually deal with these, uh, additional pattern structures where you have, uh, node relationship, node patterns, uh, that you provide as additional context, uh, to the LLM.

75. 14:20

    Um, and then of course, I mentioned that you can enrich it using graph algorithms. So basically, you can s- do things like, uh, clustering, link prediction, PageRank, and other things to enrich your data.

76. 14:30

    Cool. Let's look at some, uh, practical examples. We don't have too much time left. Uh, so one is, uh, knowledge graph construction from unstructured sources. So there's a number of libraries.

77. 14:39

    Uh, you've already heard some, uh, today from people that, uh, do these kind of things. Um, so one thing that we built is an, a tool that allows you to take PDFs, uh, YouTube, uh, transcripts, uh, local documents, web articles, Wikipedia articles, and it extracts your, uh, data into a, an, a graph.

78. 14:59

    And let me just switch over to the, to the, uh, demo here. Uh, so this is the, this is the tool. Uh, so I, uh, uploaded, uh, information from different Wikipedia pages, YouTube videos, articles, and so on.

79. 15:13

    And here's, for instance, an Google DeepMind, uh, extraction. So you can use a lot of different LLMs here, and then you can also, if you want to, in graph enhancement, provide a graph schema as well.

80. 15:24

    So you can, for instance, say a person, uh, works for, uh, a company and, uh, add these patterns, uh, to your, um, to your schema, and then the LLM is using this information to drive the extraction, uh, as well.

81. 15:38

    And so if you look at the data that has been extracted from, uh, DeepMind, it is this one here. We can actually see

82. 15:48

    from the Wikipedia article, um, two aspects. One is the document with the chunks, which is this, uh, part of the, of the graph, right? And then the second part is the entities that have been extracted from, from this, uh, article as well.

83. 16:01

    So you see actually the, uh, connected knowledge graph of entities, which are companies, locations, people, and technologies. So it followed our, um, followed our schema to extract this. And then if I want to run GraphRAG, you have here a number of different retrievers.

84. 16:15

    So we have vector retriever, graph and full text, entity retrievers, and others, uh, that you can select. Uh, all of this is also an open source project, so you can just go to GitHub and have a look at this.

85. 16:25

    And so I just ran this before because internet is not so reliable here. So what has DeepMind worked on? And I get a detailed explanation. And then if I want to, I can here look at, uh, details.

86. 16:36

    So it shows me which sources did it use. AlphaFold, Google DeepMind, Wikipedia, another PDF. I see which chunks have been used, which is basically the full text and hybrid search.

87. 16:45

    But then I also see which entities have been used from the graph. So I can actually really see from an explainability perspective, these are the entities that have been retrieved by the GraphRAG retriever passed to the LLM in addition, uh, to the text that's connected to these entities, so it gets a richer response, uh, as such.

88. 17:01

    And then you can also do eval on that with the Ferragas as well.

89. 17:06

    Um, so while I'm on the screen, uh, let me just show you another thing, uh, that we worked on, which is more like an agentic approach, where you basically put these individual retrievers into, uh, an, an configuration where you have basically domain-specific retrievers, uh, that, uh, are, um, running individual separate queries.

90. 17:26

    So for instance, if you look at, uh, let's say this one, it has, uh, the query here and basically a tool with inputs and a description. And then you can have an agentic, um, loop using these tools, basically doing, uh, graphic with each individual tool, taking the responses, and then doing, uh, deeper, uh, tool calls.

91. 17:46

    Uh, I'll show you a deeper example in a, in a minute. So this is basically what I showed you. This is all available as, uh, open source, uh, libraries.

92. 17:54

    You can use it yourself in, from Python as well. Uh, I showed, um, Neo Converse, which was also able not to just output text, but also, uh, charts and other visualizations, uh, networks, uh, visualizations as well.

93. 18:07

    And what's interesting here in the agentic approach, you don't j- just use vector search to retrieve your data, but you basically break down the user question into individual tasks and extract parameters and run these individual tools, um, which then are either run in sequence or in a loop to, uh, return the data, and then you get basically

94. 18:25

    these, uh, outputs back. And, uh, basically for each of these things, div- individual tools are called and, and used here. And the last thing that I want to show is the, uh, GraphRAG Python package, uh, which is basically also encapsulating, uh, all of this in construction and the retrieval in, into one package.

95. 18:43

    So you can build your knowledge graph, you can implement the retrievers and create the pipelines here. And here's an example of where I pass in, uh, PDFs plus a graph schema, and then basically, uh, it runs, uh, the import into Neo4j, and then I can, uh, in the Python notebook visualize, uh, the data later on.

96. 19:03

    And with that, I leave you with, one second,

97. 19:07

    uh, the takeaway, which is on GraphRAG.com you find all of these resources, a lot of the patterns, and, uh, we'd love to have contributions and love to talk more.

98. 19:17

    I'm outside at the, at the booth if you have more questions.

99. 19:21

    Yeah.

100. 19:21

     Thank you so much.

101. 19:22

     No, that was great, and I think you're getting it all from the expert with all the tooling. Actually, Michael's team builds a lot of the tools like Knowledge Graph Builder.

102. 19:29

     Um, very excited you all came to the GraphRAG track and hope to chat with you all more. If you have questions for me and Michael, just meet us in the Neo4j booth across the way.

103. 19:38

     Thank you.

104. 19:38

     Thank you. [audience applauding] [upbeat music]