GraphRAG methods to create optimized LLM context windows for Retrieval — Jonathan Larson, Microsoft

Talk transcript

1. 0:00

   [electronic music] Um, well hello everyone.

2. 0:16

   Um, thanks for coming to the session here today. My name's Jonathan Larson, and, uh, I run the GraphRAG, uh, team at Microsoft Research. Um, some of you may have seen our paper that we actually, uh, released last year, uh, the GraphRAG paper, or you m- uh, might have seen our, uh, GitHub repo, uh, that was out there

3. 0:33

   and, uh, has a lot of stars on it. Um, when we released this last year, to our surprise, it got a lot of attention, and also it inspired many other offerings that we saw that were out there as well too.

4. 0:43

   Um, uh, and my favorites, of course, were the ones that Neo4j did with the card game, because I'm a huge bo- uh, board game fanatic as well too. Uh, but, uh, there's been lots of derivative work from it.

5. 0:51

   And today, though, I'm not here to talk to you about some of the things that have been. I'm here to talk to you about some of the new horizons.

6. 0:58

   And the th- two things I really want to leave you with here today are that, uh, LLM memory with structure is just an absolutely key enabler for, uh, building, uh, effective AI applications.

7. 1:09

   The second, as we'll talk about a little bit later in the talk as well too, are that, uh, agents, of course, paired with these structures can provide something even more powerful.

8. 1:17

   And let me go ahead and go back to this slide here. So today I'm gonna tell you about a couple things. The first is actually showing you GraphRAG as applied to a specific domain.

9. 1:25

   In this case, we're gonna apply it to the coding domain to look at enterprise productivity, uh, in, in the coding space. The second is we're gonna take a look at a new release that we are announcing today.

10. 1:35

    Uh, we're actually having a blog post on this tomorrow. Uh, we have BenchmarkQED just went open source, uh, as well. And then I'll talk about some, uh, new results that we've had on, uh, some GraphRAG evolutions on a new, uh, technology we've been working on called LazyGraphRAG.

11. 1:48

    I won't be going into the specifics of how any it works. I'll just be showing some benchmarks and talking about some of the ways you c- be able to access it soon.

12. 1:55

    So with that, let me go ahead and jump into GraphRAG for code and tell you about how we've been using GraphRAG to actually help, uh, drive repository-level understanding. And I wanna start first with a little demonstration.

13. 2:06

    Uh, everyone likes to watch little videos, so I'm gonna go ahead and hit play. And yes, it's showing on the screen there. So this is a little, uh, terminal-based video game that one of my engineers have put together.

14. 2:15

    It's a one where you jump over obstacles, and then you get points when you jump over them. When you run into them, you lose points. And two important things, though.

15. 2:22

    First, the LLM has never seen this code before, and it's small enough for the human to know the ground truth holistically 'cause it's only about 200 lines of code across seven files.

16. 2:31

    But it's complex enough that the LLMs had a lot of troubles understanding it if you just provide all the code directly into the context window. So with that as a background, let's take a look at what happens if you use typical regular RAG over the top of this.

17. 2:43

    So if you're using, uh, one of the tools that help analyze your code, this is the type of answer you might get back. And this is... We ran this, uh, through with a regular RAG system, and we asked it, "Describe what this application is and how it works."

18. 2:56

    And just to read you, 'cause the, what it says, 'cause it's too small to read, "The application is designed as a game that is configured and initiated through a main function.

19. 3:03

    The game leverages a configuration file and has its main components, such as the game screen, game logic, encapsulated in separate classes and functions." Totally useless. It just says it's a game.

20. 3:13

    That's... There's nothing more [chuckles] to it than that. Um, and it just put a bunch of other, uh, cruft in there. If you use GraphRAG for code over the exact same code base with the exact same question, you get a much, much better description.

21. 3:26

    This is just a very stark contrast you get between the two. So in the GraphRAG for code, it says, "The application is a terminal-based interactive game designed to run with a, in a curses-based terminal environment."

22. 3:37

    So far, not much better. But this next line just kills it. "It features a player character that can jump vertically, obstacles that move horizontally across the screen, and a static background layer.

23. 3:48

    The game controls via keyboard input, specifically using the Spacebar to trigger the player's jump action." So what this is showing you is semantic understanding. So if you've, um, read our paper or used any of the, um, GraphRAG code base that we've, uh, put out there, you've, you'll know about the concept of what we call local and global

24. 4:05

    queries. And so this is what we would call a global query over top of the repository. It requires understanding the whole repository to answer that question correctly. And we can see that it excels at that.

25. 4:16

    So one of the next things we decided to do is, well, if it can answer questions pretty well, can it maybe do code translation? And so that was the next thing that we, uh, aimed it at.

26. 4:25

    So taking the same code base here, we took on the challenge of taking this Python code and then asking to translate it directly into Rust. And let me show you just how that actually works.

27. 4:34

    I'm gonna play the video here. So what we're gonna do is start off with, um, in VS Code here. We're gonna look at the four source code file. Well, there's four main ones.

28. 4:41

    There's seven files total for this game. These are the source Python code, uh, files that we're working with here. We're gonna go ahead and run those just to show you again.

29. 4:49

    This is the same game. So this is again the same side-scroller game here. And then what we're gonna do is we're gonna actually take those source code files and just put them straight into an LLM.

30. 4:57

    We're first not gonna use GraphRAG for code. We're just gonna take all that source code, 'cause it's only 200 lines of code, and put in some nice prompts. We tried a wide variety of these prompts and then try and see if the LLM could actually translate that code holistically in all of its pieces into Rust and actually

31. 5:12

    just work out of the box. And so this is actually the translation happening there. Um, we're gonna go ahead and copy the Rust code that it generated. So it did generate Rust code.

32. 5:19

    I got to gr- give it credit for that. But after it generated the code and we reviewed it there, we went and we tried to compile it, and there's problems everywhere.

33. 5:27

    Doesn't work out of the box. Uh, so then we used GraphRAG for code, again bringing structure and these, uh, these graph structures to the code base there. We're gonna just go ahead and run the translate function there on GraphRAG for code, and it generates, on its side, all these s- all these new Rust files that you see

34. 5:44

    are on the side there. I'm just gonna skip past here 'cause I don't y- wanna bore you with clicking through a bunch of Rust files. But once we generated all these Rust files, we can go ahead and run those Rust files on GraphRAG for code side of things, and presto, we have a full video game completely translated

35. 5:59

    from Python, uh, now working in Rust natively. But we didn't wanna stop there. The next thing that we wanted to do was, well... That game is kind of a toy example.

36. 6:09

    It's like 200 lines of code. Let's go to a code base with 100,000 lines of code, and that's what we did next. So we went to the Doom code base.

37. 6:16

    It's about 30 years old. Now, we did run a bunch of tests over the top of this here first, uh, because I figured, well, all the LLMs are trained on the Doom code base.

38. 6:24

    I mean, it's, it's gonna just know these things on, um, natively and automatically. So we did a bunch of tests with that. We actually figured out, while it knew the Doom co- Doom code base, it didn't know any of the specifics.

39. 6:34

    So if you ask it to actually modify the Doom code base in any sort of meaningful way, the LLM models just fail again completely. Um, so then what we did next was, well, if we can, uh, reason and understand over top of this code base, 100,000 lines of code, 231 files, um, maybe we can generate some new

40. 6:50

    outputs that you otherwise couldn't generate. So with that, we then immediately had it start- used it to start generating documentation. So this is actually showing a high-level repository-level documentation.

41. 7:01

    Again, going back to the concepts of GraphRAG, we have local and we have global queries. This is showing you the global query results. And so it's not looking at the understanding of a single file.

42. 7:10

    This is looking at modules in their entirety, like across 20, 30 files, and being able to actually give you a sense for, like, what's the sound system inside of the video game, and how does that work?

43. 7:20

    But then you can still drill down into what we would call the local-style queries and actually see the individual files as well too. So then we thought, "Well, this is kind of neat.

44. 7:28

    We can look at the documentation. But if we can look at the documentation, we can do Q&A, and we can do code translation, can we maybe take this one step further, do feature development?"

45. 7:39

    So that's the next thing I'm gonna show you. So, um, we had a lot of, uh, video game players in, uh, our office, and we were kind of thinking about, like, what would be a cool thing we put over top of this Doom video game?

46. 7:48

    And somebody to me said, "Well, doesn't... You can't jump in the original video game. What if we w- added the ability for a player to jump?" And that's a complex thing to add into Doom because it requires multi-file modification.

47. 7:59

    And if you tried to use AI systems to do multi-file modification, what it'll often do is it will do a great job editing one of the files and then completely break a bunch of the other files in the process of doing so.

48. 8:09

    And then it, that just pr- rinses and repeats and continually [laughs] kind of, uh... You end up with something that doesn't work is what, what you oftentimes run into, and it's because of this lack of understanding how everything fits together.

49. 8:20

    But again, this is where GraphRAG can really help out. So with that, we went ahead and this is... If you haven't seen the Doom video game, this is actually a little video of the original Doom video game, uh, being played, just to sh- give you some context for what it was.

50. 8:34

    Then we used, uh, the new, um, GitHub Copilot coding agent that was announced at Build, and we wired it up directly to a GraphRAG for Code. So we're gonna go ahead and s- create a new issue.

51. 8:43

    This is pointing to the Doom code base. And we're gonna tell it, "Add jump capability to the player." We're just gonna add a couple sentences of description here and then tell it to go.

52. 8:52

    And then we're gonna assign it to that GitHub Copilot coding agent.

53. 8:58

    And then the next thing we're gonna do is we're gonna then go ahead and look at what's happening underneath the hood. So this is actually the GitHub Copilot coding agent reaching out to GraphRAG for Code on the back end, coming up with a plan that's holistic, and it's approaching it from the top down.

54. 9:12

    And then this is the really the mo- the aha, aha moment we had for us. It changed a whole lot of files, and it worked out of the box, where all the other agents that we had tried completely failed on this task.

55. 9:22

    And because it worked because of those GraphRAG structures that we were bringing to bear and actually using those, uh, with that. So then the end result was we were elated with joy and jumping in Doom specifically.

56. 9:34

    So, uh, that's [laughs] kind of the story of how we got to with that. So with that, um, I do, uh, have another part of the talk I'm gonna talk about today, and that's also, um, uh, five minutes to talk about that one, and that's BenchmarkQED.

57. 9:47

    Uh, just shifting topics a little bit, I just showed you GraphRAG as applied to one vertical. Uh, next I wanna spe- specifically talk about, um, how do we measure and evaluate systems like GraphRAG?

58. 9:56

    How do we build systems to evaluate those local and global quality metrics? And so for that, today I'm announcing BenchmarkQED. This is available now open source on GitHub, so you can go ahead and, uh, check it out at the link.

59. 10:09

    We just, I think, got it live last night. Um, and there's three components to this. The first is what we're calling AutoQ, and AutoQ is focused on doing query generation for target datasets.

60. 10:21

    So it allows you to take a dataset and then generate queries for it. AutoE is the evaluation using LLM as a judge to actually then evaluate how those queries performed on that said dataset.

61. 10:32

    AutoD is the third component, and that really focuses on dataset summarization and dataset sampling. Let's jump into just a couple of these in a little more detail. So if you take a look at AutoQ, AutoQ is taking a look at the local and global aspects.

62. 10:46

    You can see that there on the X-axis. On the Y-axis, it's then combining that against data-driven basic questions that are generated based on the data itself or persona or activity-driven, which is the second type that we have there.

63. 10:59

    Those are more complex, like if you take on the role of a person in that domain field and then using that to generate questions. So to show you a few of these sample questions, I'm just gonna choose a couple here.

64. 11:09

    This was built on a AP News dataset that was focused on, like, medical type of events. And so a data local question might be, "Why are junior doctors in South Korea striking in February 2024?"

65. 11:20

    There's a lot of highly specific information in this, and I would expect regular RAG to perform very well on this type of question. In contrast, take an activity global question here.

66. 11:30

    It might be, "What are the main public health initiatives mentioned that target underserved communities?" There is nothing in that question that you can really pivot on in terms of, like, embedding or, uh, indexing that would really work on it.

67. 11:41

    You really have to holistically know the entire dataset. And so this, uh, AutoQ will help generate questions across that whole spectrum, from local to global, from data-driven to activity-driven, and give you these categories of questions that you can use.

68. 11:55

    Now, once you generate those questions, we can then start measuring them using AutoE, which is the evaluation platform that we released along with, um, this as well too. So just to get to show you how this in, in practice and how it works, it c- use...

69. 12:07

    It gives you a composite score of four metrics. The first is comprehensiveness Diversity and empowerment, which if you're familiar with our paper, those are the three original ones that we used back in that one, and a new one that we also call relevance.

70. 12:19

    And so we actually did some comparisons, uh, just to show you how this works on Lazy GraphRAG, which is one of the newer technologies that we've been working on, and we compared it to Vector RAG on 8K, 120K, and million-token context windows, and we had a few takeaways from that.

71. 12:33

    The first is if you take a look at these charts, any bar that is above the 50% mark, uh, gives, uh... means that Lazy GraphRAG is winning in that benchmark against those specific Vector RAGs.

72. 12:43

    So Vector RAG blue here is 8K, 120K, and a million-token context windows, and it's winning at, correspondingly at 92%, 90 and 91% of the time against data local questions, which is kind of a surprise because one of the first things that we started seeing is that Lazy GraphRAG was actually prov- providing dominant performance, um, across the entire

73. 13:01

    span of questions, whether they were local or global. And we do expect RAG to be performing better on local questions, and you can see that from the fact that data local and the data global, there's a little bit of a lift between these bars between the global and the, and the local ones.

74. 13:15

    Second thing we noticed is that the long context window didn't really make much of a difference. We were expecting that maybe the long context windows might give you a little bit of a, um...

75. 13:23

    'cause you have a better understanding towards those global types of questions. But it actually turns out that Lazy GraphRAG in this case was still able to dominate those metrics, and in fact when it, when we ran the test, it's not on the slide here, we actually found that Lazy GraphRAG was a tenth of the cost of what

76. 13:37

    we saw in the one-million-token context windows as well too. So those are a couple things to show there. Now, uh, with the last minute that I have, I did want to address another thing too with Lazy GraphRAG.

77. 13:48

    Um, you may have read the blog posts about it in November last year when we first discussed it. Um, it is now, uh, officially being lined up for launch in a couple products.

78. 13:56

    Uh, the first is Azure Local, and so, uh, it will... they just announced at Build that, uh, Lazy GraphRAG will soon be incorporated into their platform for that, so you can actually try it out for yourselves there.

79. 14:06

    And it's also being incorporated into the recently announced at Build Microsoft Discovery, uh, platform tool as well. So if you're not familiar with Microsoft Discovery, it does graph-based scientific co-reasoning, and just to give you a quick, uh, summary of it, it goes from hypothesis to experiment, learning, and knowledge.

80. 14:23

    I'll just play a quick like five-second video here, and the questions... the, the answers to the questions that you start coming back to with on this are powered underneath the hood by GraphRAG and Lazy GraphRAG as well too.

81. 14:32

    So as you can see, the copilot here is generating the deep reasoning over graph-based scientific knowledge. Those graphs are being powered by, uh, um, GraphRAG and Lazy GraphRAG. So with that, I just wanna leave you with a couple takeaways, and the first is just that LLM memory structure, uh, [laughs] LLM memory with structure is just a really, really

82. 14:51

    powerful tool, uh, to keep in your tool belt. Um, and the agents, of course, can massively amplify this power. And if you have any questions, I'll be outside if you, uh, want to talk about any of this further.

83. 15:01

    Thank you so much for your time. [outro music]