Agentic GraphRAG: AI’s Logical Edge

[on-hold music] My name's Stephen Chin.

I run the developer relations team at Neo4j. Um, actually, I just started writing a, um, a new book on GraphRAG with O'Reilly, so that was my weekend was writing chapters.

Um, first chapter done, so I think, I think we'll actually put, like, the pre-release version of it up soon. Um, and what I am gonna talk about for the next ten minutes is agentic GraphRAG and a little bit about how you can accomplish this.

Okay? So, um, first of all, who, who here is familiar with graph databases?

Okay, that's really good. Okay, you are, you are well above and ahead of the curve. Um, how many folks have, have given, um, GraphRAG a try to, to build systems using graphs?

Okay, so one guy in the back, he's the expert, if you have any questions [laughs]

ask him. And just to set the context on, like, like, why is, um, we, we need... We basically have a problem with a lot of agentic systems where they're not meeting use cases.

This was, you know, Gartner's prediction of doom and utter failure. Um, and they have a lot of hallucinations in them. And so, um, this is an example. I'm like-- I'm gonna kind of breeze through this a little bit, but I basically asked the OpenAI o3 reasoning API to solve, like, a, a question about biases, reasoning, and math.

And the, the basic problem is it, it doesn't do a good job if you don't give it enough deep information sources of answering things correctly. And, um, the, the question is like how many girls you can fit in a classroom.

So, like, it's a tech computer science bias. It's a little bit about, like, math and reasoning 'cause I ask it about a grid. It, it, it inaccurately anchors, um...

These are the grid sizes which you could choose and the number of girls you can fit. It inaccurately anchors on some article about a, um, um, non-attacking kings problem, assumes the square's-- it's a square, like grid.

So, like, the audience, you guys always win against the reasoning AI, even though it takes forty seconds to, to noodle on this. And it, it further... So I ask, like, um, are the, are the girls and boys gonna go to home economics or to, um, um, whatchamacallit, to, to sports?

And again, it-- this is both a bias and also I misled it by giving information about the ratio of girls and boys, and it comes up with an answer calculating that all the girls will go to cooking class.

Which is, which is horrible. I mean, I, I'm the chef at home and I, I love cooking. Um, so I think that these, these are the biases. Now, this is kind of funny because we can reason about the situation and it's like, it's a problem that we can think about.

But imagine if this was, like, in life sciences about drug discovery or if you're sup-- or if you're solving a supply chain issue. So the fact the LLM has, has gone and inserted biases, it's done incorrect reasoning along the way, this means you're gonna get the wrong business results.

And it's very hard to figure this out. So basically, the, the problem is the, the LLM is good at extrapolating information, like, like doing language tasks, figuring out things, and it, it gives the impression of intelligence where there's, there's no real intelligence.

There's no real kind of human reasoning behind it. And so we, we over-ascribe things it can do, and there's a bunch of things it can't do well. Now, those, those are things that knowledge graphs are actually really good at.

So one way we can solve this problem is by throwing more computers at it, more agents, right? And, um, agentic systems are good at improving the quality of results because you have LLMs talking to each other and reasoning with each other about the problem.

So, like, you know, basically you have agents who observe, they think, and they take actions. So you have different types of agents which are doing different things in the workflow.

Um, an agentic runtime might look something like this, where you, you're building out and you have a, um, orchestration layer which is, is working on the agents. You have some gen AI models hooked up.

You have some tools. Um, and then these all collaborate to give you better results than one LLM to give you to-- can give you together. Now, the challenge with this is that it's a very monolithic architecture.

It's hard to maintain. It's hard to swap out the tools. And it also kind of puts you in a situation where you can't secure the system, you can't do a bunch of things.

So a good way to solve this is using MCP.

You kind of use MCP as your tools where your agents are talking to them. Um, with MCP now, you have your servers and you have your data sources which are now talking to each other.

Um, you have your client and server, so now you can give it, um, you know, files or database records. You can give it a graph database as the system of record.

And we built a bunch of tools at Neo4j on top of MCP. So we built a, um, a Cypher tool, which Cypher is the query language for graph databases.

Um, so what basically it'll, it'll use, it'll-- When you ask the MCP server, it gives capabilities to generate Cypher queries off of prompts or questions or the things you want to pass it.

We have a memory module, so this gives you some agent memory you can use to plug into agentic systems. And then we also have, um, uh, MCP on top of our cloud APIs if you wanna provision databases or do different things on top of it.

And I think this is a pattern you'll see with a lot of people who are the vendors who are building things, is now you can plug these tools into your agent architecture and you can use them together with your, your graph.

Typically, like, agents are represented in some sort of graph. This is a picture of a LangGraph agent. And you can layer memory on top of it. So these are all folks who just spoke in our panel.

So Zep, Cogni, Memzero were all talking about their approach to agents. Um, actually, they all run on top of Neo4j. Um, so they use Neo4j as the core graph database as, uh, some of them are pluggable, so you can choose your graph database of choice.

But they're a really good way of giving memory to your agents, which is graph based and matches the way LLMs want to store, communicate, and retrieve information. And, um, also, um, one of our other speakers on the GraphRAG track showed his architecture for doing video search and summarization, and if you notice, um, they, they do a bunch

of this already, right? So they, they do short-term memory, they have, um, a whole bunch of lookup information, and they give you a choice of, they, they have both a GraphRAG pipeline and a vector pipe.

So I, I highlighted the GraphRAG in, in red, and the reason they're doing this is because when you use GraphRAG, you get some advantages in terms of the results coming back, and typically a lower rate of hallucinations.

This would be your, like, direct LLMs are kind of, you know, they give you very generic responses to a healthcare question. Um, baseline RAG, you get better results back, but it's, it's incomplete because it's doing, um, [lip smack] basically it's doing vector similarity.

So similarity is not relevance. It, it doesn't mean it actually understands the problem. Um, this would be a system which does GraphRAG, and the, the typical pattern that would get you this is, um, first do your search in a, in a vector search.

You could use a vector database. We also support vector search on top of Neo4j, and it gives you back results, and that's a good way of translating the question into, like, vectors.

And then you have mappings from the vector embeddings to your graph, and you f- you pull back the nodes which are relevant. So in this case, like you're asking about a, um, emphysema, you'd get back the, the node for emphysema.

You'd pull back all the related nodes and diagnosis and conditions, and you see it's just, it lists them all, right? So it's a very effective way to get really good responses back when you're dealing with something where you, you kind of have this mix of structured and unstructured data.

Here's a quick architecture of how you could put this together using, you know, you're using traversal and vector similarity. Um, you take in the question, you do the query against either vectors or knowledge graphs.

Um, graph data science or graph analytics are helpful as well to do community algorithms and groupings and things like this. Um, some of this is in the Microsoft GraphRAG paper and other research which is going on in this area.

And, um, you feed that back as context to the LLM to improve the quality of the answers. Um, some of the patterns I've talked about quickly, so text to Cypher is what our MCP server does.

It's, it can be good, but sometimes it can be really bad because the generation of Cypher by LLMs is not as good as you need it for some cases.

Um, the one I was talking about is, is, um, which one was it? Um, ba- basically vector search with graph context, right? So you do a vector search, and then you use that to pull back related nodes and graph context.

That's a really good pattern to start with, and, um, you can also do pre and post filtering of vector results to bubble things which are more relevant to the top of the context.

Um, certain systems, this is quite good as well because all you wanna do is make sure the LLM gets things higher up in the buffer for context windows. Even if LLMs now have larger context windows, basically what, what the eval- evals show is they ignore most of it, and they look at the stuff at the top.

Okay, so a quick example of a company which is doing this. So Klarna basically replaced all of their SaaS systems with a gr- GraphRAG project. Um, they're one of our customers.

Um, they took an enterprise wiki's HR system's internal documentation, 250K employee questions asked in the first year, 2,000 daily queries processed, and 85% employee adoption. So like a really good adoption of this technology.

Um, and I'll give you a couple resources, and I think we're at time. Is that about right, crew?

You have five minutes.

Oh, okay.

You still have five minutes.

Oh, I see there's five minutes between sessions. I was, I was rushing to do this even quicker. Okay, so we have time for questions, which is great. Um, so one resource I'd recommend is the, um, Neo4j Certified Developer Program.

Um, so with the number of hands [laughs] in the room here for folks who said they know graphs, I'm pretty sure you could all pass the Neo4j certified exam if you just took it today.

And we basically will, will mail you a Neo4j certified T-shirt. You get a little LinkedIn badge to put on your profile, and it's a nice way to just show the world that, like, you, you actually know this stuff, like, you know graph technology, you know stuff.

We'll probably add in additional certifications for like, you know, GraphRAG and other stuff in the future. But the base certified developer class is a good way just to get base knowledge in this, and we do have classes in Graph Academy on building chatbots, using LLMs, using all of this stuff as well.

Um, second resource is our Nodes Conference. So the Neo4j Nodes Conference is an annual conference. It runs in three different time zones, 24 hours, all free content, free sessions.

You know, come, come out and join and check out some of this content.

Okay, and I'll let people finish who want to do the QR code. And thank you very much for coming, and we have a few minutes for questions. Okay, so what we'll do is anyone who has questions, just raise your hand and shout it out.

If anyone wants to leave the room, feel free to do that as well. I don't wanna keep you trapped in here. Okay, so in the back.

To be clear on the architecture, user submit query, you embed that, do a sem search, get the nearest node, and grab related nodes. Is that the correct kind of basic-

Yeah, yeah. Okay, so the question is, like, what the pattern is for-

Yeah

... for doing this type of search, and that is exactly right. So, so basically, what you're doing is you're, you're using the LLM for what it's good at, which is language translation.

So the user can enter whatever convoluted question they want, which would never translate to a beautiful Cypher query, and then you, you first tell the LLM, "Well, do a vector search on that.

Like, find vector similarity in this." And then because you've generated like the graph and the embeddings to point to each other, now you can go to the graph and you can say, "Well, this, these em- th- these embeddings all point to this node in the graph, so it's probably about, in the, in the case, emphysema."

Now, I wanna pull back the nodes which are either, like you could use cosine similarity or you could use, um, um, community grouping algorithms or different algorithms to figure out what's relevant and then pass that as context.

In a simple architecture, the chunks are the same. What? The chunks you embed in your nodes are the same pieces Yeah. So, so like what, what we typically do, a- and this is what'll happen if you, um, import unstructured data into Neo4j, we'll...

We, we, we can create a node con- structure out of it using LLMs, and then you hang your embeddings, your text embeddings of, as properties off of the nodes.

And who does that? Sorry. Who does that association? Um, so we have a couple plugins for this. We have a Neo4j Python library- Okay ... which that has-- will do a lot of this.

Uh, we also have an integration with LangChain. Oh, okay. So you can use LangChain or, um, LlamaIndex- And that- Or Haystack ... you can pick your own, whatever Yeah.

So whatever framework you wanna do, you can choose, and we pretty much have integrations with all of them to, to help with the, um- Associations ... yeah, the associations.

Thanks. Okay. And there are a bunch of hands, but I don't know who's first, so...

I think you go.

You, you, you can go. [laughs] Yeah, you can go.

I think you just said that you have an MCP server for memory.

Yeah.

Or you just... Okay. Does that know also the logic, like the update, the deleting and so on, or it exposes only tools and then other framework like Neo or other handles those logic?

Okay. So, so the question's about like how the MCP agent for memory works. Um, so that, that is a great question, but I actually don't [laughs] know the answer. It's a, it's an open source MCP server.

Now- Go to- Now, if, if you want the answer, um, the session which I'm giving with Michael Hunger and Jesus, um, in, I don't know, in a, in a bit.

Um, Michael's team built all the MCP servers, so he, he actually will know the answer and... Yeah. It's, it's today. I think it's right after this or shortly after this.

And Michael will go on for hours if you ask him that, so that's a [laughs] great question. Okay. And we're at time. You get the last question.

Yeah. So I want to ask about your opinion between, uh, LangChain and LangGraph frameworks. Uh, are they like complementary or they're like... What, what's your perspective on that?

Um, the... Okay. So the question is like, like LangChain versus LangGraph.

Yeah.

I thought, I thought LangGraph was like the agent thing that the LangChain folks built, no?

Did, did you, did you use any of that? Like, I'm, I'm also new to that, so.

Yeah. So we, we have a bunch of experiments and prototypes with, um, LangGraph-

Mm-hmm

... for, for doing agents and things. Um, we have integrations with LangChain.

Mm-hmm.

We also integrate with all the other memory vendors. I mean, like I, I would say from our perspective, use the tool that's best for you and we'll integrate with everything.

Mm-hmm. [laughs]

Yeah. Okay. Thanks for coming.

Thanks. [outro music]