Why your agents need decision traces, not just documents — Zach Blumenfeld, Neo4j

Talk transcript

1. 0:00

   [on-hold music] So my name is Zach.

2. 0:16

   Uh, I work for Neo4j. We're a graph intelligence company. You can think of us like a knowledge layer graph database at the core. We help connect and resolve information so AI systems can be a little bit more accurate and explainable.

3. 0:30

   Um, I used to work in technical marketing. I very recently am going back to my AI and machine learning roots, uh, and transferred over to a, to a research engineering role.

4. 0:40

   So I'm gonna talk to you today about context graphs. How many people in this room have heard about context graphs before?

5. 0:49

   Okay, so we've got about half. And how many people have actually coded anything with a context graph or with a graph like Neo4j in general?

6. 0:58

   Okay. Okay, about half, so that's good to know. So I'm gonna talk a little bit about how we think about context graphs at Neo4j, define what they are, and then I wanna go over some tools, um, that are built by, uh, William Loin, who's one of our product managers, um, so that you can get started with them

7. 1:14

   very quickly, um, with your own code and your agent framework. So context graphs [audio cuts out]

8. 1:22

   Foundation Capital, I think back in December. And essentially, uh, we saw [audio cuts out] of noise around this idea of creating decision traces and reasoning, um, to help agents make better decisions.

9. 1:35

   And so to kind of think actually about what a context graph is, you need to ask yourselves, what do agents need to really be accurate, right? And so for one thing, doing a lot of retrieval, you're gonna need a knowledge base, obviously.

10. 1:50

    Um, so a knowledge base for something like RAG or graph retrieval basically helps an agent or a chatbot answer questions correctly.

11. 2:00

    What a context graph does in the evolution of this is really the information required to not only answer questions correctly, but make better decisions. So if we took an, a concrete example, and I'll show you a demo of this, [audio cuts out] financial analyst agent, um, and say that this agent has a question around improving [audio cuts out] increase and a request for

12. 2:22

    a certain amount of, of money. To allow that a- [audio cuts out] information about the s- the system, so customer info, transactions, and policies. The response from that is likely to look something like this, where it could maybe assign some sort of risk score and recommend some sort of review and talk about some key risk factors, which is helpful.

13. 2:46

    What a context graph enables an agent to do is actually give an answer, should you reject, accept, and why, right? And it does this because in addition to getting that customer information and those transactions, it's also gonna get you past decision traces and precedents, dynamic information about why decisions are made.

14. 3:09

    Um, um, systems of record really about facts, entities, [audio cuts out] state.

15. 3:21

    About precedents, causal chains, ext- [audio cuts out] outcomes, and enabling the agent to act with subject matter expertise, um, and telling the agent really [audio cuts out]

16. 3:34

    model for a context graph. Um, the entities about things that exist. There'll be, uh, events, so decisions, transactions, approvals, things like that. And then context, [audio cuts out] policies that [audio cuts out]

17. 3:50

    um, and different reasoning by AI that requires memory, but, um, by employees and, and past humans that have made decisions. [audio cuts out]

18. 4:04

    some of it because, you know, with internet connection, I never really know, you know, how things are gonna go. Um, all the code for this is available. I'll, I'll show links at the end.

19. 4:13

    Um, basically, [audio cuts out] data that we generate replicates taking data in from a CRM and a support system and some other places. Um, it's using, um, Claude in sort of the agent runtime, has some OpenAI embeddings.

20. 4:31

    Um, there's Neo4j [audio cuts out] database [audio cuts out] the data with some vectors. Um, and then we have our Next.js front end, um, as well. So

21. 4:43

    let me show you this record [audio cuts out] running here so you can kind of see what this looks like, 'cause again, [audio cuts out] You ask it a question, and it will call basically a series of tools, and you can see it bringing the graph data back.

22. 5:01

    Um, and then you see it getting these different decision traces, and eventually it will get this reject decision, right? And if [audio cuts out]

23. 5:27

    then it does this thing called Find Precedents, which is gonna be [audio cuts out] more in a second, but it's gonna look at structural information and graph to pull a bunch of [audio cuts out]

24. 6:14

    In addition, the internet would work, connect to each other in these causal chains.

25. 6:33

    So you build off of each other. Is, um, so like I was saying before, um, we queried

26. 6:58

    just context around Jessica's profile. We pulled back decision traces, and we did a special type of hybrid search, especially around those precedents, both on semantic similarity and structural similarity inside of the graph that actually looked at how the decision traces were made, um, from previous decisions, and it tried to match that.

27. 7:17

    And I'll talk a little bit about how that works with graph embeddings in the next slide to ultimately come up with the recommendation.

28. 7:23

    And so we add a-- we have a vector index, so we're able to search,

29. 7:30

    like fraud rejection, for example, sort of on, on the semantics. But then we have this concept called graph embedding. So you-- a lot of you in here are probably familiar with text embeddings, right, on, on words, right?

30. 7:42

    A graph embedding is the same concept, except those green nodes that I was showing you before, everything was connected. We actually embedded those into a vector. And so what that means is similar decision traces are now gonna be able to be looked up by vector similarity.

31. 7:57

    And so if you have a system right, where you've had past decisions and tickets that have been, now you up just like you would with vector search for, for another type of text.

32. 8:07

    And that might be very, very hard to pull out if you just had it inside of documents. Um, this is just some more information around, uh, the context graph demo and some of the scenarios you can run, um, and some of the tools that were used here.

33. 8:22

    Um, GDS is called Graph Data Science. That's what we use for the, uh, for the graph embeddings themselves. Um, and then I'll link you over to the code here to rerun that.

34. 8:34

    But what I really wanted to show you today, um, was another tool that was just recently created

35. 8:41

    and then a month ago, um, which allows you to create a full, um, stack application to start with, with just a one-line command in the terminal to actually create the context graph and the front end and the back end.

36. 8:56

    Um, and so basically you can think about this like, um, create React app or create Next app. Like you basically get this boilerplate, um, and all of the scaffolding that you

37. 9:08

    and you can specify the domain when you create the graph, um, or when you create the application rather in the command line. Um, and it will give you this, you know, basically the back end, the front end and, and, and everything, um, out of the box.

38. 9:22

    So if I was to show you, uh,

39. 9:26

    if I go over here, kind of looks like this, and I know it's probably hard to read. There's just a uvx command, uvx create-context-graph, um, specify the name of the app, the domain, um, the framework I wanna use.

40. 9:38

    There's a ton of them that, that we can use here. I'm choosing Pydantic AI, um, and I'm, and I'm specifying demo data. And it will go and basically create this folder for me with a bunch of fixture JSON, the data.

41. 9:51

    In this case, gonna use its, uh, you know, basically dummy data that I can use to help host the application,

42. 9:58

    which we don't have enough time to run through all of them. It takes a few minutes to basically install the application and seed the data and stuff and start.

43. 10:05

    But I do have it run, um, that it would give you. I don't know how helpful that is 'cause it's probably hard to see.

44. 10:15

    If I go back to, uh, it'll give you, um, you know, data inside of a graph and then, um, you know, you can go ahead and ask questions, um, and it will go and, and, you know, basically do this type of retrieval with Cypher and, and everything else.

45. 10:39

    Cypher's our graph query language, um, to pull data back. Very similar to what we saw in that, uh, context graph demo that was hosted. Sorry. I'll let it run for a little bit here and, um,

46. 10:57

    get moving 'cause we only have... All right. Can you hear me now? Okay. How long did that go on for, by the way? How long was it?

47. 11:08

    Uh, just a minute. Okay. All right. Okay. Well, here's our response back. So you can see, um, you know, I asked it about, um, prescription medications and, and it, and it went ahead and, and pulled everything back.

48. 11:22

    And there's a visualization of the schema here, and like I was saying, different decision traces that you can look through. Um, so this is a great place to get started if you just want to-- you're interested in context graphs, and you just wanna code something up really quickly.

49. 11:36

    Uh, there's a website that we have for this too that you can go to. Um, it explains this step-by-step, um, around how to, you know, get the app started and explore everything.

50. 11:47

    Um, there's also some very interesting new tools if you wanna import from, um, software as a service. So if you wanna import from GitHub or Notion or Jira or Slack, so you don't have to use just demo data.

51. 11:59

    You can import data from these other tools. Um, and then some other features, and I'll give you a link to all of this at the end, like I said.

52. 12:08

    Um, so we were looking at a Pydantic AI application, but you can also do it with OpenAI and LangGraph and CrewAI and Strands and Google ADK and all these others.

53. 12:18

    There's a 22 built-in domain, so healthcare was the one that I just showed, but there's also FinServ. You can also create your own custom domain, and it will actually help generate an ontology, basically a graph schema for you and put everything together.

54. 12:32

    We talked about, um, the different data connectors for GitHub, Slack, and et cetera. Um, and it has all of the graph-native stuff to power your queries, um, that we were seeing to pull the data, look at decision traces, um, generating an MCP server, and doing actually multi-turn conversation inside of that app.

55. 12:52

    Um, so again, this project is, uh, just on its-- just getting started. Um, so it's open source. People are welcome to commit, um, and contribute there. Now, I'll talk a little bit about what this project is, uh, based on.

56. 13:08

    So, um, underpinning this, one of the big dependencies is our Neo4j Agent Memory package. Um, so this is a complete memory API. Uh, it includes... And context graphs really need all three of these things, um, a short-term memory, a long-term memory, and reasoning.

57. 13:26

    So short-term is more conversation history and session context. I think we, we understand that. Uh, long-term is the entities that are extracted from that, um, and those that repeat over time and resolving those down.

58. 13:41

    And then the reasoning is gonna really be sort of the, um, the traces inside of that context graph.

59. 13:48

    Um, and there's also inside of here, which is very useful, I know a lot of people think about, "Well, if I have this text data, how do I put it into a knowledge graph," right?

60. 13:56

    Is, is u- usually one of these big questions that people have. Um, so here, i- we actually implemented this inside of the package, um, where it goes through a few different stages on raw text.

61. 14:08

    There's other tools that you can use sort of outside of this if you want to, um, take in structured data, um, or you can use your own language model extraction, um, NER processes.

62. 14:20

    Um, but this one just uses a few stages where you go from spaCy to GLiNER and, and more advanced to an LLM fallback. Um, and then there's a separate merging, deduplication, and enrichment strategy.

63. 14:31

    So, um, it's a little bit more well thought out, and that helps, especially with the short-term memory, take that information out and transition it into useful long-term memory and entities that can be resolved over time.

64. 14:41

    And the schema kinda looks like this, where you have your conversations, um, and then in yellow, you have your, um, your entities that get pulled out, and those connect to your reasoning traces.

65. 14:56

    All righty, so we're getting towards the end here, and we might actually have a few extra minutes for, for questions if, if that's allowed. Um, but here are the resources.

66. 15:06

    So the context graph demo that I showed, if you go to that blog, it'll also link you to a live application that's just hosted. Um, and so that, that's very useful if you just kind of want to explore the concepts out of the box.

67. 15:21

    Create context graph in the middle. On the top one, I know on our Wi-Fi

68. 15:26

    for this con- for some reason, it blocks that website. It won't block the website if you go on a normal Wi-Fi network. I don't know why it does for this one.

69. 15:36

    Um, but that link does work. And then on the one on the bottom is basically just the GitHub repository. Uh, Neo4j Agent Memory, um, is the agent memory package that underpins, um, all of that.

70. 15:48

    And that also integrates with Microsoft Agent Framework, um, as well as Google, uh, ADK and, and a whole bunch of others. Um, so that's actually it for me. I made it through with a couple minutes to spare, so I'm happy to open it up to any questions for the next couple minutes if, if there are any in

71. 16:08

    the room. Yes.

72. 16:09

    So you had causal chains in there, right?

73. 16:12

    Yep.

74. 16:13

    Is there any sort of concept in there around temporality? Like, you know, how, um, how do we go these causal events happen and how relevant it is to retrieve them at that time?

75. 16:26

    Yeah, there is. You can add timestamps. Um, you can add timestamps, and then obviously, as they occur in time, the different steps, they'll be linked by like caused or next.

76. 16:37

    Yeah.

77. 16:38

    So does it put more weight on more recent events or does that get used to follow-

78. 16:43

    Um, I, I don't know if it does that quite yet. I think that's something that will happen as the, you know, as the technology matures a little bit. But yeah, that, that is a good point.

79. 16:52

    It's a good idea.

80. 16:53

    Uh, and the ontology is fully general. It's just, uh, you prepare the nodes and the edges, the kinds of them.

81. 17:00

    So you'll prepare the ontology beforehand. For those domains that, um, I showed you, the preexisting ones, a lot of them will use this, this something called PolE, which is like, um, entity, um, basically policy.

82. 17:14

    There's like a few like predefined entity and relationship types, and that's used to guide the extraction and all of the mapping and stuff. Um, so yes. Yes.

83. 17:23

    Uh, if I have a giant amount of information that has been collected over time, would I be able to somehow automate creating a front out of this?

84. 17:34

    Automate creating a front, uh-

85. 17:36

    Automate extracting this information and turn it into an ontology essentially.

86. 17:42

    Yeah. I would look at the, I would look at the create context graph package. At minimum, if you're able to describe the ontology, um, then describe what's in the data, um, then you can definitely create a graph schema from there.

87. 17:57

    Um, and then that will help you do, if it's unstructured, do entity, entity extraction. Um, I-- and it depends on how structured your data is. So if it's like very structured and it's in like, um, for example, like CSV files or tables, um, then it's just a matter of like mapping the Cypher statements over.

88. 18:16

    Uh, yeah. So I think that, that tech stuff, I think that can be handled inside of here. If you look at, like, create your custom domain, there might be some things that you can do there to help you.

89. 18:25

    Again, it's a new project, so it's al- still a little bit rough around the edges, but at least you'll see the code and you'll see how it can generate an example ontology and then how that can be fed to an extraction process to, to move everything through.

90. 18:39

    Yep.

91. 18:41

    Um, it kinda seems like over time this will grow a lot, so if there's like-- Is there sort of a way of determining whether certain, like, traces are good or bad, or would that just be like a human in the loop kinda like thumbs up or thumbs down?

92. 18:55

    Or like, whether there's some sort of automated way of determining whether a trace is new and novel and needs to be added to the graph, or if it's kind of like represented by existing traces in the graph, et cetera.

93. 19:07

    So like, kind of wondering if this could be an entirely automated self-learning system, or like, like users should be involved in that process?

94. 19:16

    Yeah. I think we're still figuring that out at the moment. Um,

95. 19:21

    I know in, in this first demo that it is-- Like, you can ask it to store, like if I, like from my previous conversation, I can ask it to store decisions, but I don't think it will do it unless I prompt it to.

96. 19:33

    Um, but yeah, that's a good point. And then in the create context graph, we're still working on how you would write, um, new decision traces. Um, so, so yeah, it's something to think about, like some sort of sentiment or, um, like quality score on the decisions.

97. 19:52

    All right. Cool. Well, thank you guys. [clapping] [outro jingle]