Why Your Agent’s Brain Needs a Playbook: Practical Wins from Using Ontologies

Talk transcript

1. 0:00

   [upbeat music] Welcome.

2. 0:16

   Thank you for joining my session this morning. I was thinking that I was in some kind of niche space where people would not be interested, but I'm curious to hear, you know, what brought you,

3. 0:26

   uh, to, to, to this session. Anyway, I'm, I'm, I'm Jesús, Jesús Barrasa. I'm the, I'm the Field CTO for AI with Neo4j. And, uh, and yeah, for the next, like, uh, fourteen-ish minutes, I'm going to be diving a bit deeper into this, uh, really powerful combination, which is knowledge graphs and, uh, and large language models to build,

4. 0:44

   uh, uh, AI applications. And, and in particular, I'm going to be focusing on these interesting, uh, knowledge artifacts that are ontologies and, and how we can use them, uh, to get some, some practical wins in, in our favorite, uh, architecture, which you I'm sure have heard about over the last couple of days, which is GraphRAG.

5. 1:03

   I mean, it is part of this, of this track. Um, but, um, anyway, e- even if you attended the, the workshops or, or you didn't, I'm gonna spend a minute just refreshing what, uh, what GraphRAG is.

6. 1:15

   But basically, GraphRAG is, is your RAG, where you have replaced your vector database with a knowledge graph that's built on a, on a database, on a graph database. And, uh, and the flow is, is the same, right?

7. 1:29

   So your, your AI application will get a prompt from the user, and instead of passing it on to the LLM directly, what it will do is will go out to a, a retrieval, to a controlled, curated, trusted, uh, sort of knowledge base to retrieve potentially relevant information that will be passed to the LLM in the context window

8. 1:46

   so that the response or the task is accomplished based on, on more grounded, uh, information rather than purely generated. So now the, the interesting thing, of course, is that, uh, by using a knowledge graph, what you get is a much richer collection of retrieval strategies.

9. 2:03

   And we're gonna see that, uh, you have not only the vector semantic search over a vector index, you can also, uh, contextualize results because a graph captures the connections between data points.

10. 2:13

    So you're gonna be able-- We're gonna go deeper in, in, into how that happens. Is it al- also possible to, to generate structured queries? So there's a rich collection of retrieval strategies, like I was saying, that augments what you traditionally have with, uh, with you restrict yourself to only vector search.

11. 2:29

    And, and the result is that you get, uh, better quality results, uh, more completeness, better completeness, more relevance, better precision, more faithfulness, a lot of metrics that I'm sure have been mentioned over the last couple of days, and I won't spend too much time.

12. 2:43

    But, um, but what does the graph look like? Uh, again, uh, especially for those of you who have not, not familiar with this. But the-- we-- Neo4j implements the property graph model, which has two main primitives, is nodes and relationships.

13. 2:55

    Pretty simple. So for us, nodes represent entities, that's things, persons, objects, locations, events, documents, anything. And, and they're connected to each other with relationships that are directed. And then these, you can think of them as objects that will have a collection of key value pairs that characterize them, like description, uh, uh, like name, uh, date of birth,

14. 3:13

    whatever. And, uh, this is what we call the domain graph. Now, the thing is, when we build the graph out of unstructured data, we often enrich this representation with what we call the, uh, the lexical graph or some kind of form of description of the, of the source, uh, documents, uh, from where we have extracted the graph.

15. 3:31

    And, and you can see here that we will have a node representing a document, which maybe has a sequence of chunks, depending on how we partition the document. It can be, in the simplest case, a sequential list of chunks.

16. 3:42

    But if it's, think of a legal document like a, a, a contract with sections, subsections, like definitions, clauses, you might have a richer kind of tree description of what that document looks like.

17. 3:54

    Uh, and, and that would be represented also as a graph. Now, of course, these chunks then will be connected to the domain, uh, graph by, uh, entity extraction. So we'll find that this person is mentioned in this chunk, and we'll keep track of where that's men- that's mentioned, 'cause we're gonna use that when we do the, the

18. 4:09

    retrieval. Right? And, um, and one interesting thing to keep in mind here is that, uh, these properties that we store in a node can also, well, can be numeric, can be string, can be, uh, all sorts of data types, but it can also be vectors.

19. 4:21

    So these chunks will typically contain the text, the raw text that you have, uh, uh, in which you have, you have split your document, but you can also have the vectorized representation, and then you can index it.

20. 4:32

    So you-- It's, it's pretty compelling, the fact that you can have in the same platform a vector storage with, uh, enriched with, uh, with, um, with the connected representation that the graph brings.

21. 4:42

    Now, of course, in order to get to, to GraphRAG, you have to build a graph. There's no GraphRAG without a graph. And depending how, uh, you, where you're getting your data from, whether it's structured or unstructured, you probably will follow different, uh, type of, of pipelines, right?

22. 4:56

    So if you look at the more structured approach, you will have some, uh, uh, um, maybe tabular representation in some database or even files. You will have some description of the target schema, the graph that you wanna build, and you would map those, basically saying, in the simplest case, these records will populate these type of nodes.

23. 5:12

    These are the records will define the relationships, and you will write that to the graph. What you have here in the URL is, is the, the, the UI, the, the interface that our product offers, and basically it does what I was describing here, uh, in a visual way.

24. 5:24

    So you define your graph model, you map to your data sources, you map the, uh, uh, you connect the dots, and, and you get a graph generated. Behind, of course, there's an API base, and you can do the same thing programmatically.

25. 5:34

    But important to understand that you have to pass the target schema. We're gonna get to that in a minute. If you're loading, uh, you're building a graph out of unstructured data, and this is the pipeline that our GraphRAG package follows.

26. 5:45

    But if you look at our integration with LangChain, with LlamaIndex, it will look pretty much the same. What you're doing is you're processing the document. You will split it in some way, based on what we described before.

27. 5:54

    You will take these chunks, and you will embed them, and you will run this kind of entity extraction process that I was mentioning before.

28. 6:01

    To which you will have to inject your, your schema, right? So you'll have to tell what kind of entities do I care about? What's the-- my domain of discourse?

29. 6:08

    Am I looking at, uh, a medical documents where I'm gonna find information about patients and, and symptoms? Am I looking at, uh, you know, financial information like, uh, account holders, accounts, tran- you know, mon-money transfers, et cetera?

30. 6:20

    So that's the kind of information that you find in the schema, and then ultimately you will write it to the graph. But the common bit there is the fact that we have to inject that schema, and that's where things diverge because depending on the path that you follow, maybe, you know, this approach will have a adjacent base

31. 6:34

    representation of your entities and the relationships. Maybe here you're using something like a Pydantic description of the kind of entities that you wanna get, or even a natural language one if you're prompting, you know, your, your, your LLM to, to do the extraction.

32. 6:46

    Now, this divergence is something that's, uh, you know, not ideal to manage. And what you wanna do is have some kind of agnostic and, and, and general approach to, to representing schemes.

33. 6:56

    And that's exactly what ontology is bringing. And, uh, an ontology is just a shared, you know, description of, of a domain. You can think of it a- as a, as a formal schema that's, that's implementation agnostic.

34. 7:07

    And what's interesting, I'm looking at ontologies, uh, there's a bunch of, of standards for defining ontologies, and there's, some of them defining by the W3C. They can be serialized in a number of ways.

35. 7:17

    I've picked the XML one. Why not? Who's using XML in twenty twenty-five? But anyway, there's JSON-based as well. There's many others, right? The, the important thing is what an ontology describes is it's exactly what we're after.

36. 7:28

    It is a domain, right? So it, uh, you will find a definition of a class which actually matches very well what a, what a graph model looks like. So this is a fragment of the FIBO ontology, which is the financial, uh, industry, uh, um, ontology that's a public one.

37. 7:41

    And, and you see the definition of, uh, of um, what's it there? Like a privately held company. I don't know if you can read it there. But anyway, there's, uh, another class there that's a stock corporation.

38. 7:49

    We are specifying that one is a subcategory of the other, so there's a subclass of this, you know, relationship between the two of them. Then there's even connections between, uh, classes of other nature.

39. 7:58

    For example, we say that, uh, a stock corporation is governed by a board agreement, right? So these, all these sort of elements are described in the ontology, and that's a perfect, you know, like I was saying, uh, implementation agnostic approach to, uh, to drive the knowledge graph creation, and that will work for both, uh, pipelines, the structured

40. 8:15

    and the unstructured. And my-- but basically, my, my take on this is, is, um, fundamentally be, be, be model driven, right? Because the, the... Not only because it's, uh, data engineering, but good practice, but also because, uh, you will build a better graph in the first place and, uh, and it will pay dividends in, in, in the

41. 8:37

    long run, and we're gonna see that in the retrieval phase. 'Cause if you have a good description of your graph, your text to structured query is gonna generate better and more accurate queries.

42. 8:46

    But also when we use the, the vector plus contextualization, that's gonna pay off as well. You have there, uh, an example of, of, uh, use of ontology to, to populate a graph with structured and unstructured data.

43. 8:56

    I don't have time to go over that today, but you can, you can have a look there. Now, on the retrieval side of things, let me spend a minute explaining how this magic happens, 'cause I said that you have, um, better retrieval strategies, richer ones.

44. 9:08

    And, um, and what's happening there, you see, uh, that the graph captures. And I'm gonna use a visual. Bear with me. I hope that's, that's useful. So you have a graph, right?

45. 9:15

    In the graph, some of those nodes will contain pieces of text, the, the chunks that we have embedded. So when we embed them and put them in a vector index, we're effectively creating a new search space.

46. 9:26

    Like think of this as a new, you know, uh, way of entering the, the graph, right? So a, a, a search in that space would be, uh, you, you embed your query that defines a point in your multidimensional space, and you will find vectors in the vicinity.

47. 9:39

    So that's what you do when you do a vector search, right? So the interesting thing is that in the graph, when you find the, the, the, the vectors that are in proximity, because these vectors are properties of a node, we can dereference them and get back to the graph, and from there we can contextualize, we can navigate

48. 9:56

    and enrich and filter out, aggregate. So that's where we do the contextualization, and we can bring, uh, uh, additional richness to the, to the, to the context that we pass to the, to the LLM.

49. 10:06

    Now, what does that look like in, in code? So... Oh, I don't know if it's gonna be readable. But, um, let's read from the bot- from the motto map.

50. 10:12

    So this is again, a fragment from our, our GraphRAG package, Python package. But if you look at the, the, the integration with, uh, LangChain is gonna be very, very similar.

51. 10:21

    So we do a, we do a search. We specify, we pass a, a, a query. This is a, a movie database. What we're looking for, uh, we have embedded and put, uh, in a vector index the plots of the movies, right?

52. 10:31

    So we're looking for something. You can imagine the movie that I'm looking for, right? We're looking for computer's, uh, uh, a hacker that discovers that the world he knows is simul- is simulated reality controlled by machines.

53. 10:40

    So that's my search, uh, uh, term and, uh, I'm passing that to my retriever and specifying I want the top three results. That's my-- That's for me to decide.

54. 10:49

    And basically what I'm doing is I've created this handler that, that's, uh, a handler over the vector index, and all I'm doing is telling it to do this search, but I'm passing a retrieval query.

55. 11:01

    So basically what you see up here is this retrieval query, which is basically the logic, the exploration logic that I'm running of- on every single result that I get from my vector index, right?

56. 11:10

    So that's, um, that's powerful. That's great. Now, there is a kind of a problem there, and my problem is that, um, in order to create this retriever, I need to know my model.

57. 11:22

    I need to be familiar with the fact that my graph contains actors that acted in movies, that these movies have, uh, uh, have plots and things like that. And, and that introduces...

58. 11:32

    I mean, that, that's, that's great, but to some extent is a, is a little bit rigid 'cause I have to hard code part of the logic in my retriever.

59. 11:40

    So how could I work around that? Well, the great thing is that if we take the ontology that we saw before and we persist it in the graph, now we have the graph of our data, our movie database, and we have an ontology that describes that schema, what I can do is create a dynamic query.

60. 11:56

    And I understand that you guys are not, uh, Cypher experts, but, you know, if you bear with me on that, I mean, what we're doing is I'm, I'm querying my ontology where there's a definition of a relationship.

61. 12:08

    So I know that actors act in movies, and there's a definition of this relationship. That's what my domain model describes. Now I'm going to query that, and I'm looking for relationships that I have annotated as contextualizing relationships.

62. 12:21

    So I'm defining a subcategory in this case, so property of. So what I can do is drive the behavior of my retriever by consulting the ontology. So I look at the ontology, I find the relationships that have been specified as contextualizing relationships, and those are the ones that I navigate.

63. 12:37

    So these R is the one that I'm using to then find nodes that are connected to other things. So in other words, my ontology is driving the behavior of this query, and it's determining the fact that when I find the movie The Matrix, I'm gonna n- I'm going to be navigating just the acted in relationships and not

64. 12:53

    the produced or the directed, right? I understand that, you know, it might be, uh, going a bit too deep into, into the, the Cypher aspect without having you, you know, you train a little bit on it, but I hope, I hope it makes sense.

65. 13:03

    I mean, the idea is that some of the way in which ontologies can drive can give us a kind of a, a level of indirection that makes it possible for us to kind of build dynamic retrievers that are driven by data.

66. 13:14

    So I can change my ontology, and by changing my ontology, which is a data artifact in my graph, I'm changing the behavior of my retriever, uh, on, on the fly and, uh, and dynamically.

67. 13:24

    So that, that's what I wanted to share with you. So the two takeaways, I mean, the first one is ontologies for knowledge graph creation as an, uh, uh, implementation agnostic, uh, um, data model.

68. 13:34

    And the second one, storing ontology can help us drive, uh, dynamic behavior in our retrievers. That's what I have for you today and just in time, so thank you very much and happy to take the q- uh, questions or the conversation back to our booth.

69. 13:46

    Cheers. [audience applauding] [upbeat music]