Wisdom-Driven Knowledge Augmented Generation at Scale

Talk transcript

1. 0:00

   [upbeat music] So hi.

2. 0:16

   Hi, everybody. Uh, my name is Chin Keong Lam. Um, I'm a founder and CEO of Patho.ai. Uh, bit of background about my company. Uh, Patho.ai started two years ago with, uh, invitation from National Science Foundation, from the SBIR grant funding investigating LLM embed.

3. 0:34

   We did, uh, LLM embed-driven, uh, drug discovery application. Uh, since then, we branch out to leverage what we learned about building AI system for large corporation. We are currently building expert AI system for several clients.

4. 0:49

   Currently, the system we build goes beyond RAG system. Um, many of our client is asking for AI system that perform tasks like, uh, research and advisory role based on their area of interest.

5. 1:01

   Uh, today, the talk is about sharing with our fellow AI engineer what we learned so far building this kind of system. Okay. Uh, what is knowledge? Okay, generally, philosophically, I say, uh, knowledge is the understanding and awareness gained through experience, education, and a comprehension of facts and principle.

6. 1:20

   And that lead to the next question, is what is knowledge graph, right? So knowledge graph is a systematic method of preserving wisdom by connecting them and creating a network or interconnect relationship.

7. 1:32

   That's important. The graph represent a thought process

8. 1:36

   and comprehensive taxonomy of a specific domain of expertise. That's why this is, is very important for people moving forward. It's about AI system that think a lot and return, uh, advice instead of just retrieve, you know, data from your database, right?

9. 1:53

   So that comes to the development of this, uh, KAG. Okay, what is KAG? KAG stands for Knowledge-Augmented Generations, and it's different from RAG, okay? It is enhanced language model by integrating structured knowledge graph for more accurate and insightful response, making it smarter, more structured approach than a simple RAG.

10. 2:14

    KAG doesn't just retrieve. Remember, it understand. This is different.

11. 2:21

    Okay. After inve- interviewing a lot of my client, okay, so, uh, we also are expert in a certain area of scale. I found that there are common ways of their thinking, decision-making process.

12. 2:34

    The way that make them expert in their area, knowledge graph seems to be a perfect fit. So here is a graph or state diagram if you are computer engineering grad like me.

13. 2:44

    So, um, it shows wisdom, the f- the wisdom node, as you can see, is the, is the core, right? It's wisdom, it just isn't static. It actively guide decision and fused by other element.

14. 3:02

    The output from the wisdom actually goes to decision-making in the blue, right? Wisdom isn't passive. It guide decision, helping us choose wisely, okay? And then the decision-making analyze the situation given in the circle in the, uh, green.

15. 3:20

    And decision aren't make, you know, in a vacuum, okay? They analyze real-world situation. That's the difference, okay? So look at the wisdom input, okay? Look at the relationship feedback from the knowledge to wisdom in gold color.

16. 3:36

    Example of that is knowledge to wisdom, like all your book smart and encyclopedia, Wikipedia, whatever you store. Plus, once that data get absorbed by LLM, whatever model you use out there, it need to regurgitate that and understand.

17. 3:55

    That's why it's very important that wisdom is able to synthesize the data after you ingested knowledge. You know, that's a kind of abstract, but I'll, I'll come, come to that later, what I'm talking about, okay?

18. 4:05

    Okay, from insight. Example of that is wisdom derive pattern from chaos. Like, some of my client has a lot of social media. They, their product, how do they, you know, track their product sentiment from, from social media, right?

19. 4:19

    Sometimes it's very chaotic. And from X tweet, right? So, so from that, you can see some pattern of their competitor versus, uh, current what my product is. That, that's like example of that, and I will go to that later.

20. 4:31

    Okay. When all these connected nodes matter together, why do they matter? All the nodes relate to one another to ever increase, um, enriching our wisdom storing system. Okay, this talk is about storing wisdom, right?

21. 4:45

    So knowledge tells you what it is, right? And experience tell you what worked before. Insight invent what to try next. Right? Like a pizza, knowledge is recipe.

22. 5:01

    Experience is knowing your oven burn crust. Insight is like, hey, it is at adding, you know, honey to the crust, it will caramelize perfectly, right? So the most important part of the knowledge graph is feedback loop, okay?

23. 5:16

    Feedback isn't one-way street. It learn from itself. Look at the feedback from the, uh, going back to all the node from insight to wisdom, okay? Um, situation inform future wisdom.

24. 5:29

    Experience deepen it. Insight sharpen it. Like a tree growing roots. The more it fed, the stronger it get. Now, I want to ask you a question, in general. Where do you see this circle in your life?

25. 5:44

    Maybe a tough decision that, you know, taught you something.

26. 5:50

    So one practical application for leadership is wisdom. Avoid knee-jerk reaction by learning from feedback. As for personal growth, ever notice how past mistake make you wiser? That's the loop into action.

27. 6:05

    All this. So the takeaway from the slide in this is wisdom isn't a trophy you earn. It is a muscle you exercise. The more you feed knowledge, experience, insight, the more that guides you.

28. 6:19

    Now, I will show you how it being mapped to my current client. You know, all this is like very abstract, right? So how I-- one of my clients actually doing a competitive analysis, uh, they used to have a marketing department doing that, but they want AI to do that, right?

29. 6:36

    They, they asked me to build a system. This exactly what I did with the same taxonomy of storing all this. So this tax-taxonomy will be... Later on, I talk about how multi-agent is going to handle all that.

30. 6:49

    Here is one of the chatbot that I built for my client to do, you know, not just some, uh, we-- not just some chatbot, okay? It's our Wisdom Graph-powered AI designed to turn data into strategy, right?

31. 7:01

    Dominant. So what kind of question I talk about? Talk about how do I win my competitor in this market space? That's kind of very sophisticated question, right? So without, uh, if, if you do simply just RAG by...

32. 7:11

    First speaker talk about RAG, right? So it's not going to cut it. They're not going to able to answer that kind of question, okay? What I did is this.

33. 7:18

    Uh, we retain the same taxonomy and, uh, the wisdom is then mapped. The same engine there, the wisdom engine. Wisdom engine is like a orchestration agent that does a lot of decision-making, including advising what the, the LLM is able to see based on the current situation, what to do next, right?

34. 7:34

    So, um, what I did is, uh, for the, uh, decision-making, I map it to a strategy generator. So these customers talk- are talking about a competitive analysis, right? So, um, I map the knowledge.

35. 7:46

    In term of knowledge, what do they have? They have market data, right? So I map this experience to HP. It's one of a kind, okay? Past campaign. So they have a lot of campaign, doing a lot of, uh, marketing.

36. 7:59

    And then, um, the insight is actually mapped to, uh, in-industrial insight. They have a database doing, storing that. And then, of course, the most important is this, the s-situation.

37. 8:10

    The situation is how, how am I doing? How my product selling, right? So, so that, that is like a situation, and then I map that to a competitor winners.

38. 8:19

    That means to say, if you make sh- the LLM aware of that, you probably get a very good answer, and then, you know, the chatbot will probably be doing the right thing, advising.

39. 8:30

    So from here, very high level, you know, state diagram, all that. How do I map it to a system that drive? Well, here comes the trick. So anybody here heard of n8n?

40. 8:42

    All right. All right. It's all good. So, so I, I first encounter similar situation when my, my past IoT project, which is Node-RED developed by, uh, uh, IBM, right?

41. 8:53

    So it's the same kind of thing. It's like no code. But, but underneath the hood, there's a bunch of code, okay? It's all Node.js code, okay? So, uh, but, but for the, for, for proving your concept and all that, it's very, very, very flexible.

42. 9:05

    And I, I re-highly recommend that. And, and, and here, here you can take a look at the, the workflow, the workflow. I enable the implementation of this complicated state diagram with, um, uh, what I say is there is a different community node.

43. 9:18

    One of the very powerful node is the AI Agent node. Well, previously, n8n is just a workflow automation tool. I'm not selling for n8n here. I'm just telling you I'm using it, uh, for pro-prototyping.

44. 9:28

    Uh, further down the road, maybe the client say, "Oh, it's too... Like I, I really need to, you know, go lightweight." Maybe we will switch over to some other LangChain or whatever.

45. 9:37

    But, uh, we actually use this. I, I mapped the previous, uh, state diagram from the wisdom engine. I actually mapped that to our, our wisdom agent. Okay, wisdom agent is now have the option to drive a different model, like OpenAI model, Anthropic model, and even on-prem model.

46. 9:54

    And then that, the key in making the state, uh, mach- the state machine work is that my wisdom agent is now overseeing, like a supervisory agent, uh, all these other agent that do, uh, whatever I say on the state diagram logic.

47. 10:07

    Um, for example, the, uh, state of, uh, going into a node of insight. Insight a-agent will has to do, go to the social media, look for the sentiment of all your product, and then collect that, and then pump that.

48. 10:22

    The... You can see that at the dot at bottom that, that we are connected to a, a, a, a, a centralized, uh, graph. The centralized graph will be able to get updated by different agent.

49. 10:34

    Uh, insight agent will update the, their, their perspective, like part of that graph for the, uh, as I say, for this particular, uh, uh, insight node. So, so all the unified knowledge graph will contain the taxonomy that eventually just think like the marketing strategies.

50. 10:54

    The way that they, they was probably, if you are doing manually, they probably would think in your, in your SharePoint or all this, you know, folder will store the same kind of, uh, you know, wisdom I call it, to make decision based on that.

51. 11:07

    So the, the final decision is LLM also depend on the model that you use. Uh, but I, I, I, I pretty much think that not really the way that I think the final decision come when you make a right decision from the advisor output is basically depend on all the taxonomy, the graph structure.

52. 11:24

    That's very important. So come to that, I, I want to go deep down how I implement one of the node, uh, just to go a bit technical on this competitive node, how do I implement that.

53. 11:34

    Okay, before I do that, okay, competitive analysis, right? Why, why, why you can actually just use RAG? Why do you want to use a knowledge graph like now Neo4j?

54. 11:43

    Well, if you ever been asked that question, tell them these five, uh, five reason. Okay, first reason is knowledge graph, you know, uh, system excel at capturing and representing complex relationship with the entities.

55. 11:56

    That is covered by the first speaker, but I'll just reiterate that. This lead to a deeper contextual understanding, which is crucial for competitive analysis, where this, in this case, the nuance insight can be- Signific-- make a significant difference, okay?

56. 12:09

    You want to find the gap in your computer awareness. Now, this is very important. The second is improve accuracy. By leveraging structured data and semantics relationship, knowledge graph can provide more accurate and relevant information compared to traditional vector RAGs.

57. 12:24

    Um, this ensure generated content is not only relevant, but also precise and reduce the noise and improve decision-making, making. This, in this case, the bot is supposed to help the guy that is marketing department make decisions.

58. 12:37

    So, so you better make this work. Improve accuracy. Any inaccurate data, you will be out of the contract, out of the door, right? So the, the very important, okay, you are talking a contract work like me, I have to make the RAG as accurate as possible.

59. 12:49

    So the third is scalability and flexibility. The graphic, you know, knowledge graph are inherently scalable and can integrate to new data source and relationship. The flexibility allow the continuous improvement.

60. 13:00

    As I say, if your taxonomy is correct, you will continue to improve and, and reach, right? So, so that is important. And also, rich query capability. Knowledge graph support complex query, traverse to multiple relationship entity, provide richer and more detailed insight.

61. 13:16

    This is particularly advantage for a competitive analysis where multifaceted query like, like what the first speaker said, it is super authoriously good in answering things that normal, normal, uh, normal RAG will fail.

62. 13:28

    It's like multi-hop question, okay? This is very important. And then the final one is the enhanced data integration. Uh, knowledge graph can seamlessly integrate diverse data source, pictures, graphics, videos.

63. 13:42

    Uh, however it is, now that LLM is so powerful, we have OCR capability, it can do that. As long as you have a right structure of the graph, semi-structure and structure, the holistic approach ensure comprehensive view of the competitive landscape enable more informed strategic decision-making.

64. 13:58

    Okay. So one of the-- This is, uh, I'm gonna just very briefly go through this. It's just a, a example of the some of the thing. Like, um, problem of a vector's RAG, you know.

65. 14:08

    Vector RAG is really, really bad in answering limited numerical res-reasoning. Vector store Excel, you know, at semantic sim-similarity, but struggle with complex numerical calculation. This is why, uh, for, uh, market an-analysis, uh, that I'm building the chatbot for, uh, they actually rely on number instead of just, you know, returning example like this, like kind of...

66. 14:30

    If you ask like, uh, what is the Apple, uh, revenue, uh, between, uh, two, uh, you know, what's the revenue in two thousand twenty-two? They, they probably will give you a bunch of these kind of a passage, right?

67. 14:42

    Retrieve a graph. Instead of, uh, this kind of a very, very precise thing like, uh, the, the answer is, uh, you know, uh, knowl-knowledge graph is able... Because the, uh, the, the, the data is already there in structured form.

68. 14:54

    The data source assume a knowledge graph name, this particular, uh, in, in this particular case, Apple financial data. The query will be able-- The query engine will be able to select the, the revenue figure from twenty twenty-one to twenty twenty-two, and, and then do a function call.

69. 15:09

    The function call will eventually give, come out with fifteen point two three, which is exactly what the marketing guy was looking for. A very quantitative stuff that most of the decision were based on that because you have the evidence, not just some passage that you retrieve from the data.

70. 15:23

    It, it's basically evidence-based decision-making. It's very important for this kind of, uh, complicated RAG system that, you know, uh. So, um, there's a jungle out there right now. You can use different kind of, uh, uh, uh, uh, uh, of thing to build yours, uh, you know.

71. 15:38

    Uh, this is just a snapshot of that, you know. You can actually use, uh, LangChain plus Chroma to, to build your own RAG, and then you also can combine that with your knowledge graph.

72. 15:48

    Depend on, on, on your user case, okay? If, if, uh, the, this slide show that the RAG and the KAG can be built with many, okay. I adopt that wisdom graph in red color.

73. 15:59

    Normally, you will see if client is just asking for a simple RAG that perform product information query, you can just use a simple Chroma DB with LLM agent. And if you start to ask so complicated questions like, "How can I beat my competition based on my current market share?"

74. 16:15

    Well, this will be able, the, the, uh, the, the, the thing that I will probably be adopting is knowledge graph here with, uh, graph DB plus Cypher query and it will create A and A, and also train my RAG to perform several loop of, uh, we call multi-hop query.

75. 16:32

    And this probably will give a very good answer.

76. 16:35

    So, uh, and then it come to the another question. When I was trying to extract my, uh... Oh, I think my time is, uh, is almost up. Okay. So anyway, this is like to say, uh, the first speaker talk about the extraction, right?

77. 16:49

    There's a very simple way to extract. On the right side is, like automated, totally automated LLM graph transformer. On the left is like manual. I would probably rec-recommend the center hybrid model, which is like after you use the LLM to extract your graph, you ask to interview the, the expert that you're gonna build, uh, to, to build

78. 17:06

    a taxonomy, right? To prune the graph. We call it pruning a graph. Remove a lot of relationship there. Then that, that will be okay. And, um, I will try to just highlight this.

79. 17:15

    This is the result of benchmark that we did. Okay. Anybody ask you, you know, why you want to use graph, right, or KAG? Okay. First is accuracy. I had achieved ninety-one percent because it's really good in extract structure.

80. 17:26

    Second is flexibility, eighty-five percent. Third is rebo-ducibility, reproducibility, deterministic. And then the fourth one, traceability. And finally, uh, most important is scalability. So in conclusion,

81. 17:41

    by leveraging structured nature of wisdom knowledge graph, we can significantly enhance the quantitative capability of KAG system and enable more accurate and insightful response to com-complex query. By using wisdom-driven system as highlighted, together we can build smarter AI system that can scale and store wisdom with the right framing, potentially surpass the, uh, intelligence of the initial expert

82. 18:04

    that we meant to serve. [audience applauding] So we, uh, talk to Jesus, you know. What do Jesus do? Talk to Jesus. He's in, uh, in, in a notebook. This is my good friend.

83. 18:15

    And, uh, anybody that want to build graph, we have a good, uh, so-called LLM GraphRAG, a stack on GitHub that, um, is sponsored by Neo4j. And out of the box, just spin up your Docker.

84. 18:27

    The next thing you know, your text is going to be converted to your graph, and you can start happy pruning your graph. Thank you. Thank you so much. [audience applauding] [outro music]