Build Your First Demand-Driven Context Base: Let AI Agents Tell You What They Need

Talk transcript

1. 0:00

   [upbeat music] Uh, thank you.

2. 0:16

   Maybe let, we can get started. Uh, first of all, thank you so much for coming for the workshop, especially ones who didn't get the seat. Uh,

3. 0:26

   I, I promise you, I'll do my best to make it entertaining, especially for you who are sitting. [laughs] Uh, thank you so much.

4. 0:33

   We are the volunteers, right?

5. 0:34

   Yeah. [laughing] Uh, actually it, it, it makes sense, so now I know like why the tickets got sold out, right? Uh, which workshop actually sold out the tickets. [laughing]

6. 0:45

   So, uh, s- let's start with, uh, my introduction. So I'm Raj. Uh, I work as a staff software engineer, uh, at IKEA. Uh, I work for a domain called Delivery and Services.

7. 0:57

   Uh, basically we are like almost, um, more than 100 engineers and six pedics teams altogether. It's like a mini company within the company itself. Uh, I'm very interested with architecture, neuroscience and linguistics, and now AI.

8. 1:11

   So if anyone has some cool projects, because everybody is building cool projects these days, please find me after this meeting.

9. 1:20

   So quick pulse check, uh, with the, uh, audience. Uh, who is visiting London for the first time?

10. 1:28

    Okay, cool. Welcome to London. Uh, who is here, uh, from the engineering background?

11. 1:36

    Also with white coding and, uh, prototyping.

12. 1:38

    Mm-hmm.

13. 1:38

    No all of... Okay. So, uh, who actively uses agents like Copilot or... Okay, this is going to be tough for me then.

14. 1:47

    Yeah. [laughs]

15. 1:48

    So extensions? Okay, so everybody's pro. Okay, fine.

16. 1:52

    Not so much as I wanted to.

17. 1:53

    Okay. So much tension now. Uh, so you're gonna sit here in this hot room for more than an hour. So first I will, uh, give a bit of introduction of what I'm gonna present today.

18. 2:07

    Uh, it's basically on agent and the context management. Uh, I, I divide it into three parts. Uh, one is the situation which all of you already know, so I'll keep it tight and short, like five minutes.

19. 2:18

    Uh, then I will talk about the problem. This is where I'll spend a bit more time on the slides because I think like, uh, nobody is actually seriously looking into the problem, uh, and I want to bring it up.

20. 2:32

    Then less slides and more into some kind of a hands-on, uh, how the, uh, actual demand-driven context actually works. All good?

21. 2:45

    Yes.

22. 2:45

    Okay. So let's start with the first one.

23. 2:49

    How many have seen this movie? Memento.

24. 2:53

    Oh, yeah.

25. 2:53

    Okay. Okay. Okay, cool. So I'll, I'll give the gist of the movie. So this guy is very skilled, uh, very talented. The only problem he has is, uh, he can't hold memory more than 15 minutes.

26. 3:05

    So every 15 minutes he has to take his notebook, uh, watch his [REDACTED:physical_attribute] that he put it on his one and figure out, "Okay, what I was doing before the 15 minutes?"

27. 3:14

    And he does it again and again. If you relate to the AI and AA agents things and all, it actually fits exactly how the movie is and how the agents are right now.

28. 3:23

    If you go and watch this movie, you don't need to y- watch YouTube or blogs to understand agents and MCP. This movie actually tells you ab- about everything, literally.

29. 3:32

    Uh, and as this guy has a memory problem, uh, in the same way, the AI, uh, that we got introduced couple of years ago, uh, is very good with reasoning, computation, uh, code generation.

30. 3:44

    It's, it's benchmark as like, uh, above the par. The only problem is the institutional knowledge, right? The domain knowledge that you have, that, that's the only thing we have to be, uh, a, a bit problematic.

31. 3:55

    So from AI to agent, if you look at the eva- uh, evolution, uh, it exploded. So it started from prompt engineering, first of all. Uh, then there was RAGs, uh, now MCPs, then multi-agent.

32. 4:09

    Now it is deep agents. Uh, I recently found out like, uh, using Replit, actually, you can build a full stack app in 10 minutes. That means the... By the time you make the instant noodles, you already have a million-dollar, uh, app already working on your laptop.

33. 4:24

    So we, we got it to this point, uh, like it's extraordinarily good.

34. 4:31

    Now, that's AI and agents. So let's talk about enterprise AI.

35. 4:36

    Okay. Uh, I don't know how, how many have you have this question, but most of the enterprises I see the question is, okay, AI is pretty smart. It's doing, uh, code generation, full stack apps, reviewing your PRs, uh, doing incident management, all those things, uh, right?

36. 4:53

    So if I was doing that much, why is the Jira tickets or epics are not moving on the dashboard, right? Why do I don't see the delivery actually? So everybody is speaking about look at like three minutes, everything is ready.

37. 5:05

    Yeah, okay, fine. Why are my Jira epics are not moving? Because that defines the business delivery and that defines the return of investment, right? Uh, and as you see, like, uh, it's, it's from the McKinsey this year, like 88% of all companies use AI, but they only see like, uh, 6% of value creation.

38. 5:33

    Okay. So I think this is the problem, uh, that we have. Uh, I have four Jira tickets, different ones, uh, sample ones, and you can see the green ones that I have, uh, marked is basically which LLMs are already trained on, like API standards or like, uh, uh, things very, uh, they already know it's a general knowledge,

39. 5:54

    right? That's fine. Those tasks from the ticket it can pick up and it can do it. Now, there are second part orange ones, which we have to teach them actually.

40. 6:02

    So you have... You know this, but do this in this way. So all this kind of an orange color things will fit into, you know, your agent extension like skills or like, uh, [audio cuts out]

41. 6:16

    But red ones, that's what the institutional knowledge is, which sits within the company and within the people. So unless if it picks a task, uh, if it picks a ticket, it has to fulfill all of them.

42. 6:31

    It is so good at with, uh, green ones and orange ones, but it struggles with the red one, with the institutional knowledge. And, uh, what I believe is, uh, right now the coding agents are getting so, so better.

43. 6:45

    I, I feel like if there is an AGI coming, the first AGI will be a coding agent for sure. Uh,

44. 6:56

    so to fix the, uh, giving the institutional knowledge to, uh, the agents, so we have an industry solution already. So this is basically your return of investment on AI pipeline will look like, right?

45. 7:10

    So you have a LLM model quality, uh, you have agents, and you have a agent harness, and your, uh, your institutional knowledge sits under Confluence, Jira, SharePoint, GitHub, all those things.

46. 7:22

    And basically, retrieval layer is what industry is telling us will fix that issue. So you build that retrieval layer, and then it will fetch all those things and give it to an agent, and the agent should be able to do it, right?

47. 7:33

    So basically, uh, so forty percent, uh, actu- factual accuracy can be achieved through RAG or like knowledge graphs, actually, uh, but with a documented, uh, knowledge base.

48. 7:52

    Now, basically, if you build a retrieval layer, it has to work, right? Now, let me ask you a question. How many have you built a retrieval layer things like RAGs and MCPs?

49. 8:02

    Okay, cool. All of them. Okay. Now the question is, how many did you build? How many M-MCPs did you build? How many have you built more than twenty MCP servers?

50. 8:13

    Okay. Okay. So nobody beats my record then. Uh, so what I see is, uh, mostly in the enterprise organizations and all, people are building like ten to fifteen or like twenty MCP servers or like RAGs, uh, knowledge graphs on top of their institutional knowledge and to the agent, right?

51. 8:33

    So the assumption is if we, if I can build all these MCP servers and give that agent, I don't need to work anything like it will do. But the thing is, uh, when you plugging this MCP servers, basically all this data coming out is mostly undeterministic, it's unreliable, and it's untested, right?

52. 8:54

    So especially in engineering, nobody does evals. Actually, it's, it's more like a data, uh, machine learning concept, but we don't do eval. So for me, if you pl-- Sorry.

53. 9:04

    Uh, uh, uh, if you plug, uh, an MCP or like a RAG and all, we see whether the output is coming or not, rather than is it really valuable actually?

54. 9:15

    Is it really solving the problem or not? Uh, that's the main problem that I see. B- I'm not saying pointing other people because I was that person. I was like, "Okay, let me build all MCP servers, plug in my institutional knowledge.

55. 9:27

    I'm gonna prove that point that agents can s- semi-autonomously can continue and f- uh, fill those Jira tickets and finish it," right? But every time when I build those MCP servers, ten percent, twenty percent, thirty percent of time it was accurate, but the rest of the time I was doing the data entry job for them, actually.

56. 9:46

    So I was filling the gaps, asking the questions. So basically, I'm doing more work than actually doing less work. Uh, so I think this is the main problem, and I, I actually was in this fourth stage where I literally started to write the domain context with handwritten, actually.

57. 10:04

    So okay, let me write everything and prove the point, but I got really exhausted, uh, of doing it.

58. 10:11

    Okay. So how-- I don't know how many can you relate, uh, with this pie chart. Uh, but most of the enterprise, uh, the institutional knowledge is kind of something like this.

59. 10:23

    So twenty percent if you see it's outdated, twenty percent it's unreliable, uh, twenty percent i-- uh, ten percent is al- always duplicated with different places. And the major problem is forty percent of the knowledge is always, uh, tribal knowledge, which means people know how things work.

60. 10:41

    So it's, it's never documented, actually. So in this situation of an enterprise and you build like a hundred MCP servers and plug into that monolith, it doesn't matter how many you build, it won't work because basically your whole institutional knowledge is a, is a monolith.

61. 10:58

    Uh, I think like, because you're all from an engineering background, so you already know, uh, the transformation of monolith legacy system to microservices, right? So in the same way, unless we break down that monolith knowledge base into some kind of a context blocks which are useful for agents, then only we can actually make it useful for them,

62. 11:19

    uh, for the agents, and actually make them semi-autonomously can actually do the tasks.

63. 11:24

    So that said, uh, uh, we are going to talk in this workshop mostly on that monolith, how to break it, uh, what is the approach to break it, and how it is useful, uh, when once we break it.

64. 11:36

    And this is a job we need to do because, uh, uh, the LLM providers will, uh, focus on the, uh, LLM, uh, model quality. The agents will focus on the harness things, and there is a big re- uh, uh, retrieval market of nine billion.

65. 11:52

    They're focusing on retrieval. But nobody is going to come to your company and fix your knowledge base. You have to fix it yourself, right? So how can we do it?

66. 11:59

    Uh, okay. So the demand-driven context, uh, is what the other solution, uh, I was trying to propose, right? So basically, if I have to give an abstract of it, what it is, uh, like we have mic-- uh, monolith Uh, services, and we have this process of breaking them into microservices.

67. 12:20

    We have waterfall model, which we transform into agile. In the same way, when you have a monolith of institutional knowledge, how do you transform into a context blocks using an approach?

68. 12:30

    So this is an approach of how we can do it.

69. 12:34

    Uh, before starting, uh, just not an idea. So, uh, we already tried with some datasets and tried to prove this approach works. And, uh, in the March we have published a preprint, uh, uh, in arXiv.

70. 12:47

    So if anyone interested in reading a academic papers, uh, you can find it, uh, with the Demand-Driven Context, or like I can also give you a link after the, uh, workshop.

71. 12:59

    Okay. So how does it work actually? Uh,

72. 13:06

    when we are giving institutional knowledge to agents, basically what we are trying to do is we are trying to do a push strategy, right? So we build everything and we push it to, uh, to it.

73. 13:16

    So in, in this approach, it's more pull approach, uh, which means, for example, let's say, uh, a new joinee has joined your company, right? How do you onboard a person?

74. 13:26

    So you onboard them, uh, for, uh, one, two days. You give some initial orientation, and then you tell them like, "Okay, these are the Confluence links. This is the GitHub.

75. 13:34

    This is the some, some kind of a documentation. You have to follow things and all." Then you just assign a task to the person. So but you're not gonna tell, "Okay, go and gr- get graduated on, on this knowledge and come back, then I will give you work," right?

76. 13:48

    So you'll just assign the work item. And when you assign the work item, the person will start asking questions, fill the gaps. If the person is very much into documentation, he will also fill the documentation for you.

77. 14:00

    Uh, he gradually, uh, get his knowledge of, of the institutional knowledge, right? In the same way, we don't push all the knowledge to the agent. Rather than we start giving problems to agents, like work items, and let them actually pull the information from us.

78. 14:17

    And once, uh, pull the information, also ask them to document it,

79. 14:24

    uh, in, in a, in a better way, rather than in a monolithic structure. So if you s-- So that's the four layers. So you have a monolith, uh, a framework, and it actually pulls and actually creates a good better context, uh, blocks.

80. 14:38

    You can actually relate it to more into a, a legacy monolith to microservices directly if you have to have an, a knowledge of it. So this is how it works.

81. 14:48

    So this is one cycle of, uh, how...

82. 14:55

    The problem to an agent, and the, in the first attempt, the agent will fail to do it. So it will say, "You know what? You gave me a problem, but most of the, uh, documentation, I couldn't able to find anything.

83. 15:07

    I couldn't able to do it. Then these are the things I need to do [coughs] to finish this task," and it gives a checklist of things. So we fulfill the checklist, like we, uh, fulfill the checklist.

84. 15:19

    So once it is, uh, given, the problem is solved, it will take that knowledge, and also it will update. That means curate the knowledge in a particular place so that it can reuse or like other agents can also reuse.

85. 15:32

    This is one cycle. So the idea is if we can do it in multiple, uh, sessions with multiple problems, so it will gradually, uh, curate your knowledge, monolithic knowledge base, uh, and also document it for you.

86. 15:49

    Uh, you can also relate it to TDD. So how many are g- uh, do TDD? Or like... Nobody hates TDD, right? Before, before I, uh- [inaudible] Yeah. [chuckles] Okay. Okay. So, uh, in, in the same way, right?

87. 16:03

    In, in a TDD approach, what we do, we just write the fail test cases. We don't build the product, first of all. We just write the fail test cases.

88. 16:10

    Uh, we see what is the code that is missing, uh, for the fail test case to pass, and we just give that code. And we gradually build a, a product based on the fail test cases.

89. 16:20

    In the same way, we give problems that agent will definitely fail, and we gradually, uh, fill those gaps. And at a certain point, it becomes semi-autonomous with a good, uh, institutional knowledge already.

90. 16:35

    Okay. So I think I can jump into, uh, some kind of a, a demo already. Uh, so I will use terminal, so don't hate me. Uh, I, I think like all of you are from engineering background, so I think like you will like terminal.

91. 16:54

    Uh, let me switch to terminal. Okay. Okay.

92. 17:11

    So on the, on the far left, what, uh, right, what you see is how, uh, under the hood it works actually. So when you have given a problem, uh, how does the agent will fail?

93. 17:23

    How does it demand for the knowledge, uh, that the problem has to be solved? And a human, like a domain expert and all, f- uh, fills those gaps. And then it will curate, uh, a new knowledge base for you, which is, which is much better.

94. 17:38

    And then the agent succeeds, and you repeat on the next problems. So that is how one cycle, uh, uh, things are done. So how I-- it can be implemented, it can be implemented, uh, using any agent.

95. 17:49

    There is no, uh, it can be implemented on Claude or Copilot because it's an approach. You can, uh, do, uh, in any way you want. At work, I use Copilot, uh, so I implemented this way using Copilot.

96. 18:01

    Uh, but because everybody, I, I believe, loves more Claude code, so, uh, I created this demo with Claude code. Uh, and you can see it's just a combination of skills, rules, uh, agents, and hooks, and some kind of a, a place to save the knowledge base.

97. 18:17

    On the middle pane, what you're seeing on the top is your monolith, basically. Uh, this is a representation of your Confluence, Slack, GitHub, and all, but just for the sake of demo, I just put some flat files that look like them.

98. 18:32

    Uh, so that is how your monolith, uh, knowledge base will look like. On the down, what you're seeing is on a live. So when it's solving a problem, how it is actually adding the new knowledge, uh, to it.

99. 18:43

    So this is how, uh, uh... So let me...

100. 18:49

     So what I'm gonna do is I'm going to go to an agent.

101. 18:55

     Okay.

102. 19:12

     I'm gonna basically give an incident problem to do the root cause analysis, right? So, okay, what I did is... Uh, you remember in the previous slide, there is a Jira ticket samples that I showed, right?

103. 19:22

     It's a combination of, uh, uh, knowledge that is documented, not documented things and all. So this incident also represent the same kind of, uh, combination. So there is some knowledge that is documented already on your monolith.

104. 19:35

     Some it is not there or outdated or things like that. And most of it, it doesn't, couldn't, uh, wouldn't able to find because it's never written down actually. So when I gave this problem, so it uses those skills that I have developed using this approach, and it will try to actually first go to your, uh, monolith actually

105. 19:56

     on the knowledge base and try to find information on what is, uh, there. So,

106. 20:02

     uh, think about it like this. So, uh, first part is retrieval. That means it's already doing what RAG and MCP is doing, first part. But what else it is doing is, after it fetches the data, what it will do with the data actually.

107. 20:16

     So that is a missing part. For example, when you give, uh, new Confluence links to a new employee, the employee goes there, looks into it, but doesn't find information.

108. 20:25

     But it-- he doesn't stop there actually. He continue asking questions so-- uh, to solving the problem then, then just adding more knowledge and things and all, right? Those are the next steps missing right now.

109. 20:37

     It's-- We just stop at retrieval. So this is the next three steps it does actually. So you can see the confidence score is almost one to five because it says

110. 20:50

     these are the particular terminologies. I don't understand actually these terminologies and these business logics, uh, is not needed. So one thing you need to look at here is whatever it has said, this is the undocumented information.

111. 21:04

     That means it was never written down. So unless you don't do this way, you will never know w-what is not documented. For example, if somebody says like, "Okay, there is documentation missing, we need to write."

112. 21:15

     Okay, what do you want me to write actually? So tho-- there is so much in the people's head, I can't write so much, right? So-somehow it has to surface.

113. 21:23

     So when you give a problem, it actually surfaces what is not documented. And it tells me, "Okay, this is missing. I need to have a new information there." Uh, so what I will do is, so it does all the three steps.

114. 21:36

     So then what I will do is, I already have a pre-prepared answer.

115. 21:49

     Very high level prepare, uh, answer I gave it, to it, uh, of like what is the missing information. So, okay, this is the missing information you asked me, uh, to solve this problem.

116. 21:59

     Can you solve this problem now? Uh, I didn't expected this one. Okay. Uh, notifications is... Yes. I'll just say yes.

117. 22:14

     It asked also what fictional name should I use for it.

118. 22:17

     Ah, okay. I didn't see. See, when I did the test run, it didn't ask me the questions.

119. 22:21

     Yeah.

120. 22:21

     Yeah. Let's see. It knows it is a demo and-

121. 22:23

     It's smart for- [laughs]

122. 22:25

     You know, to... I trained it too well, I guess. Okay.

123. 22:30

     No, it is already-- What it is doing is already...

124. 22:36

     So you can see on the live, it is a-already adding the entities that, uh, the new knowledge base has been come into the place.

125. 22:48

     So the knowledge base is, uh, managed as a file system, right? As a, as a system files.

126. 22:53

     For the demo, I'm just showing it as a file system, but it's basically your MCP servers, uh, uh, the data will stay in Confluence, Slack, or things. Uh, you can just plug in and use the same MCP servers or RAG and all.

127. 23:08

     So it no need to be a f- uh, flat file.

128. 23:10

     So you treat this as your persistence layer for, for this agent, right?

129. 23:15

     Uh-

130. 23:15

     Just like a mem... Do, do you use any like a memory tool for, for this?

131. 23:20

     Yeah. I, I will show you on the next slide like how I'm going to save, uh...

132. 23:29

     Okay. So it started from fifty-six entities or something with the, uh, this one, right? Now, one problem actually surfaced six entities that are never been documented, and when I gave that information to it, it is able to actually discover, curate another five or six, uh, new entities that were never documented.

133. 23:49

     So it does discovery of the gaps. It also gets information from me and also stores information, new information and all.

134. 23:59

     Uh, this is one. Okay. S- Next, let's see.

135. 24:05

     This is a busy window. I tried to actually do things, but, uh, it didn't worked out. Okay. So what you are seeing on the, on the window is like fourteen incidents.

136. 24:15

     You have seen one problem that I solved with an agent, right? The communication. What if I took like fourteen incidents, uh, and I just go and have fourteen cycles of this thing, and how it does?

137. 24:26

     So if you see on the left side, it was the first incident, right? So right now it has- 1.5 confidence and everything is critical. Every-- So basically, nothing is documented.

138. 24:38

     So everything is critical, high, uh, the data is missing. So I started giving answers to-- on the first incident, then I repeated for the same second and third and continuously for, like, 14 incidents.

139. 24:49

     But on the 14 incidents, it basically actually able to go to a confidence level of 4.4 because first it discovered on every incident, it got the list of answers for me, and also it documented everything for me.

140. 25:03

     So it gradually, from 1.4 to almost like five range of knowledge, uh, it improved. Uh, so if you look at the traditional way, in traditional way, what we do is we solve all the context problem, right?

141. 25:16

     We have to deal with it first, then I have to give it to agent. In this one, we are moving agent from a consumer to a knowledge manager. So you just don't consume from me.

142. 25:26

     I'm gonna tell you, but the whole knowledge management is also your job and you have to, uh, do it for me.

143. 25:41

     Okay. I think we can get back to the slides a bit. Okay. So what we have seen is, uh, we have-- I have run one cycle and also have s- shown w- how it, uh, look like when I run in like 15 or 16 different cycles, right?

144. 25:58

     But if you ha- want to do it manually, it would be really painful because [chuckles] I tried it after 15 cycles. Like, uh, nobody would like to actually sit with an agent and, uh, you know, you have an incident, but you won't be sit with your agent and keep actually asking questions and telling you about your problems, right?

145. 26:14

     So, uh, that is super painful. Uh, so but the thing is, we can automate this process. So this is where actually it's, it's really good and gets interesting. So here is the thing.

146. 26:27

     You all-- We already have all the work items, right? We have Jira, we have incidents, uh, we have, uh, customer support tickets like that. All those kind of a work items already there, right?

147. 26:39

     Sitting in the archive. So why can't we take, uh, them and actually use the framework, uh, and, uh, validate across your monolith database, run an automation, and see actually what is the state of your content actually right now?

148. 27:02

     Okay. Let me see. Let me show you how it looks like. So

149. 27:07

     rather than actually doing it manually, at a scale, if we do this approach, so how does it look like? So the demo that you're seeing is almost like everything is preset.

150. 27:17

     Uh, for example, I have the demo. I have like a platform operations agent and, uh, I'm saying, okay, these are the recent incidents. Let's say I have 20 recent past incidents I have, uh, like an MD file or a JSON file, right?

151. 27:31

     It has all the details of description of it, things and all, uh, comments and everything. And the rest of the files are your knowledge base. So it's, it's a file system, but you can also actually connect with the same way, the Confluence and things and all.

152. 27:44

     Just for a demo purpose, I'm just showing it as a flat files. Now, what I'm trying to do is, I'm gonna take all those incidents and validate each incident across my knowledge base and ask the agent, "Okay, tell me, uh, how much of the document is good, how much of the documentation is I can't trust it or

153. 28:04

     like old or outdated, and how much is actually missing, not documented as per these incidents?" So let me

154. 28:13

     run it. Okay, it will take some time. So it will take three steps actually. So one is, uh, it generates probes, which means a basic test it will write to actually test your knowledge.

155. 28:25

     Uh, then it will run those tests and then analyze the gaps actually.

156. 28:36

     Okay. It's a little bit hot in the room actually. [sighs]

157. 28:47

     Before identifying the gaps, it's just like a clever prompt, I imagine. [lip smack]

158. 28:53

     Uh, okay. For example, let's, let's say you have an incident called the notification service is not sending, uh, customer, uh, messages to,

159. 29:03

     uh, the SMS service, right? So the notification service then you mention that the agent sees, is there a documentation related to notification service?

160. 29:11

     Mm-hmm.

161. 29:11

     So it doesn't find, that means you never wrote a documentation on notification service. I do understand what is a customer, uh, SMSs things and all. The customer notification service when you mention, it's a gap because it's never documented.

162. 29:24

     Or it takes the customer notification service, goes to Confluence and sees like the documentation, how old it is. If it is, say, it's like, uh, it's like [REDACTED:age], it will tell you, "Look, I looked into it, it's like [REDACTED:age].

163. 29:37

     I don't know whether I need to trust this documentation or not," or like incomplete, uh, documentation. So if you see, it scored each incident it took it, and it looked at all the knowledge, uh, base that you have connected and have a consolidated list of like scoring of like, okay, partially the agents can handle, uh, the basic

164. 29:58

     edge cases of the incidents that you give, uh, because your knowledge base is not complete and it will show you how much of the tribal knowledge is missing. System information, business process, what are actually are missing from your, uh, uh, institutional knowledge when whatever is not documented.

165. 30:14

     Uh, these are the probes. And it will also identify, uh, what is critical and what is high. This is really important because let's say there is some kind of an, um, example of notification service which I mentioned, right?

166. 30:34

     It is repeatedly, uh, appearing in like 20 agents and you don't have-

167. 30:41

     This is the first that you need, you need to fix as per your documentation. So, uh, it will also help us actually understand when you're, uh, uh, breaking down your, uh, knowledge base, you need to understand what is critical actually, what I need to focus on first, what makes value for me.

168. 30:59

     So we'll organize into critical, high, medium. So this is what, like, uh, I showed the flat files, but you can also connect it to the various data sources that we have.

169. 31:15

     So the step one is basically what it does is demand extraction. That means every incident, it will extract, uh, the checklist of information, what is missing.

170. 31:26

     On the second step is what it will consolidate everything what is missing. So it will create like systems and APIs and all, and how many are clean, how many are stale, uh, which is incomplete, what is entirely missing, uh, something is tribal.

171. 31:42

     Uh, those kind of a classification it will also do.

172. 31:47

     And it will create a Kanban board for you. So what happens is, so if you want to fix your institutional knowledge base, basically, you just, just like Jira tickets, we finish it.

173. 31:56

     We actually has to document these missing pieces and all. And the, the moment you started to-- So it also saves in the context lake, so it also has to build its own, uh, uh, knowledge base.

174. 32:08

     And you can see the performance. So once you're fixing those tickets on the Kanban, uh,

175. 32:22

     the institutional knowledge. So that how-- So what we've seen is, one is the approach, first of all, which means not the pull, push approach, but the pull approach, how to do it.

176. 32:40

     One cycle or multiple cycle, how it look like. But if you put it in a scale of automation, then how much valuable it, uh, would be. [object clattering]

177. 32:55

     Okay. Now, uh, uh, the important question is-

178. 32:59

     Let's have another mic on you. Sorry about this.

179. 33:03

     Sure.

180. 33:04

     Double mic. Yeah, it's cutting out a bit, so I'm just gonna put that one on as well.

181. 33:09

     Should I do a voiceover later, actually?

182. 33:12

     Can you hear? Working well.

183. 33:13

     I think you need to repeat that from, [laughing] from the beginning.

184. 33:17

     I have the patience. Who has the patience for a second round? [laughs]

185. 33:22

     So the question is, so I was all, uh, all the time I was talking about, okay, it, uh, receives the context, we give the information, it will store it.

186. 33:32

     But the question is: Where does it actually store it? So I have a very opinionated opinion, uh, hear me out. Uh, I prefer, uh, it has to go to a GitHub repository, uh, because eventually somebody will actually come up with a, you know, 20 million seed funded, uh, SaaS solution for you.

187. 33:49

     Uh, but before that, I prefer, uh, to actually put it in GitHub as a repository. Why? Because if you look at it a scale, if you want to do this, there will be multiple agents, multiple teams actually contributing to the same knowledge base, and there will be conflicts and resolutions, right?

188. 34:06

     So the, uh, the easiest way to do is using GitHub because it actually comes with inbuilt, uh, uh, PR processes, uh, review processes, things and all. So if multiple domain expert are sitting and uploading the files or like agents are contributing to it, the most efficient way to manage is in a GitHub, something like a structure like

189. 34:27

     this. And the other advantage is also if you put it on GitHub, you can also publish it to Confluence later or like Slack la- uh, wherever you want to publish it to under the, uh, solution that you want to use.

190. 34:39

     Uh, so I prefer to have it on GitHub, but if you want to directly integrate it to Confluence and all, you can also and so do it.

191. 34:47

     Next is, uh, a meta-model. Uh, how many are aware of the word meta-model and the... Okay, maybe I can quickly show you how does it look like.

192. 35:01

     So meta-model is basically something like this, right? So, and, uh, how does your, uh, uh, domain actually structured around? Like, uh, is a business process or, uh, how it is related to systems?

193. 35:14

     How systems are related to, uh, an APIs? And how is this, uh, business jargon or like tech jargons are actually linked to which one? So these kind of a relationship, uh, meta-model is really important.

194. 35:27

     It's not necessary, uh, for the approach that I have proposed, but it's an add-on. And why, uh, you need to have this one is right now, think of it as like a map.

195. 35:38

     Right now, your agents doesn't have any map actually to navigate with your, uh, knowledge base. Basically, what you're doing is you're dumping like, uh, these many number of files, and it need to figure out which file I need to need, right?

196. 35:49

     But your file structure is actually a representation of your meta-model. It actually knows how to navigate. For example, let's say, can you fix this system? It will understand if I ch- make changes, which business processes will be affected and which APIs I need to change or like touch thing, these kind of things.

197. 36:07

     So it's also important to have a meta-model. If you have it,

198. 36:12

     uh, then it will produce more value. So I strongly prefer to have a meta-model along with this approach.

199. 36:26

     Okay. So the last part, uh, is what is the value it created? So, uh, there's a lot of slides that you have seen, a lot of demos that you've seen.

200. 36:36

     So personally, I need to also share like what is the value that I see when we, uh, I was using it or like, uh, the other people who I shared with already were using it, uh, came back with the feed- uh, feedback and told me.

201. 36:48

     Uh, first, the most valuable thing is Knowing the unknown. So what is never documented is something can be surfaced only by this approach actually. Uh, otherwise, you will just end up in, um, an endless Miro board of like, uh, putting tickets on, okay, this is missing, this is missing, I need to add it, I need to add

202. 37:07

     it, and, uh, keep on doing it. So this is the fastest and better way to discover, uh, with your previous work items and all, what is never documented, uh, things and all.

203. 37:16

     Uh, second is, uh, basically, I can now give work to agents rather than I do all those things. Like, uh, rather than I becom-- I give the agents all this information, let it manage my knowledge management.

204. 37:32

     I don't want to be the knowledge manager of it, so let, let it do it. So those are the two big values that I have seen. If you want to u-use it, I think like you will also see the-- those two as the most valuable.

205. 37:44

     Uh, but these are the other things, uh, what I seen.

206. 37:51

     Now, okay, so I also need to, uh, tell you like what is the, uh, drawbacks of also using it, right? First of all, if you are coming from a small team or like if you say like, "No, no, no, my documentation, my knowledge base is really good," I'm like super happy for you.

207. 38:07

     Uh, you are the lucky ones in this world right now with agents. Uh, for you, it might not be really relevant unless you have a very, uh, very complicated, uh, uh, documentation that you have.

208. 38:20

     Uh, second is, I already mentioned, the manual-- manually doing is, it's very painful. I don't prefer anyone to do it. If you want to just try it for testing purposes, you can also do it.

209. 38:31

     But, uh, automation is the most best way to actually use this one.

210. 38:36

     Uh, this is very early, this approach, so by tomorrow morning on YouTube, somebody would have already posted something differently, uh, better than me [chuckles]. So, uh, in the A-- uh, in the, uh, era of AI, nobody knows like, uh, how long a thesis or an approach or an app or product going to survive.

211. 38:54

     So for now, I see this is the best approach.

212. 38:59

     Okay, so the whole workshop, so we started with, uh, o-one pipeline, right, on the ROI. And so the demand-driven context actually sits between this monolith and also the retrieval layer actually.

213. 39:15

     And what it does is, uh, it actually helps you build curated context blocks for you. You can also think of it like a, uh, cache database that you have.

214. 39:26

     So every time your agent doesn't need to go and, you know, s-boil the ocean for, uh, fixing an issue, rather than if you have a good context block of information, most of the time, eighty percent of the time that can be usable.

215. 39:39

     Because what I also believe is, it's always the eighty/twenty percent rule. So twenty percent of your documentation is most useful, eighty percent is some corner cases you have to look into it.

216. 39:50

     So rather than giving hundred percent of, uh, things, you need to figure out what is my twenty percent of that, uh, that is super helpful for agent and have it like a cache database, uh, the context block of it using it.

217. 40:02

     And rest of it, you can leave it like, uh, links. So whenever agent feels I need more information, then only it can go and check the, the whole, uh, monolithical, uh, institutional knowledge.

218. 40:15

     Okay. So from here, what you can take from this workshop is three things. One, uh, I hope, uh, I makes-- I made sense of this approach. So, uh, there is a GitHub repo, uh, uh, which I detailed it out and also a starter guide on it one, if you want to go home and try with it, you

219. 40:34

     can, you can try it. Uh, you already have know how the framework works, so you want to go home and just remix the whole approach, you can do it and let me also know.

220. 40:43

     Uh, I'll leave this one and I'll join with you for contribution. Uh, you have a context ga-gap scanner that I showed you, which is live already with presets. Uh, I think I added like twenty dollars on it, so hit it as much as possible as well.

221. 40:57

     Two hundred, right? Okay. Okay. So after twenty dollars, so, uh, first come, first served [chuckles].

222. 41:04

     Uh, so all these three you can use, uh, you can take away from this workshop.

223. 41:11

     Okay. So because this is a workshop, so I also would like to

224. 41:17

     want you to try something. Uh, what you can try is three things. One is either, uh, if, if you say like, "You know what? I'm so, so tired already.

225. 41:28

     It's almost like four, uh, it's almost about to go for a party. I don't wanna do it." So you can just go to the context gap scanner. Uh, everything is a preset here.

226. 41:37

     You can just try it out, hit it, and see how it works. If you think it can be done better, let me know so that we can work together.

227. 41:46

     Uh, or otherwise, uh, let's say, "No, I've-- I'm very technical, I want to know how it works under the hood." Uh, this is a GitHub repository. Uh, it's, it's under, uh, maybe I'll just take this out.

228. 42:05

     Uh, this is a GitHub repository, and it has all the information. Uh, plus there is a starter guide also if you wanna, if you wanna try it out. But, uh, if you still feel like, "No, I want much more simpler,"

229. 42:30

     you can also try this one. So you don't need to do anything. Basically, take this prompt, uh, take one of your Jira ticket or incident that you have right now.

230. 42:40

     If you already built MCP servers, uh, or like, uh, any other kind of a things, you just use that, uh, prompt, give it to your agent, uh, uh, with the incident or a Jira ticket and ask it Uh, give me the quality of the knowledge base that I have as per this incident or, uh, Jira ticket in

231. 42:59

     this way, and see how man- how much of it comes in the red, which is never documented. So you can, you can try also the simple one.

232. 43:09

     I'll just leave it like this. Maybe you can take a picture. Maybe I can switch to the slide if any- any- anyone want to.

233. 43:23

     Cool.

234. 43:29

     So while there's time for the hands-on with scanner, yeah.

235. 43:33

     This is yours line. You can-

236. 43:35

     Yeah. So I see some drawbacks in this approach.

237. 43:38

     All right.

238. 43:39

     And I find it very interesting. So my f- my first question is, have you already used this way of working at scale? Or... Because we've seen mostly toy examples, right?

239. 43:47

     Yep, yep.

240. 43:48

     In your, uh, slides so far.

241. 43:49

     Uh, I used it, uh, not at a scale. Uh, I started a bit simpler because you also need to see what is the scope of it. Let's say I have an enterprise, and I try it an enterprise level, I can't do it because it's multiple domains things and all.

242. 44:04

     Even if I do it at domain level, I need to understand. I tried it at domain level. Then even at a domain level, there is so much of a domain expertise I need to fill it up and, uh, uh, fill those gaps.

243. 44:16

     So again, cut down into maybe what is the smallest team that I have and the smallest teams Jira tickets, the smallest teams incidents, and the teams Confluence page, uh, with a bit of a scope.

244. 44:29

     Then if I s- drill down the scope, then I feel like it's more, uh, fast, more useful.

245. 44:35

     Mm-hmm.

246. 44:35

     But if I do it at a bigger scope, what happens is not one person has the whole, uh, domain expertise. So basically it again becomes like, uh, somebody has to come and, uh, you know, five or six people has to sit down and, uh, start doing these things.

247. 44:50

     Yeah. I'm, I'm a bit concerned that this might, uh, denial of service attack your, your team members-

248. 44:55

     Mm-hmm

249. 44:55

     ... in a certain way because our LLMs are fine-tuned to, uh, keep eliciting information, to keep, uh, getting more information out of us to ask follow-up questions.

250. 45:05

     Uh-huh. Okay.

251. 45:06

     Um, so I think it will be hard on the engineers that have to do the question answering. Um, and secondly, the, the scanner is nice, but that still builds on, uh, the assumption that all of your team members and the rest of the enterprise-

252. 45:18

     Mm-hmm

253. 45:19

     ... are still using your enterprise IT, well, as planned, that they're actually filling in their tickets with all the details and-

254. 45:26

     Yep

255. 45:26

     ... et cetera. And I know from practice that it is most of the time not the case.

256. 45:31

     Uh, that is true. That is true. I agree with you. Uh, even if I go-- My, my assumption is also even if I go to a leadership to buy in, like, "Hey, can you give me a bandwidth?"

257. 45:40

     Or like, uh, you know, I need these people to actually sit and fix the context. I don't think right at this point of time nobody will do. But I think it will happen because, uh, slowly, I think we are slowly moving towards an agent managers where, uh, agents are becoming semi-autonomous or autonomous, and we manage them.

258. 46:00

     But at that certain point of time, somebody has to fix that knowledge because it's not gonna come from anywhere. You have to. So then the enterprise focus will shift towards the gap.

259. 46:10

     Uh, that's what I started saying. I don't think nobody is looking into the problem yet. Uh, everybody is very focused with agent, how good the agent is, how good the retrieval is, but how good the context is, uh, y- y- you're not solving.

260. 46:26

     It to-- I think like in, down the line in a year or so, I think people will realize the importance of it and, uh, the Kanban board will definitely come into reality actually very soon.

261. 46:37

     Thank you.

262. 46:38

     Yep. Thanks. Yep.

263. 46:39

     Yeah. I think actually building on the same points, I think when we look at large enterprise, actually the s- search of truth is not actually the documentation, it's actually the code.

264. 46:48

     Just wondering, have you applied it to the code base?

265. 46:51

     Um, I did. Uh, I also applied the code base, uh, but I got a mixed result when I, uh... So, so the-- here is the thing. What happened is when I only use code base, uh, it is particularly good, or when I only use Confluence or like textual data, uh, like, uh, uh, it gives good results.

266. 47:11

     But when I combine it, somehow actually, uh, it conflicts because it, it creates a theory out of the GitHub repository, but the same GitHub repository documentation is also on Confluence.

267. 47:24

     So there it gets a conflict of, okay, what is the source of truth? Code says this. Should I implement it this way as per the documentation? So then aga- again, I need to create an additional skill or rules like, okay, what is the ranking that you need to give?

268. 47:37

     If you see it in GitHub, that means that is the source of truth. Or if you see it-- If you don't see it, then you have to, uh, look the information in Confluence things and all.

269. 47:48

     But those are still... I'm trying to fix those things actually. So seeing the gaps and fix those things. But I definitely see, uh, that issue combining those two.

270. 47:57

     And the second question is, um, quite interestingly, has actually applied the same approach with skills. Because what we find out is actually like you have your, um, kind of like your, um, like the, the process starts like by running agents, which is bring in a context and identify the right skills that need to be used, right?

271. 48:15

     Mm-hmm.

272. 48:15

     Then you go do the task and you fail-

273. 48:17

     Mm-hmm

274. 48:17

     ... I think, right? The moment you fail, you identify what it you need to solve.

275. 48:20

     Mm-hmm.

276. 48:21

     You go back, you curate, you, you fix, and then you're fixing the knowledge kind of base. Once we find out actually if we go back and fix the skill-

277. 48:30

     Okay

278. 48:31

     ... that increments kind of the, the next iteration. Sorry. So I'm not sure if skill-- 'cause I saw you had some skill, but-

279. 48:37

     Yep

280. 48:37

     ... is skills part of the, uh, the iteration loop or how's it-

281. 48:41

     Uh, I think right now skill that I have built is static, but what you are more proposing, if I'm not wrong, it's like se- evolving skill, right? If the skill fails, it has to evolve, right.

282. 48:51

     Uh, I agree with you. I never tried it, but I think like it has to be, uh, like that because I'm also more concentrating on, um, how to do it at scale.

283. 48:59

     Uh, the reason is also Uh, I, I want, I want the context to be fixed before retrieval itself,

284. 49:08

     not during operational. So first, when I started with it, I, I started doing with when operational, which means, oh, I have a work item, I will assign to it, it will fail, then I'll start giving context and all.

285. 49:20

     But it takes a lot of time, it takes a lot of patience for me. So rather than doing it, you know what? I'm gonna fix the context, but before retrieval.

286. 49:28

     So if I can, uh, uh, uh, while I was, uh, answering her question, if you take a team, the, the context that you need to fix is very small.

287. 49:37

     So you can use a, uh, context gap scanner, uh, kind of a thing. And maybe if you're good, have a good domain expert, I think like couple of weeks you can actually fix your documentation.

288. 49:49

     Not like 100%, at least like 60, 70, 80% of a good quality that you can al-already build it. So my proposal would always be don't do it at an operational level, uh, at an, a real-time level, but do it before retrieval, uh, itself.

289. 50:04

     That is much better than this approach. Uh, yep.

290. 50:08

     I, I have a question. The retrieval part, like, especially at scale, if you have a lot of documentation and that documentation sitting in a GitHub repo. You may have situations where you ask questions that may need, I don't know, like five or six different docs.

291. 50:22

     So you will have like LLM going there, reading all the docs. Like, this takes time for retrieval of information, and that takes a lot of context also for the-

292. 50:33

     Uh-

293. 50:33

     ... LLM to get the right information

294. 50:35

     ... right now, after Claude code announced one million of, uh, tokens in the context window, I don't... No, I don't have any problem. So I calculated it. Uh, at an average, it's like, uh, 96K tokens, because I tried with different domains actually.

295. 50:49

     Per domain, I see like around 96K tokens, uh, if I consolidate everything like Confluence things and all. Uh, so easily fits in the context window actually. Uh, I tried to do some experimentation around, you know, a graph RAG, put them there, rather than just take all the files, use a graph RAG, understand the intent.

296. 51:07

     But for me, just putting the whole context right now in the window, uh, gives you more results than actually doing, uh, uh, RAG. Unless you have a very big, uh, almost around a million tokens of a context that you want to fit in, maybe then you have to use a bit more, uh, retrieval mechanisms between it.

297. 51:28

     But otherwise, I think like, uh, it should be fine.

298. 51:32

     I, I have a question. Uh, I opened your paper, and could you explain this graph, like, uh, comparison between different techniques like domain knowledge, strategy knowledge, axis, um...

299. 51:47

     Uh, which one? Yep, sure.

300. 51:49

     This one.

301. 51:51

     Okay. So, uh, I also did the citations from other papers.

302. 51:55

     Okay.

303. 51:55

     Uh, so not directly related, but, uh, you have the paper of ACE, uh, uh, which is also, uh, does a similar thing. So, uh, but ACE is not exactly into, uh,

304. 52:10

     uh, how do you say, uh, discovery and curation actually. Uh, if I remember it correctly, uh, maybe I need to refresh my memory.

305. 52:18

     What do you, what do you mean between, uh... The difference between domain knowledge and strategy knowledge in here?

306. 52:24

     Okay. So s-strategic knowledge... Okay, so what ACE and all, uh, are doing is, when you are trying to have a conversation with, uh, AI, uh, you can see in the cloud code and all, it updates its, uh, memory or like the relationship with you or things like that, right?

307. 52:42

     Mm-hmm.

308. 52:42

     So, and also from the chat history, it understands what is the most important context I need to remember, those kind of things. So when you are in communication with it, that operational conversations with AI improvement, they propose.

309. 52:55

     So what I, I, um, my proposal is not based upon your conversation with an AI, but rather your domain knowledge which is documented actually.

310. 53:08

     Mm-hmm. Okay.

311. 53:10

     Uh, somebody else has a question. Sorry.

312. 53:14

     Um, when, uh, relating to remote knowledge, uh, things on Confluence-

313. 53:17

     Mm-hmm

314. 53:18

     ... and other stuff, um, how do you ensure that your agent only points to the updated or like the, the, the relevant documentation as-

315. 53:28

     O-

316. 53:28

     ... because in your local file system, you, you have tags like outdated-

317. 53:31

     Mm-hmm

318. 53:31

     ... or piece of that, but remotely that's harder to tag, I guess. Right? So...

319. 53:35

     Okay. So when I wrote a pipeline for extracting from Confluence, it also allows actually to give you a date and also last updated, who created kind of an analytics on, on the space.

320. 53:45

     So you can use it, uh, to actually put a threshold of like, okay, on this particular date, w-whatever is, is old, consider it as an outdated one. And let me know.

321. 53:56

     Don't just consider it as an outdated one. You let me know, because sometimes the document can be stale for so long, but it could be an important document actually.

322. 54:04

     So it lets you know, but not like take decisions actually right now on this one. So you decide, uh, which one is stale and which one is not.

323. 54:11

     You, you don't have like an intermediate layer where you... In the repo you store this is still available.

324. 54:17

     Okay. So, uh, when it is curating the context, uh, also it updates with the, uh, date and also the state of the document, like stale, active, and clean. So it also looks into, okay, this is stale, I know I'm not gonna touch it, and I'll just go to just look for any other new other documents are there,

325. 54:33

     uh, in this one.

326. 54:34

     Great. Thanks.

327. 54:36

     Uh, did you think about, uh, how to manage access or permissions later to this knowledge? Like if you have some-

328. 54:42

     Mm-hmm

329. 54:42

     ... knowledge in the company that's sensitive and on- and only specific people can get access to it. Because currently I guess you just have all the knowledge and everything is accessible, but, uh...

330. 54:50

     Okay. So because it's not a product or a SaaS solution, it's just basically GitHub- For me right now, permissions and things are not difficult to implement because GitHub, uh, out of the box gives me who I can give the permission to as GitHub, who can have write, read access things and all, who can merge, uh, those things

331. 55:09

     and all. But in case if it evolves into a product, and for example, context, uh, gap scanner as a product, and I want you to test it because the, the reason why I was using presets, uh, for this, uh, workshop, not actually asking you to upload the files is because I don't want to take your IP data

332. 55:25

     on this one, right? So unless it becomes a product, you don't have any problem, uh, GitHub and all. But if you have a SaaS solution for this one, uh, then it's between how the SaaS solution will manage it ri- right now.

333. 55:39

     But the approach has nothing to do with the, uh, access things and all. How you implement those access on, on the knowledge base is up, up to you.

334. 55:48

     You have a question.

335. 55:50

     Yes. So this is of course about documentation, but did you give any consideration about using it on some like central tooling that a company would use? Like let's say that you have a platform team and you have a CLI that the different teams are using-

336. 56:05

     Mm-hmm

337. 56:05

     ... and so now it's used by different agents, right?

338. 56:08

     Okay.

339. 56:08

     And so the agents can also be like, "Well, this action is available for a resource, but I don't want to do five hundred calls just because I have a list of five hundred resource."

340. 56:16

     Okay.

341. 56:16

     It would be nice if the tool could do that. Uh, I don't know if, if you've given any consideration to this.

342. 56:21

     Uh, I think that is how it has to work. Uh, in an organism, you need to have a central solution for it. Uh, but how you want to do the solution is up to the organization.

343. 56:31

     For example, we are doing agile, right? So agile can do by Scrum, Kanban, or like Lean or something, and also you can do different apps to do it. The, the process is the same, but how you do it, which method you will choose, and which app that will you choose in your organization is different.

344. 56:47

     In the same way, what we have discussed is the approach. If you want to put it in the organization, you can use the approach, and you do-- you can do it in whichever way you want.

345. 56:59

     My point is more like, so with this you can identify gaps in your documentation-

346. 57:02

     Uh-huh

347. 57:03

     ... right? Could you use it to identify gaps on your tooling, so to say?

348. 57:09

     Okay. Uh, when you say tooling, it's the agentic-

349. 57:12

     Internal tools that, I don't know, maybe a team is building for the rest of the company.

350. 57:17

     Just infrastructure in general, right?

351. 57:18

     Maybe, yeah.

352. 57:20

     Uh, can you give me an example of, uh, like, uh, how it could-

353. 57:23

     Let's say that, uh, I don't know, you build some sort of abstraction on top of Kubernetes.

354. 57:27

     Uh-huh. Okay.

355. 57:28

     You don't want your developers to necessarily know what to do with that.

356. 57:31

     Okay.

357. 57:31

     And then you have a different CLI, or you, you have something, right? Um, but then like I say, I don't know, maybe you thought that they would list one of your custom applications or corporate applications-

358. 57:43

     Mm-hmm

359. 57:43

     ... one by one, but a team has grown into using more of that, and suddenly they have a lot, and they don't want to do that many calls, or perhaps even the agent is like, "Well, this is inefficient.

360. 57:53

     I would like this internal tool to work in a different way."

361. 57:56

     Okay.

362. 57:57

     And, and in that way, you would identify like gap in the tool or, or a performance improvement, kind of like these tasks for documentation.

363. 58:04

     Could be extended actually.

364. 58:06

     Mm.

365. 58:07

     So, uh, because, uh, we have seen the business processes also, right? So it can also document business process. Well, business process is nothing but how, uh, the process in the application it actually runs and does things, right?

366. 58:19

     So you can extend it to also find out the gaps in the business process or like how it works. It could be an extension to it.

367. 58:26

     Yeah. Okay. Yeah. Thank you.

368. 58:29

     How would you ensure that maintenance won't kill you?

369. 58:33

     Uh, sorry?

370. 58:34

     How would you ensure that maintenance won't kill you? Because the knowledge, the knowledge of a company changes with time.

371. 58:41

     Yeah.

372. 58:41

     So if the answer for a question today is B, tomorrow B1, on Friday it could be C, right?

373. 58:48

     Mm-hmm.

374. 58:49

     You have to-- at C, you have, you, you have to identify B1 and B and overwrite them. If it's stored as text, that's a problem.

375. 58:59

     Okay. So, uh, when I showed the context gap scanner, you also saw like an indicator of duplication, right? So if today you have a document, tomorrow you have version two point zero and something else, actually, it will find out the same information is having in three different...

376. 59:15

     It al- it also will find it. If you have only one, it is changed, it will take the latest updated one, because as a human, you changed it. So it will take it as a source of truth, right?

377. 59:25

     But if you have three versions of the same document, that's a duplication, and it will flag it as duplicate.

378. 59:33

     Right. But it, it's a search problem. How, how do you ensure it, it performs well? Let's say you have a document of one hundred thousand words.

379. 59:42

     Uh-huh.

380. 59:43

     You just say you just change a, a word like, uh, passport, let's say. It won't be there, but for example, right, it has changed.

381. 59:49

     Okay.

382. 59:50

     So you have to find the specific word, compare those three, in three documents-

383. 59:56

     Mm-hmm

384. 59:56

     ... check them and re-replace them, sort of, right? How it's feasible. Which tools would you use ma- to ensure it will, it will, it will not kill you cost-wise?

385. 1:00:08

     Um, I didn't quite get your question, actually. Uh, is it like the token usage you're worried about? Like, uh, the, that many tokens that we used? Uh, how, how is-- what is the cost saving, uh-

386. 1:00:20

     Yeah. Uh, uh, precisely. It, it, it will, it will grow-

387. 1:00:24

     Mm-hmm

388. 1:00:24

     ... presumably be, be big. And if there are a lot of changes, you would have to maintain, ma-maintain that whole database, right? So it has to be-

389. 1:00:31

     Mm, okay

390. 1:00:32

     ... some sort of structure. I don't know, maybe Graph or Ivy-

391. 1:00:35

     Okay

392. 1:00:35

     ... or whatever, right? Um, my question is, well, 'cause what you presented is sort of a happy path where you have a gap-

393. 1:00:45

     Mm-hmm

394. 1:00:45

     ... you fill it, and then you reuse it. But in a while, you will have a bigger problem where you pretend to have that gap filled, but actually it doesn't contain up-to-date information

395. 1:00:59

     Done contains wrong information, right? So you want to preserve that

396. 1:01:03

     Ah, okay. Okay. Uh, so it can flag as per when it is created or the last updated. You can set such kind of a filters. But let's say you have a latest document which has a wrong information,

397. 1:01:19

     right?

398. 1:01:19

     But you don't know that, right?

399. 1:01:20

     No, that, that's true. Uh, but as, for example, as a human being, right? So you go and look into documentation. You told somebody to look into documentation, and the person looked into the documentation, and as per the documentation, this is being implemented in this way.

400. 1:01:33

     The person will do it, right? It's not an agent or a human issue.

401. 1:01:36

     But that's way too late, right? I mean-

402. 1:01:38

     Okay

403. 1:01:38

     ... you solved one ticket, one Jira ticket-

404. 1:01:40

     Mm-hmm

405. 1:01:40

     ... and the agent is trying to solve the second one. The assumption is the knowledge is pristine.

406. 1:01:46

     Okay.

407. 1:01:46

     So it's solving it, but the solution is wrong. So that's too late, right?

408. 1:01:52

     Okay.

409. 1:01:52

     So you say there is a scanner. Fine. Do you run it daily? How much will it cost you?

410. 1:01:58

     You, you, you can have another process spawned that would try to see, like, last updated-

411. 1:02:03

     Yeah

412. 1:02:04

     ... uh, try to see on some cadence, try to run a process that will up-update the knowledge.

413. 1:02:08

     Fantastic. How much will it cost you? Yeah.

414. 1:02:11

     Uh, for ex- for example, uh... Sorry, you were saying?

415. 1:02:14

     Less than, than doing s- more meetings, uh, every day to onboard someone or-

416. 1:02:19

     How, how-

417. 1:02:20

     ... update the knowledge

418. 1:02:20

     ... that's a strong claim.

419. 1:02:22

     But I don't think it will cost that much as is-

420. 1:02:24

     It's on-premise of AI.

421. 1:02:25

     Yep. As I said, like, uh, when I tested it, uh, there is none of the domains which cross more than 100K tokens actually. So I don't think we will, for example, context gap scanner, right?

422. 1:02:37

     You-- I don't think, like, you have to do it, like, on a daily basis or anything. Even if you run daily basis, like, 100 tokens and do one scan.

423. 1:02:46

     For example, right, if you try to start hitting all of them, uh, all of you, the context gap scanner, I think you can't even burn, like, one, $1. I think so, if I'm not wrong.

424. 1:02:56

     It, it already had like... Oh, you're ready. Okay. Uh, okay. I'll, I'll cancel the subscription now. [laughs]

425. 1:03:03

     But I-

426. 1:03:03

     Because then I go back and depend two things. One is the complexity of organization you're trying to solve. Like a good example, like say airline-

427. 1:03:10

     Mm-hmm

428. 1:03:11

     ... industry. We're talking about like hundreds if not thousands of like repository system documentation third parties. So I agree with you. If you're in a s-specific domain-

429. 1:03:20

     Mm-hmm

430. 1:03:20

     ... solvable. The moment you scale, you will have to solve this, uh, cost question.

431. 1:03:24

     Okay.

432. 1:03:25

     Which is, which is gonna be depend also how fast the, um... To, to his point, like how fast the, the, the data change-

433. 1:03:32

     Mm-hmm

434. 1:03:32

     ... which I think most of the time not that much. So the moment you get to like 80%, ninety percent, you just continue kind of evolving the-

435. 1:03:38

     Yep

436. 1:03:38

     ... last one. Uh, I see it's different use case, different use case.

437. 1:03:42

     Yep, use case by use case. Yep. Yeah. Uh, any other questions?

438. 1:03:46

     Yeah. I was curious, so I, I ran the scanner for like finances domain, right?

439. 1:03:50

     Okay.

440. 1:03:51

     It has a bunch of recommendations. How do I know that that's enough?

441. 1:03:56

     Uh-

442. 1:03:56

     It actually makes it even better.

443. 1:03:58

     It, it-

444. 1:03:58

     How did you assess that?

445. 1:04:00

     It actually tries to detail out as much as possible. Right now, I haven't actually exposed everything what it did just for the UI purpose. Uh, but the, all the per ticket, what it actually found, like, uh, it writes like 100 or like 150 lines of, uh, markdown files and save it somewhere.

446. 1:04:18

     So, uh, that gives you more details in case if you want to know actually. So for, for the demo, I just put the, you know, nice UX stuff on, on top of it, but you also have a detailed information at, uh, test.

447. 1:04:31

     Anyone else has any questions?

448. 1:04:33

     Can I have another one?

449. 1:04:34

     Yeah, sure. Go on.

450. 1:04:35

     Right. So you-

451. 1:04:36

     Easy one, huh? [laughs] At least try. [laughs]

452. 1:04:41

     Yeah. You had a chart where claims or red line, that's forty percent-

453. 1:04:46

     Yeah

454. 1:04:46

     ... that's not gap, right? And if I got the presentation right, your claim is you define, like, amount of Jira tickets you can fill that gap.

455. 1:04:56

     Not just-

456. 1:04:57

     You said a couple of weeks, right? Couple of weeks and you will have filled that knowledge base.

457. 1:05:01

     Not fill the gaps, uh, but discover. But if you scope down to a team, uh, within weeks you can do it.

458. 1:05:08

     Right. Because, I mean, if you wouldn't fill it and would have to keep asking those questions, then solution doesn't make sense, right? So at some point you have the knowledge base that's created, available, and delivered by-

459. 1:05:21

     Yeah. So you do it one time, uh, first of all, or like m-multiple times at first. See the whole picture first of all. What is the state of your knowledge base?

460. 1:05:31

     First, fix it, uh, at that level. Then you go into operations, right? You still can actually, uh, also-

461. 1:05:37

     Sure. Having the-

462. 1:05:38

     You, you can still continue doing it, uh, with the agent, with skills.

463. 1:05:42

     But at some point you will, you will, you will kind of squeeze that, that forty percent gap, right?

464. 1:05:46

     Yep.

465. 1:05:46

     And, um, I think the same process of kind of providing that knowledge, because that knowledge you say is in people's heads, right?

466. 1:05:55

     Uh-huh.

467. 1:05:56

     It's not documented.

468. 1:05:56

     Oh, okay. Yeah, yeah.

469. 1:05:57

     That filling happens pretty frequently when a new member joins the organization.

470. 1:06:03

     Mm-hmm.

471. 1:06:03

     So wouldn't a replacement that be just get Zoom calls, transfer them, and use them as a source, assuming all the calls or all the knowledge transfers or maybe introductions happen over Zoom?

472. 1:06:19

     Mm.

473. 1:06:20

     Or Teams or whatever the communication. Like if there's a new member joining-

474. 1:06:24

     Mm-hmm

475. 1:06:24

     ... asking questions and you have access to that-

476. 1:06:27

     You also have access to Zoom also

477. 1:06:29

     ... that's also a solution.

478. 1:06:31

     Okay. So you mean like, uh, you can also give all the transcripts rather than, uh, uh, doing the cycle, you mean?

479. 1:06:38

     Yeah.

480. 1:06:38

     Uh, that can also be done, uh, if you're only have the... All the time you have a discussion in the meeting, everything is documented in meetings transcript itself. But I don't think like that's the same case for everyone at least.

481. 1:06:51

     Uh, uh, for-

482. 1:06:53

     Sure. But that should be easier to solve, right? If, if, if... 'Cause if I come across a Jira ticket that I don't know or I'm unsure about, I will call someone or chat with someone.

483. 1:07:04

     I think the amount of time people spending in Teams-

484. 1:07:07

     Absolutely

485. 1:07:07

     ... if you do use the transcripts, actually, those are the ones actually who, which, uh, have more tokens actually. There are so many useless meetings, uh, s- that transcripts actually, uh-

486. 1:07:20

     Totally agree. But the compression happens only once, and then that's part of the database.

487. 1:07:25

     Could be. Uh, again, it depends upon institution to institution, right? So are you like more into meetings, have solving problems within the conversations, and those conversation has the data?

488. 1:07:35

     Or like your Confluence or things has the data. If you have it, use th-those transcripts as your kn- knowledge base. And the, at the same time, like, yeah, the compression actually, that works actually.

489. 1:07:44

     That, that is more useful. Yep. Anyone else? Any questions? No. All good.

490. 1:07:54

     Uh, then, uh, thank you so much, uh, for attending the session. [upbeat music]