1 00:00:02,226 --> 00:00:03,186 Hello, everyone. 2 00:00:03,586 --> 00:00:08,886 I would like to walk you through that how we can transition in AI agents from an experiment 3 00:00:08,886 --> 00:00:10,426 to the real enterprise environment. 4 00:00:10,806 --> 00:00:13,866 So before I begin, a quick introduction about me. 5 00:00:14,006 --> 00:00:14,906 My name is Harish Gaggar. 6 00:00:15,126 --> 00:00:18,026 I'm a senior staff software engineer at Intuit Credit Karma, 7 00:00:18,306 --> 00:00:22,566 where I built and own the AI analytics infrastructure and services. 8 00:00:23,446 --> 00:00:28,006 And beside that, I'm advisor, author, researchers, 9 00:00:28,706 --> 00:00:32,086 and also a program committee members for various organizations 10 00:00:32,086 --> 00:00:34,086 where I mentor the young researchers as well. 11 00:00:34,906 --> 00:00:35,566 All right. 12 00:00:36,626 --> 00:00:40,766 So let's walk through the agenda here. 13 00:00:41,046 --> 00:00:54,306 So first, we would like to talk about what are the challenges we have when we build an AI agents and deploy into the production systems and how we can build a different framework to address that different challenges here. 14 00:00:54,826 --> 00:00:57,366 Then how the architecture looks like for it. 15 00:00:57,886 --> 00:01:03,066 Then how you can build your own MCP servers and MCP gateway. 16 00:01:03,926 --> 00:01:07,506 and then some monitoring on top of that gateway as well. 17 00:01:08,166 --> 00:01:13,366 Then further, we're going to talk about what are the different type of agent framework you can have 18 00:01:13,366 --> 00:01:18,906 and then some additional framework around managing the memories, tracking and eval, 19 00:01:19,146 --> 00:01:20,026 how you can 20 00:01:20,026 --> 00:01:25,666 block the PII data going before the LLMs, and then Context Optimizer to save your token. 21 00:01:27,146 --> 00:01:37,486 So without delay, let me walk you through the challenge. Of course, there are many agents that generally get failed in production because their foundation controls are missing. 22 00:01:38,286 --> 00:01:54,666 And the key reason behind is that, you know, like most of the people are missing the gateway to make sure that the agent can, you know, communicate to the gateway to get authenticated first with the right auth providers so that they can redirect to the right MCP servers. 23 00:01:55,066 --> 00:01:59,786 Right. Second, that MCP servers are missing the right tool to execute your query. 24 00:02:00,546 --> 00:02:11,746 Third, do we have like enough tracking and evaluation system to make sure that every query when you are sending to the agents, whether they've been tracked or not, and are you evaluating their quality response or not? 25 00:02:12,046 --> 00:02:15,266 Right. Do we have any memory management system or not? 26 00:02:15,446 --> 00:02:23,326 In terms of, let's say you are doing the interaction with agent for long or for many days, is it able to recall that what you have been asked in the past? 27 00:02:23,666 --> 00:02:29,246 Right. And then there are other framework, of course, like do you have a context optimizers to save the token? 28 00:02:29,246 --> 00:02:31,486 or do you have enough production in place 29 00:02:31,486 --> 00:02:34,546 to make sure that any secrets should not go into LLM 30 00:02:34,546 --> 00:02:38,286 or tracking or tracing or any memory DB or so, right? 31 00:02:38,566 --> 00:02:40,466 So a lot of open questions. 32 00:02:40,806 --> 00:02:43,086 So in order to resolve this, 33 00:02:43,626 --> 00:02:45,306 we are not going to, you know, 34 00:02:45,386 --> 00:02:47,806 like build a magical agent here in this demo. 35 00:02:48,426 --> 00:02:51,186 We're going to talk about like different framework 36 00:02:51,186 --> 00:02:52,886 that you can build to address this. 37 00:02:53,486 --> 00:02:55,186 So first is the agent framework. 38 00:02:55,666 --> 00:02:57,866 This is where like we use the, 39 00:02:57,866 --> 00:03:02,326 So like we choose to use the LangGraph framework, which is provided by LangChain community. 40 00:03:02,986 --> 00:03:08,066 This has helped us in building the task-specific node based on your questions, 41 00:03:08,266 --> 00:03:14,606 and which is easy to plug in any like marketplace plugins or skills and prompts 42 00:03:14,606 --> 00:03:18,906 to make sure that agents respond efficiently based on your query. 43 00:03:19,326 --> 00:03:26,326 Further, the second part is we are also building the MCP gateway to make sure that whenever agents make the call 44 00:03:26,326 --> 00:03:29,066 or any service account from that agent make the call, 45 00:03:29,186 --> 00:03:31,246 make sure that they are securely get authenticated 46 00:03:31,246 --> 00:03:32,426 to that MCP gateway 47 00:03:33,106 --> 00:03:35,866 and then get redirected to the right MCP servers. 48 00:03:36,326 --> 00:03:38,086 Third is the customized MCP, 49 00:03:38,246 --> 00:03:40,766 which means that you are building your own MCP servers 50 00:03:40,766 --> 00:03:43,806 with the right tool in place to execute your query. 51 00:03:43,986 --> 00:03:45,506 And then finally, some audit 52 00:03:45,506 --> 00:03:49,046 on top of what actually operation you're performing, right? 53 00:03:49,246 --> 00:03:51,726 And then we talk about some additional framework 54 00:03:51,726 --> 00:03:55,086 around, you know, like how you can block the PIA secrets, 55 00:03:55,086 --> 00:03:59,426 how you're going to store the tracking value into the cloud, and how you're going to 56 00:04:00,286 --> 00:04:06,446 purchase the memories, especially any threads and checkpoints on every context. And then also how 57 00:04:06,446 --> 00:04:13,566 you can optimize overall your context to make sure that you don't burn too many tokens when 58 00:04:13,566 --> 00:04:21,866 you run your query. All right. So we'll talk about now the agent infrastructure. So firstly, we'll 59 00:04:21,866 --> 00:04:27,526 go through the local development. The way the approach should be is that, you know, like 60 00:04:27,526 --> 00:04:31,786 whenever you build an agent, right, first you have to register your agent into some system. 61 00:04:32,006 --> 00:04:37,886 You usually tell that, hey, look, this is my, the use case, and these are the use case going to use 62 00:04:37,886 --> 00:04:43,946 for this particular vertical or system. And you feed all those information and generate an ID for 63 00:04:43,946 --> 00:04:48,966 your agent, right? Once you have that, then you can have a dev portal, internal portal, where you 64 00:04:48,966 --> 00:04:53,006 You can ingest all the details saying that, hey, look, this is my agent use case. 65 00:04:53,126 --> 00:04:53,966 This is my agent ID. 66 00:04:54,306 --> 00:04:57,846 This is the framework I'm going to use, either LangGraph, LangChain or so. 67 00:04:58,046 --> 00:05:01,066 These are my language I'm going to use, either Python or TypeScript. 68 00:05:02,006 --> 00:05:06,326 And then you can choose, you know, like your skills or prompts. 69 00:05:07,046 --> 00:05:09,866 And then you submit your information, right, to this. 70 00:05:10,046 --> 00:05:14,866 Once you submit this, the dev portal is going to create a repository, which is going to 71 00:05:14,866 --> 00:05:19,926 contains not just the agent framework like your LangGraph agent framework but is also going to 72 00:05:19,926 --> 00:05:25,406 contains your memory framework your tracking eval framework right and then your context 73 00:05:25,406 --> 00:05:32,166 optimizer framework and your PII blocking in web framework as well so the reason for that is 74 00:05:32,166 --> 00:05:39,246 because not necessarily all your developers are taking care of all this framework right they just 75 00:05:39,246 --> 00:05:41,566 sometimes create the framework 76 00:05:41,566 --> 00:05:43,706 what generally AI assistant tool 77 00:05:43,706 --> 00:05:45,026 are generally support, right? 78 00:05:45,186 --> 00:05:47,026 Or they stick to one or two framework, right? 79 00:05:47,186 --> 00:05:49,386 So we just want to make sure that 80 00:05:49,386 --> 00:05:51,826 when you roll out such framework, 81 00:05:52,086 --> 00:05:54,166 make sure they are consistent across the company 82 00:05:54,726 --> 00:05:56,246 and they can take care of 83 00:05:56,766 --> 00:05:58,946 and they can be centralized in one place 84 00:05:58,946 --> 00:06:01,006 so you can visualize those results as well, right? 85 00:06:01,366 --> 00:06:03,466 So once you have that repository, 86 00:06:03,706 --> 00:06:06,046 so developers basically clone that repository, 87 00:06:06,306 --> 00:06:07,586 they can build their agent 88 00:06:07,586 --> 00:06:09,106 as a Docker image into containers 89 00:06:09,106 --> 00:06:13,046 and test their agent input and output through the query. 90 00:06:13,286 --> 00:06:15,646 And they can also test this on their local memory, 91 00:06:15,966 --> 00:06:17,606 maybe cloud spanner evaluator here. 92 00:06:17,806 --> 00:06:20,686 And you can also store this tracking in some of your DB, 93 00:06:20,826 --> 00:06:22,366 maybe SQLite or maybe BigQuery, 94 00:06:23,046 --> 00:06:26,026 to make sure that agents are fully working on your local. 95 00:06:26,666 --> 00:06:28,946 So once your agents are running in local, 96 00:06:29,486 --> 00:06:31,766 then you're ready to go ahead and generate a PR. 97 00:06:32,046 --> 00:06:33,966 And then once you get your two PR approval, 98 00:06:34,406 --> 00:06:36,006 then it's go to the CI-CD pipeline. 99 00:06:36,826 --> 00:06:47,766 So the PR is basically going to, firstly, is going to register your agent into some centralized agent registry system, which provide a Langfuse URL to it to make sure that you see all the traces. 100 00:06:48,426 --> 00:06:56,306 And then the CI-CD pipeline is basically, you know, whatever the local Docker container you built here is going to build that container into production system. 101 00:06:56,646 --> 00:06:59,226 It will scan that image, make sure that there's no vulnerability. 102 00:06:59,926 --> 00:07:05,446 If everything looks great, security approved, you can move that image back into the registry. 103 00:07:05,446 --> 00:07:14,406 Right. Further, you have your own deployment platform or orchestrator platform, which is going to deploy that agent into the Kubernetes cluster in a given namespace. 104 00:07:14,906 --> 00:07:20,466 Right. So the good part here is that this portal is not just creating the repository. 105 00:07:21,206 --> 00:07:25,886 Right. It also do create a bunch of other production resources. 106 00:07:25,886 --> 00:07:34,806 For example, the workload identity. Right. And then memory DB or tracking and eval tables and then agent secret management. 107 00:07:34,806 --> 00:07:43,946 Right. So it based on whatever the name we provide it, not just again, like not just repository, create all those different resources as well. 108 00:07:44,226 --> 00:07:52,826 So what happened is when this internal orchestrator, when deploy the agents, the agent already know that, hey, what workload identity is supposed to work with. 109 00:07:52,926 --> 00:08:00,846 Right. That binding should already happen. So once this case to GSA binding happens, the agent will start communicating to all these resources. 110 00:08:01,506 --> 00:08:12,726 Once everything in place, agent is further going to communicate to the rest of your endpoint services, like, for example, AI gateway or MCP servers or Langfuse for tracing or some other API endpoints. 111 00:08:15,186 --> 00:08:19,826 So now, firstly, we talk about the MCP server as well. 112 00:08:19,886 --> 00:08:20,826 So we talk about architecture. 113 00:08:20,826 --> 00:08:22,326 Now we're going to the different component. 114 00:08:22,566 --> 00:08:27,126 So MCP server is something that you can definitely use, something which is outside available. 115 00:08:27,126 --> 00:08:32,306 But if you're working on air gap environment, it's highly recommended that you build your own MCP servers. 116 00:08:32,886 --> 00:08:37,846 So, yeah, and you can use the I will not go with the definition. 117 00:08:38,046 --> 00:08:49,906 Everyone knows about it. So the way the framework has been built here is that you can leverage the FastMCP server package or wrapper to, you know, build that MCP servers. 118 00:08:49,906 --> 00:08:53,986 and you can create a different set of tools inside that FastMCP wrapper 119 00:08:53,986 --> 00:08:57,266 to construct the overall servers. 120 00:08:57,546 --> 00:09:02,646 And then you make sure that every tool when you define an MCP server, 121 00:09:02,866 --> 00:09:07,206 they have to be fully reviewed by your own internal security team 122 00:09:07,206 --> 00:09:11,566 to make sure that only authorized tool can be go in production. 123 00:09:11,946 --> 00:09:18,086 And that's how you can enforce some filters, limits, and block the fills, 124 00:09:18,086 --> 00:09:22,366 which is really not required to go to LLM and get you the information, 125 00:09:22,366 --> 00:09:23,826 which is you're not supposed to get. 126 00:09:25,046 --> 00:09:29,546 And third part is, of course, you need to make sure that you build your MCP server 127 00:09:29,546 --> 00:09:34,486 in a way that whenever the traffic is coming from your MCP client to MCP gateway to MCP servers, 128 00:09:34,726 --> 00:09:37,166 either is coming from user or service account, 129 00:09:37,466 --> 00:09:40,746 they has to be authenticated to the MCP gateway first 130 00:09:40,746 --> 00:09:45,766 so that they can access your MCP tool to get that right information. 131 00:09:47,106 --> 00:10:04,266 All right. Now the gateway part. So the how the gateway works. So this is again, you can build it your own internal MCP gateway if you are again working on your air-gap environment. So the way it works is you are working on like local, you know, code assistance tool or maybe your agent. Right. 132 00:10:04,266 --> 00:10:11,766 So the moment you submit your first query to your agent, it will send that one-time authentication request to the MCP gateway. 133 00:10:12,286 --> 00:10:15,546 MCP gateway recognized and take your request. 134 00:10:15,706 --> 00:10:18,326 First, it will save your session state into the Redis. 135 00:10:18,886 --> 00:10:24,626 And then it will return back the JWE access token back to your client. 136 00:10:25,126 --> 00:10:31,126 Once client will receive that JWE token, it will know that what MCP server needs to call. 137 00:10:31,126 --> 00:10:36,626 right so it will submit that tool slash call request on a given namespace slash mcp meaning 138 00:10:36,626 --> 00:10:43,526 let's say artifactory or github slash mcp with the auth bearer uh generally um token um into that 139 00:10:43,526 --> 00:10:49,846 so once mcp gateway um received that uh it will understand that what namespace meaning mcp server 140 00:10:49,846 --> 00:10:56,966 needs to look up for and then it will unwrap that jwe token and then it will attach that user token 141 00:10:56,966 --> 00:11:00,146 in that request and send it to the respective MCP servers. 142 00:11:00,486 --> 00:11:02,446 Now, once MCP server receives it, 143 00:11:02,746 --> 00:11:04,786 it will make that respective API calls 144 00:11:04,786 --> 00:11:08,126 to that endpoint of that, you know, to the service. 145 00:11:08,366 --> 00:11:10,446 And then it will exchange that ID token 146 00:11:10,926 --> 00:11:13,886 and return the response based on whatever scope 147 00:11:13,886 --> 00:11:15,966 that user is authorized to see, right? 148 00:11:16,186 --> 00:11:17,966 So once the user receives the token, 149 00:11:19,006 --> 00:11:19,986 you know, MCP 150 00:11:20,026 --> 00:11:31,726 server basically unwrapped all the data, make the payload which is compatible for MCP Gateway to receive, and then finally the final payload will send back to the client to see the data points. 151 00:11:31,726 --> 00:11:47,666 So what it shows is that, you know, like we build this MCP gateway to make sure that we have a centralized authentication and routing hub that securely connect any coding assistant tool or agents to the endpoint via this MCP gateway and MCP server. 152 00:11:47,926 --> 00:11:52,626 There should not be direct access provided on any endpoints. 153 00:11:52,826 --> 00:12:00,346 There has to go through the MCP gateway through proper authentications before they're reaching to the MCP server and the endpoints. 154 00:12:00,666 --> 00:12:01,386 All right. 155 00:12:02,766 --> 00:12:06,226 So now the part is the gateway monitoring, right? 156 00:12:06,306 --> 00:12:08,126 We talk about, again, we talk about architecture, 157 00:12:08,126 --> 00:12:10,206 we talk about MCP server, MCP gateway. 158 00:12:10,706 --> 00:12:11,526 Now it's a monitoring. 159 00:12:11,766 --> 00:12:13,546 Again, the AI agent cannot be trusted 160 00:12:13,546 --> 00:12:16,586 without continuous monitoring, evaluation, and feedback, right? 161 00:12:16,706 --> 00:12:20,186 This is why we are building the MCP gateway monitoring. 162 00:12:20,486 --> 00:12:23,186 So the way the monitoring work is usually like 163 00:12:23,186 --> 00:12:25,886 the moment you submit any calls to your agent, 164 00:12:26,486 --> 00:12:29,266 it will send all the operation log into the log store. 165 00:12:29,366 --> 00:12:31,586 You can use ELK, Loki, or CloudWatch or so. 166 00:12:32,346 --> 00:12:39,506 And generally, this contains information like, show me all last 401s in my last 10 minutes call, right? 167 00:12:39,606 --> 00:12:43,146 And what did or do when the token refresh failed, right? 168 00:12:43,326 --> 00:12:45,606 So those kind of requests can go to the log store. 169 00:12:45,966 --> 00:12:52,706 Now, there are some traces and metrics data, like real-time data that goes to the OpenTelemetry as well. 170 00:12:53,646 --> 00:13:01,426 OpenTelemetry, again, is open source solutions, which is going to collect all your traces and metrics and is going to process the data. 171 00:13:01,726 --> 00:13:04,146 and redirect to the backend system, right? 172 00:13:04,466 --> 00:13:06,706 And in this case, the traces will go to the Tempo. 173 00:13:06,886 --> 00:13:08,266 This is, again, an open source solutions, 174 00:13:08,746 --> 00:13:11,506 which is going to store all your traces in terms of, like, 175 00:13:12,206 --> 00:13:13,546 traces in the sense, like, you know, like, 176 00:13:13,806 --> 00:13:17,066 information around, like, which tool was slow on this request, 177 00:13:17,386 --> 00:13:19,726 whether this call is coming from agent or IDE, 178 00:13:20,406 --> 00:13:24,306 what was the exact error on this failed call or so, right? 179 00:13:24,406 --> 00:13:26,806 So these kind of data are going to store into the Tempo. 180 00:13:27,246 --> 00:13:28,986 Oh, my bad. Okay. 181 00:13:28,986 --> 00:13:40,566 And then the metrics is about the Prometheus is all about the information about like what are P95 latencies for a given tool, right? 182 00:13:41,146 --> 00:13:46,206 Whether my auth rejection is spiking on a particular time or not, right? 183 00:13:46,426 --> 00:13:49,046 So those kind of metrics is going to store in Prometheus. 184 00:13:49,246 --> 00:13:57,966 So once you get all this data point like traces and metrics, this entire aggregate data back actually save into the Grafana for your 185 00:14:00,726 --> 00:14:14,066 All right. So the MCP server, gateway and monitoring is covered. Now we're going to talk about the framework. So the agent framework, right? So we use LangGraph for most of the cases. 186 00:14:14,066 --> 00:14:17,246 and is basically LangGraph, again, 187 00:14:17,446 --> 00:14:19,666 it's provided by LangChain community. 188 00:14:19,926 --> 00:14:21,566 It's provided a graph-based architecture 189 00:14:21,566 --> 00:14:25,226 that support loops, branching, state management 190 00:14:25,226 --> 00:14:28,266 for complex multi-agent AI workflows, right? 191 00:14:28,586 --> 00:14:30,366 And the way it works is, 192 00:14:30,446 --> 00:14:32,526 the moment you submit your first query, 193 00:14:32,746 --> 00:14:37,626 we do interpret your question first, 194 00:14:37,906 --> 00:14:39,746 what type of questions it is 195 00:14:39,746 --> 00:14:41,466 and what response it's looking for. 196 00:14:42,066 --> 00:14:45,366 and based on that, it's create a plan, right? 197 00:14:45,746 --> 00:14:49,286 Then it will send for classify area 198 00:14:49,286 --> 00:14:51,086 where it will determine that, 199 00:14:51,186 --> 00:14:53,086 hey, whether it is just like a read operations 200 00:14:53,086 --> 00:14:55,486 or whether it is just like a, you know, 201 00:14:55,586 --> 00:14:57,786 modification level changes coming up 202 00:14:57,786 --> 00:14:59,926 or whether you really want to delete something, right? 203 00:15:00,326 --> 00:15:01,946 Or some sensitive operations. 204 00:15:02,166 --> 00:15:05,806 Based on that, it adjusts the overall structure 205 00:15:05,806 --> 00:15:07,506 and then plan accordingly. 206 00:15:07,706 --> 00:15:08,686 The next step is plan. 207 00:15:09,006 --> 00:15:10,766 It's actually itself is a subgraph 208 00:15:10,766 --> 00:15:13,606 where you do the measure of your operations, 209 00:15:13,906 --> 00:15:15,026 meaning that in the plan, 210 00:15:15,206 --> 00:15:16,906 like once you know what operation you're performing, 211 00:15:17,246 --> 00:15:20,106 it goes to the, like, let's say in a fetch evidence section 212 00:15:20,106 --> 00:15:22,166 where it actually call that, 213 00:15:22,346 --> 00:15:24,226 whatever the question you ask here, 214 00:15:24,426 --> 00:15:26,666 after all, interpret and classify, 215 00:15:27,526 --> 00:15:31,166 it actually go ahead and make the call to the MCP gateway, 216 00:15:31,446 --> 00:15:33,326 make sure they get authenticated, 217 00:15:33,866 --> 00:15:36,646 and then it make the call to the MCP server 218 00:15:36,646 --> 00:15:39,726 using the right tool and get the data 219 00:15:39,726 --> 00:15:41,966 and then generate that evidence, right? 220 00:15:42,186 --> 00:15:44,006 Once you validate all your finding, 221 00:15:44,266 --> 00:15:45,466 then it will check the policies. 222 00:15:46,046 --> 00:15:47,886 It will check whether it is a read-only, 223 00:15:48,026 --> 00:15:49,126 then go ahead and execute, 224 00:15:49,306 --> 00:15:51,426 and I will save all the record points 225 00:15:51,426 --> 00:15:52,446 into our audit logs. 226 00:15:52,806 --> 00:15:54,846 But if it is a write and sensitive operations, 227 00:15:55,086 --> 00:15:57,246 definitely it will go through the HITL process 228 00:15:57,246 --> 00:15:59,646 where you wait for an approval. 229 00:15:59,666 --> 00:16:01,746 If it is approved, you go ahead and execute. 230 00:16:01,906 --> 00:16:02,766 If it is get rejected, 231 00:16:03,286 --> 00:16:06,666 in either case, it's going to save into the audit log, right? 232 00:16:08,626 --> 00:16:17,546 So if we combine all this together, like MCP gateway, MCP server, and agent, this is how things looks like. 233 00:16:17,786 --> 00:16:25,246 So ultimately, whatever the agent you are building, whatever the call you are submitting, it will go to the MCP gateway for authentications. 234 00:16:25,406 --> 00:16:36,266 And then it will redirect to the right MCP servers to execute your query on a given tool so that it make the API call to that endpoint and get the response. 235 00:16:36,666 --> 00:16:41,866 right all right if it if we think from the infrastructure perspective this is how it looks 236 00:16:41,866 --> 00:16:47,586 like so agent is communicating to the mcp gateway goes to the mcp server and to the endpoint 237 00:16:48,726 --> 00:16:56,626 all right so now the we talked a lot about mcp gateway mcp servers um the monitoring 238 00:16:56,626 --> 00:17:03,146 um and the agent framework now we talk about uh three and four additional framework that we should 239 00:17:03,146 --> 00:17:08,506 have to take care of when building the end-to-end ecosystem, right? One is them is agent memory 240 00:17:08,506 --> 00:17:17,686 management. Again, it's basically a process where you can save and restore the state of your 241 00:17:17,686 --> 00:17:23,306 workflow over the time, right? Ensuring that it's provide you a durability and seamless recovery 242 00:17:23,306 --> 00:17:30,146 across your every query run. So, the way it work is like whenever you submit any query to your agent, 243 00:17:30,906 --> 00:17:33,446 it actually generate a thread, right? 244 00:17:34,286 --> 00:17:37,526 On that given thread, when you ask your first question, 245 00:17:37,766 --> 00:17:39,626 it will going to create a run ID for that. 246 00:17:40,066 --> 00:17:43,186 So the moment you ask some deeper questions, 247 00:17:43,446 --> 00:17:46,226 it will go on a particular node in that LangGraph 248 00:17:46,226 --> 00:17:47,646 and execute that query. 249 00:17:47,826 --> 00:17:50,046 And on that node, it's going to save 250 00:17:50,046 --> 00:17:52,606 that respective checkpoint here, right? 251 00:17:52,766 --> 00:17:55,766 Now, if you ask your follow-up questions on the same thread, 252 00:17:55,946 --> 00:17:57,446 it's going to create another run ID. 253 00:17:57,646 --> 00:17:59,606 And again, depend on your follow-up questions, 254 00:17:59,606 --> 00:18:03,826 it go to a specific node and save the checkpoint over there, right? 255 00:18:04,026 --> 00:18:06,106 And all this data for your local development, 256 00:18:06,106 --> 00:18:08,306 you can use the SQLite DB, 257 00:18:08,746 --> 00:18:11,006 but in production, unless in GCP, 258 00:18:11,186 --> 00:18:12,846 you can use Cloud Spanner 259 00:18:12,846 --> 00:18:15,566 and in respect to databases, 260 00:18:15,566 --> 00:18:19,086 you can use in Azure and AWS as well, right? 261 00:18:19,466 --> 00:18:24,066 So just walk you through the quick demo here on the memory. 262 00:18:24,426 --> 00:18:27,806 So for example, I have, you know, 263 00:18:27,926 --> 00:18:29,266 like just built this agent locally, 264 00:18:29,606 --> 00:18:34,786 And then, you know, trying to run this query on SQLite DB to see if there isn't data or not. 265 00:18:35,686 --> 00:18:38,546 So I'm just, you know, initially I don't have any data. 266 00:18:38,646 --> 00:18:39,706 Now I submit my query. 267 00:18:39,846 --> 00:18:43,886 You will see that I received the one record, which is saying that, hey, this is my thread ID. 268 00:18:44,006 --> 00:18:49,806 This is the question I have been asked and exact time, you know, when this query was executed. 269 00:18:50,046 --> 00:18:55,266 Now, if I run the next query to see that, hey, how many messages I've been produced on a given thread, 270 00:18:55,866 --> 00:18:58,686 you can get this information by running the further queries. 271 00:18:59,026 --> 00:18:59,126 Right. 272 00:19:00,446 --> 00:19:18,886 Now, if I would like to run to see that, you know, like all the metadata level information on my query, I can run my query again and you can see the detail description around, you know, like how many tool I get invoked, what is my thread ID, what is my run ID, you know, like and all those informations. 273 00:19:19,226 --> 00:19:27,946 Now, if I want to also see that, hey, what tool have been executed when I run this query, you can go ahead and see the exact tool name as well on that given MCP servers. 274 00:19:28,626 --> 00:19:34,046 Now further, I also want to see that how the progression was happened on my LangGraph framework. 275 00:19:34,226 --> 00:19:39,126 That also you can see by running this type of query on your memory DB. 276 00:19:39,406 --> 00:19:45,946 You can see that here like different kind of nodes been called when actually that query has been run. 277 00:19:47,666 --> 00:19:48,346 All right. 278 00:19:49,106 --> 00:19:51,126 Now the tracking and eval. 279 00:19:51,326 --> 00:19:55,106 So just to make sure that there is no confusion here. 280 00:19:55,106 --> 00:19:58,206 like we talked about earlier the MCP gateway monitoring, 281 00:19:58,426 --> 00:19:59,826 this is agent level tracking. 282 00:20:00,066 --> 00:20:02,946 So we do this because we really want to track 283 00:20:02,946 --> 00:20:05,686 that what query you have been submitting to the agent 284 00:20:06,246 --> 00:20:07,926 and what the response is coming, 285 00:20:08,186 --> 00:20:09,826 what LLM model you have been used, 286 00:20:10,046 --> 00:20:11,926 what's the cost of your token, 287 00:20:12,346 --> 00:20:13,966 how many token you have been burned 288 00:20:13,966 --> 00:20:18,306 and what's your evaluation criteria looks like, 289 00:20:18,426 --> 00:20:20,866 what is the quality of your response or not, right? 290 00:20:20,986 --> 00:20:24,026 So you need to store that data from the day one 291 00:20:24,026 --> 00:20:36,226 And imagine that if you have a thousand plus agent, every agent will have their own, you know, agent tracking and eval project and the tables, right? 292 00:20:36,406 --> 00:20:44,086 So if you combine, let's say if you have 100 agents, you will have 100 different projects and each project has their own respective tracking and eval tables. 293 00:20:44,506 --> 00:20:52,426 Now, tomorrow, six months down the line, if somebody asks you that how your agent has been performed, what was the response, what was the quality, how much token you've been born, 294 00:20:52,426 --> 00:20:57,166 you can aggregate all this data and you can create executive dashboards around it to see 295 00:20:58,526 --> 00:21:04,866 what is the ROI in building so many agents. So let me walk you through the tracking and eval 296 00:21:04,866 --> 00:21:15,366 demo here. So here, like if I look at this one, so first, like, you know, like if you see that 297 00:21:15,366 --> 00:21:19,986 there are six tables are listed here, one is a job tracking table, and which is 298 00:21:20,026 --> 00:21:34,946 contains most of this like information around what query, like what is your query, what's the response and everything. And there are a bunch of other tables as well, like your quality related and your tool invocations and then node level executions and all that. 299 00:21:34,946 --> 00:21:46,206 So in this case, I'm going to submit my first query on my agent, and you will see that there will not be any data in the first place because it takes some time. 300 00:21:46,406 --> 00:21:47,546 So I will refresh this again. 301 00:21:49,086 --> 00:21:58,866 And this time I will see that when I refresh, I do see that for a given query, I get my request ID, my session ID, my agent name, you know, what question I've been asked, 302 00:21:59,506 --> 00:22:03,406 and what was the output of that question, 303 00:22:03,906 --> 00:22:05,766 whether it has completed or not, 304 00:22:05,826 --> 00:22:07,806 and then what is the time it took, 305 00:22:07,906 --> 00:22:09,186 what is the model I've been used, 306 00:22:09,846 --> 00:22:13,366 and then some of the metadata-level informations. 307 00:22:14,646 --> 00:22:16,266 Further, if I look for the quality, 308 00:22:16,486 --> 00:22:18,546 again, there is eight quality checks per run. 309 00:22:19,886 --> 00:22:23,346 And you can see that this one row, 310 00:22:23,566 --> 00:22:26,766 each for success, latency cost, 311 00:22:26,926 --> 00:22:28,606 tool accuracy, safety, et cetera, 312 00:22:28,606 --> 00:22:37,126 Like all this has been stored here. Right. Now, the Eval scores, this is again, this is going to be a one row. 313 00:22:37,406 --> 00:22:47,066 This is going to be a scoreboard for your given run, which is basically varies between zero to one, which is scores, you know, like all the quality dimension in a single row. 314 00:22:47,586 --> 00:22:52,606 Right. Now the quality metrics. So, again, it's a deep dive on that particular one run. 315 00:22:52,606 --> 00:23:00,986 Basically, a nested field for the response, quality, safety, latency, cost, and the tool stat in one wide row. 316 00:23:01,746 --> 00:23:03,306 Now, the tool invocation. 317 00:23:03,546 --> 00:23:10,086 So, every tool call, which MCP tool ran, how long it took, and whether it succeeded or not. 318 00:23:10,786 --> 00:23:21,326 And now, the tool agent trajectory, which is talking about step-by-step story of the run, each LLM call, and the graph state in order, like plan, summarize, et cetera. 319 00:23:22,866 --> 00:23:41,166 All right. So now. The next is the agent PIA treatment. This is very, very important because many times it happens that when you're working in agents or this thing, sometimes there is a possibility that there are some secret being ingested as a part of your query. 320 00:23:41,166 --> 00:23:44,706 Let's say you are uploading some doc, which has some sensitive information. 321 00:23:45,686 --> 00:23:48,806 And sometimes while typing, you may have some API keys or so. 322 00:23:49,166 --> 00:23:55,226 So we have a framework which defines that different sort of data types and their severity level. 323 00:23:55,446 --> 00:24:08,546 If your severity is above 90 percent or so, we will block and redact that secrets immediately so that make sure that that secret is never goes to LLM models. 324 00:24:08,846 --> 00:24:08,946 Right. 325 00:24:08,946 --> 00:24:16,906 Neither it will go to your memory DB, nor it will go to tracking and eval tables or tracing systems as well. 326 00:24:17,126 --> 00:24:24,206 So the goal here is to just block this sensor data as it is the moment it gets ingested, the moment you type. 327 00:24:26,266 --> 00:24:31,366 The last framework is all about now is the context optimizer. 328 00:24:31,366 --> 00:24:38,926 This is one of my favorite areas where, like, you know, whenever you submit, it's very important, right? 329 00:24:39,026 --> 00:24:47,246 Whenever you submit too many queries, sometimes you upload a lot of the data points and you upload a bunch of files, systems or so, right? 330 00:24:47,346 --> 00:24:53,026 It's very important that time is, you know, like you make sure that you save your token in every query. 331 00:24:53,206 --> 00:24:59,886 So we build in context, optimize your framework where you actually shape your query and then you prune it before you go to LLM. 332 00:24:59,886 --> 00:25:06,326 In the pruning system, we do select the payload based on the token limit. 333 00:25:06,486 --> 00:25:14,086 Then we re-rank that, hey, which is the most relevant information that we should target for when you submit this huge, giant context. 334 00:25:14,346 --> 00:25:25,626 And then we dedupe that, remove all the duplicates and create a very unique set of very curated data, like, I mean, the main data for the LLM. 335 00:25:25,766 --> 00:25:29,166 And then we finally compress this large payload before it goes to the LLM. 336 00:25:29,166 --> 00:25:31,366 So let's see this demo here. 337 00:25:32,886 --> 00:25:33,906 This will be quick. 338 00:25:34,206 --> 00:25:36,526 So for example, let's say I will submit my first query. 339 00:25:36,626 --> 00:25:39,486 Hey, find me all the artifacts in my JFrog Artifactory. 340 00:25:40,326 --> 00:25:45,186 You see that first when even I no submitted any query yet, there is no data here. 341 00:25:45,386 --> 00:25:57,066 So the moment I submit this first query, and if I refresh my query here, and then you can see that the record has been added. 342 00:25:57,066 --> 00:25:59,786 So you can see there is a 24,000 token has been used. 343 00:25:59,986 --> 00:26:04,526 I will go back a little bit just to show the thing. 344 00:26:04,786 --> 00:26:07,806 So you can see this 24,000 token has been used for given questions. 345 00:26:08,186 --> 00:26:14,626 Now if I ask, now I enable the context optimizer behind the scene and I'm rerun the exact same query again. 346 00:26:15,806 --> 00:26:25,366 So this time you will see that, you know, for the same query, if I refresh this run and you will see that this time it used just 2,400 token. 347 00:26:26,426 --> 00:26:27,966 for the exact same query, 348 00:26:29,266 --> 00:26:31,346 which is almost like 90% saving. 349 00:26:31,546 --> 00:26:34,766 I can rerun the same demo on the Slack bot as well, 350 00:26:34,886 --> 00:26:35,826 where I'm asking Slack bot, 351 00:26:35,906 --> 00:26:37,866 hey, tell me all my Docker images 352 00:26:37,866 --> 00:26:39,986 across all the repositories in Artifactory. 353 00:26:40,286 --> 00:26:42,586 When I run this query, there is no data. 354 00:26:42,766 --> 00:26:44,046 Now I'm going to submit my query. 355 00:26:45,846 --> 00:26:47,786 So once this query will run, 356 00:26:48,386 --> 00:26:50,606 and if I run my query again here, 357 00:26:50,806 --> 00:26:52,546 and you will see that record get created, 358 00:26:52,546 --> 00:26:56,406 it's again used like 24,000 for this particular query. 359 00:26:56,826 --> 00:27:01,166 Now, what I will do is I will turn on my optimizer behind the scene 360 00:27:01,166 --> 00:27:03,166 and I rerun the same query in the Slack bot. 361 00:27:04,126 --> 00:27:06,586 And then if I run this query again, 362 00:27:07,286 --> 00:27:11,466 it will show like it exactly used 1,500 almost token, 363 00:27:11,586 --> 00:27:12,866 which is again like 90% saving. 364 00:27:12,986 --> 00:27:16,446 In most of the cases, we get like 70 to 78% token saves. 365 00:27:16,446 --> 00:27:32,046 So imagine that you have thousands of, you know, like if you have thousands of like agents, which is running like, you know, many queries per day, imagine how much token you can save with this framework. 366 00:27:32,046 --> 00:27:42,806 All right. So now the key takeaway here is, you know, like for us is like make sure that we build an enterprise framework, not just an agent. 367 00:27:42,946 --> 00:27:48,726 Right. Which make sure that you have all the framework underneath. Don't try to be interested in building some agent quickly and deploy ship it. 368 00:27:48,846 --> 00:27:57,966 Make sure that it has to go through all the framework altogether so that you should not realize the literacy that, you know, I should have this framework in place. 369 00:27:57,966 --> 00:28:00,326 in the very first place. 370 00:28:00,506 --> 00:28:04,226 Second is make sure that you have a proper gateway system. 371 00:28:04,446 --> 00:28:06,006 Make sure that every tool called request, 372 00:28:06,126 --> 00:28:08,686 either you are doing this from your local or through agent, 373 00:28:08,906 --> 00:28:12,866 it has to go through some sort of centralized MCP gateway. 374 00:28:13,226 --> 00:28:17,426 Make sure it's authenticated before it reaching to the MCP servers. 375 00:28:17,646 --> 00:28:20,946 And then make sure that your MCP servers take a control 376 00:28:20,946 --> 00:28:27,546 on what tool you should use to execute a particular query. 377 00:28:27,966 --> 00:28:32,626 And make sure that you have monitoring in place and complete auditing in place for that tool executions. 378 00:28:32,826 --> 00:28:41,066 And of course, you should keep the always human in the loop process for every action items, every high risk items. 379 00:28:41,546 --> 00:28:47,566 Make sure that there is an observability and auditing has to be in place. 380 00:28:47,746 --> 00:28:55,346 And last but not least, make sure that context optimizer should always be there to justify your cost on your agents. 381 00:28:55,346 --> 00:29:03,806 right um so just want to share is that i released my um mcp book recently on amazon so feel free to 382 00:29:03,806 --> 00:29:13,106 download it's free and kindle um and it's a hard copy is also available all right um thank you so 383 00:29:13,106 --> 00:29:19,126 much for watching this video um and if you do have any questions feel free to you know like 384 00:29:19,126 --> 00:29:23,766 reach out to me over linkedin or twitter i'm happy to answer