Machine transcription with reviewed terminology and names. Some recognition errors may remain. Timestamps refer to the original livestream.
3:58:42Hello, everyone.
3:58:43I would like to walk you through that how we can transition in AI agents from an experiment
3:58:48to the real enterprise environment.
3:58:50So before I begin, a quick introduction about me.
3:58:53My name is Harish Gaggar.
3:58:55I'm a senior staff software engineer at Intuit Credit Karma,
3:58:58where I built and own the AI analytics infrastructure and services.
3:59:03And beside that, I'm advisor, author, researchers,
3:59:08and also a program committee members for various organizations
3:59:12where I mentor the young researchers as well.
3:59:14All right.
3:59:16So let's walk through the agenda here.
3:59:21So first, we would like to talk about what are the challenges we have when we build an AI agents and deploy into the production systems and how we can build a different framework to address that different challenges here.
3:59:34Then how the architecture looks like for it.
3:59:37Then how you can build your own MCP servers and MCP gateway.
3:59:43and then some monitoring on top of that gateway as well.
3:59:48Then further, we're going to talk about what are the different type of agent framework you can have
3:59:53and then some additional framework around managing the memories, tracking and eval,
3:59:59how you can
4:00:00block the PII data going before the LLMs, and then Context Optimizer to save your token.
4:00:07So without delay, let me walk you through the challenge. Of course, there are many agents that generally get failed in production because their foundation controls are missing.
4:00:18And the key reason behind is that, you know, like most of the people are missing the gateway to make sure that the agent can, you know, communicate to the gateway to get authenticated first with the right auth providers so that they can redirect to the right MCP servers.
4:00:35Right. Second, that MCP servers are missing the right tool to execute your query.
4:00:40Third, do we have like enough tracking and evaluation system to make sure that every query when you are sending to the agents, whether they've been tracked or not, and are you evaluating their quality response or not?
4:00:52Right. Do we have any memory management system or not?
4:00:55In terms of, let's say you are doing the interaction with agent for long or for many days, is it able to recall that what you have been asked in the past?
4:01:03Right. And then there are other framework, of course, like do you have a context optimizers to save the token?
4:01:09or do you have enough production in place
4:01:11to make sure that any secrets should not go into LLM
4:01:14or tracking or tracing or any memory DB or so, right?
4:01:18So a lot of open questions.
4:01:20So in order to resolve this,
4:01:23we are not going to, you know,
4:01:25like build a magical agent here in this demo.
4:01:28We're going to talk about like different framework
4:01:31that you can build to address this.
4:01:33So first is the agent framework.
4:01:35This is where like we use the,
4:01:37So like we choose to use the LangGraph framework, which is provided by LangChain community.
4:01:42This has helped us in building the task-specific node based on your questions,
4:01:48and which is easy to plug in any like marketplace plugins or skills and prompts
4:01:54to make sure that agents respond efficiently based on your query.
4:01:59Further, the second part is we are also building the MCP gateway to make sure that whenever agents make the call
4:02:06or any service account from that agent make the call,
4:02:09make sure that they are securely get authenticated
4:02:11to that MCP gateway
4:02:13and then get redirected to the right MCP servers.
4:02:16Third is the customized MCP,
4:02:18which means that you are building your own MCP servers
4:02:20with the right tool in place to execute your query.
4:02:23And then finally, some audit
4:02:25on top of what actually operation you're performing, right?
4:02:29And then we talk about some additional framework
4:02:31around, you know, like how you can block the PIA secrets,
4:02:35how you're going to store the tracking value into the cloud, and how you're going to
4:02:40purchase the memories, especially any threads and checkpoints on every context. And then also how
4:02:46you can optimize overall your context to make sure that you don't burn too many tokens when
4:02:53you run your query. All right. So we'll talk about now the agent infrastructure. So firstly, we'll
4:03:01go through the local development. The way the approach should be is that, you know, like
4:03:07whenever you build an agent, right, first you have to register your agent into some system.
4:03:11You usually tell that, hey, look, this is my, the use case, and these are the use case going to use
4:03:17for this particular vertical or system. And you feed all those information and generate an ID for
4:03:23your agent, right? Once you have that, then you can have a dev portal, internal portal, where you
4:03:28You can ingest all the details saying that, hey, look, this is my agent use case.
4:03:33This is my agent ID.
4:03:34This is the framework I'm going to use, either LangGraph, LangChain or so.
4:03:38These are my language I'm going to use, either Python or TypeScript.
4:03:41And then you can choose, you know, like your skills or prompts.
4:03:47And then you submit your information, right, to this.
4:03:50Once you submit this, the dev portal is going to create a repository, which is going to
4:03:54contains not just the agent framework like your LangGraph agent framework but is also going to
4:03:59contains your memory framework your tracking eval framework right and then your context
4:04:05optimizer framework and your PII blocking in web framework as well so the reason for that is
4:04:12because not necessarily all your developers are taking care of all this framework right they just
4:04:19sometimes create the framework
4:04:21what generally AI assistant tool
4:04:23are generally support, right?
4:04:25Or they stick to one or two framework, right?
4:04:27So we just want to make sure that
4:04:29when you roll out such framework,
4:04:32make sure they are consistent across the company
4:04:34and they can take care of
4:04:36and they can be centralized in one place
4:04:38so you can visualize those results as well, right?
4:04:41So once you have that repository,
4:04:43so developers basically clone that repository,
4:04:46they can build their agent
4:04:47as a Docker image into containers
4:04:49and test their agent input and output through the query.
4:04:53And they can also test this on their local memory,
4:04:55maybe cloud spanner evaluator here.
4:04:57And you can also store this tracking in some of your DB,
4:05:00maybe SQLite or maybe BigQuery,
4:05:03to make sure that agents are fully working on your local.
4:05:06So once your agents are running in local,
4:05:09then you're ready to go ahead and generate a PR.
4:05:12And then once you get your two PR approval,
4:05:14then it's go to the CI-CD pipeline.
4:05:16So the PR is basically going to, firstly, is going to register your agent into some centralized agent registry system, which provide a Langfuse URL to it to make sure that you see all the traces.
4:05:28And then the CI-CD pipeline is basically, you know, whatever the local Docker container you built here is going to build that container into production system.
4:05:36It will scan that image, make sure that there's no vulnerability.
4:05:39If everything looks great, security approved, you can move that image back into the registry.
4:05:45Right. Further, you have your own deployment platform or orchestrator platform, which is going to deploy that agent into the Kubernetes cluster in a given namespace.
4:05:54Right. So the good part here is that this portal is not just creating the repository.
4:06:01Right. It also do create a bunch of other production resources.
4:06:05For example, the workload identity. Right. And then memory DB or tracking and eval tables and then agent secret management.
4:06:14Right. So it based on whatever the name we provide it, not just again, like not just repository, create all those different resources as well.
4:06:24So what happened is when this internal orchestrator, when deploy the agents, the agent already know that, hey, what workload identity is supposed to work with.
4:06:32Right. That binding should already happen. So once this case to GSA binding happens, the agent will start communicating to all these resources.
4:06:41Once everything in place, agent is further going to communicate to the rest of your endpoint services, like, for example, AI gateway or MCP servers or Langfuse for tracing or some other API endpoints.
4:06:55So now, firstly, we talk about the MCP server as well.
4:06:59So we talk about architecture.
4:07:00Now we're going to the different component.
4:07:02So MCP server is something that you can definitely use, something which is outside available.
4:07:07But if you're working on air gap environment, it's highly recommended that you build your own MCP servers.
4:07:12So, yeah, and you can use the I will not go with the definition.
4:07:18Everyone knows about it. So the way the framework has been built here is that you can leverage the FastMCP server package or wrapper to, you know, build that MCP servers.
4:07:29and you can create a different set of tools inside that FastMCP wrapper
4:07:33to construct the overall servers.
4:07:37And then you make sure that every tool when you define an MCP server,
4:07:42they have to be fully reviewed by your own internal security team
4:07:47to make sure that only authorized tool can be go in production.
4:07:51And that's how you can enforce some filters, limits, and block the fills,
4:07:58which is really not required to go to LLM and get you the information,
4:08:02which is you're not supposed to get.
4:08:05And third part is, of course, you need to make sure that you build your MCP server
4:08:09in a way that whenever the traffic is coming from your MCP client to MCP gateway to MCP servers,
4:08:14either is coming from user or service account,
4:08:17they has to be authenticated to the MCP gateway first
4:08:20so that they can access your MCP tool to get that right information.
4:08:27All right. Now the gateway part. So the how the gateway works. So this is again, you can build it your own internal MCP gateway if you are again working on your air-gap environment. So the way it works is you are working on like local, you know, code assistance tool or maybe your agent. Right.
4:08:44So the moment you submit your first query to your agent, it will send that one-time authentication request to the MCP gateway.
4:08:52MCP gateway recognized and take your request.
4:08:55First, it will save your session state into the Redis.
4:08:58And then it will return back the JWE access token back to your client.
4:09:05Once client will receive that JWE token, it will know that what MCP server needs to call.
4:09:11right so it will submit that tool slash call request on a given namespace slash mcp meaning
4:09:16let's say artifactory or github slash mcp with the auth bearer uh generally um token um into that
4:09:23so once mcp gateway um received that uh it will understand that what namespace meaning mcp server
4:09:29needs to look up for and then it will unwrap that jwe token and then it will attach that user token
4:09:36in that request and send it to the respective MCP servers.
4:09:40Now, once MCP server receives it,
4:09:42it will make that respective API calls
4:09:44to that endpoint of that, you know, to the service.
4:09:48And then it will exchange that ID token
4:09:50and return the response based on whatever scope
4:09:53that user is authorized to see, right?
4:09:56So once the user receives the token,
4:09:58you know, MCP
4:10:00server basically unwrapped all the data, make the payload which is compatible for MCP Gateway to receive, and then finally the final payload will send back to the client to see the data points.
4:10:11So what it shows is that, you know, like we build this MCP gateway to make sure that we have a centralized authentication and routing hub that securely connect any coding assistant tool or agents to the endpoint via this MCP gateway and MCP server.
4:10:27There should not be direct access provided on any endpoints.
4:10:32There has to go through the MCP gateway through proper authentications before they're reaching to the MCP server and the endpoints.
4:10:40All right.
4:10:42So now the part is the gateway monitoring, right?
4:10:46We talk about, again, we talk about architecture,
4:10:48we talk about MCP server, MCP gateway.
4:10:50Now it's a monitoring.
4:10:51Again, the AI agent cannot be trusted
4:10:53without continuous monitoring, evaluation, and feedback, right?
4:10:56This is why we are building the MCP gateway monitoring.
4:11:00So the way the monitoring work is usually like
4:11:03the moment you submit any calls to your agent,
4:11:06it will send all the operation log into the log store.
4:11:09You can use ELK, Loki, or CloudWatch or so.
4:11:12And generally, this contains information like, show me all last 401s in my last 10 minutes call, right?
4:11:19And what did or do when the token refresh failed, right?
4:11:23So those kind of requests can go to the log store.
4:11:25Now, there are some traces and metrics data, like real-time data that goes to the OpenTelemetry as well.
4:11:33OpenTelemetry, again, is open source solutions, which is going to collect all your traces and metrics and is going to process the data.
4:11:41and redirect to the backend system, right?
4:11:44And in this case, the traces will go to the Tempo.
4:11:46This is, again, an open source solutions,
4:11:48which is going to store all your traces in terms of, like,
4:11:52traces in the sense, like, you know, like,
4:11:53information around, like, which tool was slow on this request,
4:11:57whether this call is coming from agent or IDE,
4:12:00what was the exact error on this failed call or so, right?
4:12:04So these kind of data are going to store into the Tempo.
4:12:07Oh, my bad. Okay.
4:12:08And then the metrics is about the Prometheus is all about the information about like what are P95 latencies for a given tool, right?
4:12:21Whether my auth rejection is spiking on a particular time or not, right?
4:12:26So those kind of metrics is going to store in Prometheus.
4:12:29So once you get all this data point like traces and metrics, this entire aggregate data back actually save into the Grafana for your
4:12:40All right. So the MCP server, gateway and monitoring is covered. Now we're going to talk about the framework. So the agent framework, right? So we use LangGraph for most of the cases.
4:12:54and is basically LangGraph, again,
4:12:57it's provided by LangChain community.
4:12:59It's provided a graph-based architecture
4:13:01that support loops, branching, state management
4:13:05for complex multi-agent AI workflows, right?
4:13:08And the way it works is,
4:13:10the moment you submit your first query,
4:13:12we do interpret your question first,
4:13:17what type of questions it is
4:13:19and what response it's looking for.
4:13:22and based on that, it's create a plan, right?
4:13:25Then it will send for classify area
4:13:29where it will determine that,
4:13:31hey, whether it is just like a read operations
4:13:33or whether it is just like a, you know,
4:13:35modification level changes coming up
4:13:37or whether you really want to delete something, right?
4:13:40Or some sensitive operations.
4:13:42Based on that, it adjusts the overall structure
4:13:45and then plan accordingly.
4:13:47The next step is plan.
4:13:48It's actually itself is a subgraph
4:13:50where you do the measure of your operations,
4:13:53meaning that in the plan,
4:13:55like once you know what operation you're performing,
4:13:57it goes to the, like, let's say in a fetch evidence section
4:14:00where it actually call that,
4:14:02whatever the question you ask here,
4:14:04after all, interpret and classify,
4:14:07it actually go ahead and make the call to the MCP gateway,
4:14:11make sure they get authenticated,
4:14:13and then it make the call to the MCP server
4:14:16using the right tool and get the data
4:14:19and then generate that evidence, right?
4:14:22Once you validate all your finding,
4:14:24then it will check the policies.
4:14:26It will check whether it is a read-only,
4:14:28then go ahead and execute,
4:14:29and I will save all the record points
4:14:31into our audit logs.
4:14:32But if it is a write and sensitive operations,
4:14:35definitely it will go through the HITL process
4:14:37where you wait for an approval.
4:14:39If it is approved, you go ahead and execute.
4:14:41If it is get rejected,
4:14:43in either case, it's going to save into the audit log, right?
4:14:48So if we combine all this together, like MCP gateway, MCP server, and agent, this is how things looks like.
4:14:57So ultimately, whatever the agent you are building, whatever the call you are submitting, it will go to the MCP gateway for authentications.
4:15:05And then it will redirect to the right MCP servers to execute your query on a given tool so that it make the API call to that endpoint and get the response.
4:15:16right all right if it if we think from the infrastructure perspective this is how it looks
4:15:21like so agent is communicating to the mcp gateway goes to the mcp server and to the endpoint
4:15:28all right so now the we talked a lot about mcp gateway mcp servers um the monitoring
4:15:36um and the agent framework now we talk about uh three and four additional framework that we should
4:15:43have to take care of when building the end-to-end ecosystem, right? One is them is agent memory
4:15:48management. Again, it's basically a process where you can save and restore the state of your
4:15:57workflow over the time, right? Ensuring that it's provide you a durability and seamless recovery
4:16:03across your every query run. So, the way it work is like whenever you submit any query to your agent,
4:16:10it actually generate a thread, right?
4:16:14On that given thread, when you ask your first question,
4:16:17it will going to create a run ID for that.
4:16:20So the moment you ask some deeper questions,
4:16:23it will go on a particular node in that LangGraph
4:16:26and execute that query.
4:16:27And on that node, it's going to save
4:16:30that respective checkpoint here, right?
4:16:32Now, if you ask your follow-up questions on the same thread,
4:16:35it's going to create another run ID.
4:16:37And again, depend on your follow-up questions,
4:16:39it go to a specific node and save the checkpoint over there, right?
4:16:44And all this data for your local development,
4:16:46you can use the SQLite DB,
4:16:48but in production, unless in GCP,
4:16:51you can use Cloud Spanner
4:16:52and in respect to databases,
4:16:55you can use in Azure and AWS as well, right?
4:16:59So just walk you through the quick demo here on the memory.
4:17:04So for example, I have, you know,
4:17:07like just built this agent locally,
4:17:09And then, you know, trying to run this query on SQLite DB to see if there isn't data or not.
4:17:15So I'm just, you know, initially I don't have any data.
4:17:18Now I submit my query.
4:17:19You will see that I received the one record, which is saying that, hey, this is my thread ID.
4:17:23This is the question I have been asked and exact time, you know, when this query was executed.
4:17:30Now, if I run the next query to see that, hey, how many messages I've been produced on a given thread,
4:17:35you can get this information by running the further queries.
4:17:39Right.
4:17:40Now, if I would like to run to see that, you know, like all the metadata level information on my query, I can run my query again and you can see the detail description around, you know, like how many tool I get invoked, what is my thread ID, what is my run ID, you know, like and all those informations.
4:17:59Now, if I want to also see that, hey, what tool have been executed when I run this query, you can go ahead and see the exact tool name as well on that given MCP servers.
4:18:08Now further, I also want to see that how the progression was happened on my LangGraph framework.
4:18:14That also you can see by running this type of query on your memory DB.
4:18:19You can see that here like different kind of nodes been called when actually that query has been run.
4:18:27All right.
4:18:29Now the tracking and eval.
4:18:31So just to make sure that there is no confusion here.
4:18:35like we talked about earlier the MCP gateway monitoring,
4:18:38this is agent level tracking.
4:18:40So we do this because we really want to track
4:18:42that what query you have been submitting to the agent
4:18:46and what the response is coming,
4:18:48what LLM model you have been used,
4:18:50what's the cost of your token,
4:18:52how many token you have been burned
4:18:53and what's your evaluation criteria looks like,
4:18:58what is the quality of your response or not, right?
4:19:00So you need to store that data from the day one
4:19:04And imagine that if you have a thousand plus agent, every agent will have their own, you know, agent tracking and eval project and the tables, right?
4:19:16So if you combine, let's say if you have 100 agents, you will have 100 different projects and each project has their own respective tracking and eval tables.
4:19:24Now, tomorrow, six months down the line, if somebody asks you that how your agent has been performed, what was the response, what was the quality, how much token you've been born,
4:19:32you can aggregate all this data and you can create executive dashboards around it to see
4:19:38what is the ROI in building so many agents. So let me walk you through the tracking and eval
4:19:44demo here. So here, like if I look at this one, so first, like, you know, like if you see that
4:19:55there are six tables are listed here, one is a job tracking table, and which is
4:20:00contains most of this like information around what query, like what is your query, what's the response and everything. And there are a bunch of other tables as well, like your quality related and your tool invocations and then node level executions and all that.
4:20:14So in this case, I'm going to submit my first query on my agent, and you will see that there will not be any data in the first place because it takes some time.
4:20:26So I will refresh this again.
4:20:29And this time I will see that when I refresh, I do see that for a given query, I get my request ID, my session ID, my agent name, you know, what question I've been asked,
4:20:39and what was the output of that question,
4:20:43whether it has completed or not,
4:20:45and then what is the time it took,
4:20:47what is the model I've been used,
4:20:49and then some of the metadata-level informations.
4:20:54Further, if I look for the quality,
4:20:56again, there is eight quality checks per run.
4:20:59And you can see that this one row,
4:21:03each for success, latency cost,
4:21:06tool accuracy, safety, et cetera,
4:21:08Like all this has been stored here. Right. Now, the Eval scores, this is again, this is going to be a one row.
4:21:17This is going to be a scoreboard for your given run, which is basically varies between zero to one, which is scores, you know, like all the quality dimension in a single row.
4:21:27Right. Now the quality metrics. So, again, it's a deep dive on that particular one run.
4:21:32Basically, a nested field for the response, quality, safety, latency, cost, and the tool stat in one wide row.
4:21:41Now, the tool invocation.
4:21:43So, every tool call, which MCP tool ran, how long it took, and whether it succeeded or not.
4:21:50And now, the tool agent trajectory, which is talking about step-by-step story of the run, each LLM call, and the graph state in order, like plan, summarize, et cetera.
4:22:02All right. So now. The next is the agent PIA treatment. This is very, very important because many times it happens that when you're working in agents or this thing, sometimes there is a possibility that there are some secret being ingested as a part of your query.
4:22:21Let's say you are uploading some doc, which has some sensitive information.
4:22:25And sometimes while typing, you may have some API keys or so.
4:22:29So we have a framework which defines that different sort of data types and their severity level.
4:22:35If your severity is above 90 percent or so, we will block and redact that secrets immediately so that make sure that that secret is never goes to LLM models.
4:22:48Right.
4:22:48Neither it will go to your memory DB, nor it will go to tracking and eval tables or tracing systems as well.
4:22:57So the goal here is to just block this sensor data as it is the moment it gets ingested, the moment you type.
4:23:06The last framework is all about now is the context optimizer.
4:23:11This is one of my favorite areas where, like, you know, whenever you submit, it's very important, right?
4:23:19Whenever you submit too many queries, sometimes you upload a lot of the data points and you upload a bunch of files, systems or so, right?
4:23:27It's very important that time is, you know, like you make sure that you save your token in every query.
4:23:33So we build in context, optimize your framework where you actually shape your query and then you prune it before you go to LLM.
4:23:39In the pruning system, we do select the payload based on the token limit.
4:23:46Then we re-rank that, hey, which is the most relevant information that we should target for when you submit this huge, giant context.
4:23:54And then we dedupe that, remove all the duplicates and create a very unique set of very curated data, like, I mean, the main data for the LLM.
4:24:05And then we finally compress this large payload before it goes to the LLM.
4:24:09So let's see this demo here.
4:24:12This will be quick.
4:24:14So for example, let's say I will submit my first query.
4:24:16Hey, find me all the artifacts in my JFrog Artifactory.
4:24:20You see that first when even I no submitted any query yet, there is no data here.
4:24:25So the moment I submit this first query, and if I refresh my query here, and then you can see that the record has been added.
4:24:37So you can see there is a 24,000 token has been used.
4:24:39I will go back a little bit just to show the thing.
4:24:44So you can see this 24,000 token has been used for given questions.
4:24:48Now if I ask, now I enable the context optimizer behind the scene and I'm rerun the exact same query again.
4:24:55So this time you will see that, you know, for the same query, if I refresh this run and you will see that this time it used just 2,400 token.
4:25:06for the exact same query,
4:25:09which is almost like 90% saving.
4:25:11I can rerun the same demo on the Slack bot as well,
4:25:14where I'm asking Slack bot,
4:25:15hey, tell me all my Docker images
4:25:17across all the repositories in Artifactory.
4:25:20When I run this query, there is no data.
4:25:22Now I'm going to submit my query.
4:25:25So once this query will run,
4:25:28and if I run my query again here,
4:25:30and you will see that record get created,
4:25:32it's again used like 24,000 for this particular query.
4:25:36Now, what I will do is I will turn on my optimizer behind the scene
4:25:41and I rerun the same query in the Slack bot.
4:25:44And then if I run this query again,
4:25:47it will show like it exactly used 1,500 almost token,
4:25:51which is again like 90% saving.
4:25:52In most of the cases, we get like 70 to 78% token saves.
4:25:56So imagine that you have thousands of, you know, like if you have thousands of like agents, which is running like, you know, many queries per day, imagine how much token you can save with this framework.
4:26:12All right. So now the key takeaway here is, you know, like for us is like make sure that we build an enterprise framework, not just an agent.
4:26:22Right. Which make sure that you have all the framework underneath. Don't try to be interested in building some agent quickly and deploy ship it.
4:26:28Make sure that it has to go through all the framework altogether so that you should not realize the literacy that, you know, I should have this framework in place.
4:26:37in the very first place.
4:26:40Second is make sure that you have a proper gateway system.
4:26:44Make sure that every tool called request,
4:26:46either you are doing this from your local or through agent,
4:26:48it has to go through some sort of centralized MCP gateway.
4:26:53Make sure it's authenticated before it reaching to the MCP servers.
4:26:57And then make sure that your MCP servers take a control
4:27:00on what tool you should use to execute a particular query.
4:27:07And make sure that you have monitoring in place and complete auditing in place for that tool executions.
4:27:12And of course, you should keep the always human in the loop process for every action items, every high risk items.
4:27:21Make sure that there is an observability and auditing has to be in place.
4:27:27And last but not least, make sure that context optimizer should always be there to justify your cost on your agents.
4:27:35right um so just want to share is that i released my um mcp book recently on amazon so feel free to
4:27:43download it's free and kindle um and it's a hard copy is also available all right um thank you so
4:27:53much for watching this video um and if you do have any questions feel free to you know like
4:27:59reach out to me over linkedin or twitter i'm happy to answer
