Paving the Road to Production: Governed Enterprise AI Agents — Harish Gaggar, Credit Karma

An agent can answer a question locally while leaving its most consequential production questions unresolved. Who authenticated the tool call? Which operations may the tool perform? Can a follow-up recover the previous conversation? Did an API key enter the model prompt—or get copied into a trace? Harish Gaggar, a senior staff software engineer responsible for AI analytics infrastructure and services at Intuit Credit Karma, opens with these gaps between an experiment and an enterprise deployment.

The proposed response is a collection of supporting frameworks, rather than a “magical agent.” LangGraph supplies task-specific nodes and workflow state. An MCP gateway authenticates requests and routes them to custom MCP servers, whose tools perform the operations. Memory, tracking and evaluation, sensitive-data filtering and context optimization handle concerns that a functioning prompt does not resolve by itself.

The paved path starts before implementation. A developer registers the agent’s use case and receives an agent ID. In an internal developer portal, they supply that ID, the intended use case, a framework such as LangGraph or LangChain, a language such as Python or TypeScript, and the skills or prompts they want to use. Those choices become inputs to repository generation.

The generated repository includes the agent framework together with memory, tracking and evaluation, context optimization and PII filtering. This addresses a predictable inconsistency: individual developers may implement only the pieces their coding assistant suggests, or concentrate on one or two frameworks. Supplying the surrounding components at creation time gives agents a consistent starting point and makes their results easier to inspect centrally.

Development then moves through a concrete release sequence. The developer clones the repository, builds a Docker image and tests query inputs and outputs with the supporting storage. Once the agent works locally, a pull request requires two approvals. The release process registers the agent centrally and provides a Langfuse tracing URL; CI/CD builds the production container and scans it for vulnerabilities. After security approval, the image enters the registry and an internal orchestrator deploys it into a Kubernetes namespace. Scanning is a release control, not proof that an image contains no vulnerability.

The portal also provisions the workload identity, memory database, tracking and evaluation tables, and secret management. That provisioning matters at deployment: the agent should already have its identity binding and know which resources to use. The orchestrator can deploy an agent with its supporting resources prepared, rather than leave each team to connect identity, storage and tracing afterward.

Custom MCP servers provide a place to control what tools expose. Gaggar recommends building them internally for an air-gapped environment and describes using a FastMCP wrapper to define the tools. Each tool undergoes internal security review before production use. Filters, limits and excluded fields can then constrain what the tool returns, including information that should never reach the model. The decision is about the operations and data exposed by a tool, not merely whether its server runs inside the company.

Both users and service accounts must authenticate through the MCP gateway. At the first query, a coding assistant or agent sends an authentication request. The gateway saves session state in Redis and returns a JWT access token to the client. This establishes a session before the client requests a tool operation.

For the tool request, the client supplies the JWT as a bearer token and addresses a namespace identifying the intended MCP server—for example, an Artifactory or GitHub namespace. The gateway uses that namespace to select the server and attaches the user token to the forwarded request. The MCP server calls the downstream service, handles the described token exchange and returns data within the user’s authorized scope. It then packages the response for the gateway to return to the client.

Authentication and routing therefore happen on the same required path: client, gateway, MCP server, downstream endpoint. The intended design prohibits clients from bypassing the gateway to call those endpoints directly. A central entry point makes authentication consistent across local coding tools and deployed agents, while the reviewed server tools control the operations available behind it.

A centralized gateway also creates a useful place to observe failures. Its monitoring separates three kinds of evidence, each answering a different operational question:

• Operation logs: a store such as ELK, Loki or CloudWatch records events that help investigate failures such as an unsuccessful token refresh.
• Request traces: OpenTelemetry collects and processes trace data for Tempo. A trace helps identify which tool was slow, whether a request originated in an agent or an IDE, and where a particular call failed.
• Aggregate metrics: Prometheus stores measurements such as a tool’s P95 latency and changes in authentication rejection rates. These reveal patterns across calls rather than the history of one request.

Grafana supplies the visualization layer for the traces and metrics. The distinction is practical: an authentication rejection spike tells an operator that something changed; a trace and the corresponding operation logs help explain which request failed and why. These gateway records describe access and execution behavior. Later, the agent’s own tracking tables address questions about the answer, model use and cost.

LangGraph provides loops, branches and state management for the agent workflow. The first stages interpret the question and form a plan, then classify the requested operation: reading information, modifying it, deleting it or performing another sensitive action. That classification changes how the plan proceeds. The plan itself is a subgraph containing the substantive work.

Evidence gathering uses the same authenticated tool path described earlier. The workflow calls the MCP gateway, reaches the appropriate MCP server and retrieves data through a tool. After validating the findings, it checks policy. Read-only operations can proceed to execution and auditing; write or sensitive operations wait for human approval. Approval permits execution. Rejection still produces an audit record, so the record includes decisions that stopped an action as well as actions that ran.

Where does human approval change the outcome? The branch below separates permission to execute from the obligation to record the decision. Both approval and rejection reach the audit log, but only approval allows the sensitive operation to run. This is how the workflow places a human decision between a proposed action and its effect.

Memory management saves and restores workflow state over time. A thread identifies the continuing conversation. The first question creates a run ID, and execution at a graph node saves a checkpoint. A follow-up stays on the same thread but creates another run ID, with checkpoints saved as that run reaches its nodes. Keeping those identifiers distinct lets the system connect multiple executions to one conversation without treating them as one undifferentiated attempt.

The local demonstration uses SQLite; Cloud Spanner is the named production option on GCP. Initially, the queried database contains no data. After submitting a question, a query returns a record containing the thread ID, the question and its execution time. Further queries inspect how many messages belong to that thread. The observable change is a persisted record of the interaction, rather than state held only in the running process.

What can an operator reconstruct from those records? The subsequent queries inspect thread and run IDs, the number of tools involved, the particular MCP tool executed and the progression through LangGraph nodes. Together, those views connect a user interaction to its workflow execution. The demonstration inspects stored state and execution history; it does not stage a crash-and-recovery test.

Agent-level tracking records the submitted question, response, model, token consumption, cost and evaluation information. Gaggar recommends collecting it from day one. In his example, 100 agents have 100 projects with their own tracking and evaluation tables; those records can later be aggregated into dashboards. The purpose is to make questions about performance and return on investment answerable six months later, rather than attempt to reconstruct missing history.

The demonstration starts with six tables. After a query is submitted, the first inspection finds no data yet; a later refresh reveals the tracking row. What identifies that run and its result? The record includes request and session IDs, agent name, question, output, completion status, elapsed time, model and metadata. The delay between submission and visibility is part of the example: an immediately empty result does not establish that tracking failed.

The remaining views answer different questions about that same run:

• Quality checks: eight checks per run, with separate rows for dimensions including success, latency, cost, tool accuracy and safety.
• Evaluation scores: a single scoreboard row collects quality dimensions scored between zero and one.
• Quality metrics: a wide row uses nested fields for response, quality, safety, latency, cost and tool statistics.
• Tool invocations: each call records which MCP tool ran, how long it took and whether it succeeded.
• Agent trajectory: ordered records describe LLM calls and graph states, such as planning and summarization.

These tables establish a structure for inspection, not a validated quality score for the system. The walkthrough does not explain how the quality checks are computed or calibrated. Likewise, aggregated token and cost records can support an ROI investigation, but the demonstration does not measure business ROI. The useful foundation is the ability to relate a result to its model calls, tools, timing and evaluation fields.

A document upload may contain sensitive information, and a typed query may accidentally include an API key. The PII trimmer classifies data types and assigns severity levels. Gaggar describes blocking and redacting secrets when severity exceeds approximately 90%. That threshold belongs to the filtering rule; it is not a reported detection accuracy.

The intended interception point is ingestion, before the input reaches the model, memory database, tracking and evaluation tables or tracing system. This placement matters because preventing a model call alone would still leave a secret exposed if logging or checkpointing had already copied it elsewhere. The design aims to stop that spread early. Its effectiveness still depends on detection and redaction coverage; the walkthrough does not establish that every sensitive input will be caught.

Large uploads and accumulated data can make each query expensive. The context optimizer shapes and prunes the payload before it reaches the model. It selects material within a token limit, reranks it for relevance, removes duplicates and compresses the remaining payload. Each operation targets a different source of excess: too much material, poorly prioritized material, repeated material and an unnecessarily large representation.

The first demonstration asks the agent to find all artifacts in JFrog Artifactory. The tracking view begins empty; after submission and refresh, a record reports 24,000 tokens used. Gaggar then enables the optimizer behind the scenes and reruns the same question.

How much does the recorded usage change? The optimized run reports 2,400 tokens, down from 24,000—a 90% reduction. The comparison makes the cost mechanism concrete: the question remains the same while the optimizer changes the material sent onward. The demonstration does not include a paired answer-quality study, so the smaller token count alone cannot establish that the answer retained all the required information.

A second example runs through a Slackbot and asks for Docker images across all repositories in Artifactory. Its first recorded run also uses about 24,000 tokens. After enabling the optimizer and repeating the question, the reported usage falls to approximately 1,500 tokens. Gaggar reports savings of 70–78% in most cases, distinguishing those broader observations from the larger reductions in the two demonstrations. Across many agents issuing many daily queries, reducing repeated context could materially reduce token consumption.

The closing recommendation returns to the portal’s original purpose: include the supporting frameworks before the agent ships. Every tool request, whether it originates locally or from a deployed agent, should pass through centralized authentication. MCP servers should control which tools can execute the requested operation. Monitoring and audit records should cover those executions, and high-risk actions should retain human approval.

Context optimization belongs in that initial framework too, because cost becomes harder to ignore as usage grows. The paved path makes these concerns repeatable: identity and resources arrive with the deployment; tool access follows a common route; decisions and executions leave records; context is reduced before model use. The engineering decision is to make those capabilities part of creating an agent, rather than rediscover their absence after deployment.