Build-Time vs. Run-Time: Why Dev Tools Fail in Production — Averi Kitsch & Prerna Kakkar, Google

Averi Kitsch, technical lead for MCP Toolbox for databases, and Prerna Kakkar, technical lead for EvalBench, begin with a practical distinction: database tools that help developers explore or administer a system are usually the wrong tools to place behind a production agent. MCP Toolbox supplies database connectivity, authentication, connection pooling, and observability, but those capabilities do not decide how much freedom an agent should receive. The tool interface still determines what the agent can do.

The first two database patterns favor flexibility. Control-plane tools create and manage instances or databases. Because these administrative operations can be destructive, they belong in developer-assistance workflows with a human checking consequential actions. Natural-language-to-SQL tools let an agent generate raw SQL when the necessary query cannot be known in advance. They suit exploration, such as finding California customers who bought a winter coat in July, returned it within 14 days, and then grouping them by the campaign that acquired them.

Structured SQL tools target a different operating environment. The SQL statement and accepted parameters are defined before deployment, leaving the agent to select a known operation and supply a small set of values. That limits invented queries, supports injection defenses, and can reduce latency because the model no longer synthesizes SQL for every request. The lost flexibility is intentional: production software should expose the actions the product supports, rather than every action the database can perform.

Build-time tools are atomic and flexible, which makes them useful under a developer’s supervision and hazardous inside an end-user application. A production chatbot instead needs runtime operations such as a deterministic cancel_order tool whose database behavior is fixed behind the interface.

The failure at 6:05 is blunt. After encountering an error, the agent proposed deleting the table and starting over. The table was deleted, with no guardrail stopping the operation. The observable result was total data removal; the causal chain was an error, a model-generated recovery idea, broad database authority, and no independent control capable of rejecting the destructive path. A plausible suggestion became an executable disaster because the tool exposed too much power.

The planned runtime demonstration used SimAir, a flight-booking chatbot. Prerna would claim to be Averi and ask the system to book a flight under Averi’s identity. Authentication was intended to keep the booking tied to Prerna instead. The demonstration did not load, so the recording explains the intended result without visibly verifying it; the identity-binding mechanism later in the talk shows how the application is meant to enforce that result.

“Your database is only as secure as your agent” captures the next problem. Language models can be manipulated, so database security cannot depend on the model consistently distinguishing legitimate instructions from malicious ones. In a confused deputy attack, someone convinces an agent to misuse privileges that the agent legitimately possesses but the requester does not.

The dangerous combination is the “lethal trifecta,” a phrase coined by Simon Willison: access to private data, exposure to untrusted content, and a channel that can return data to an external user. The triage-agent example at 9:38 contains all three. The agent reads a ticket, uses database permissions to investigate, and posts its findings back to the ticket.

A malicious insider inserts an instruction to query the salary database and return every employee’s salary. The agent complies because the ticket looks like a trusted work item and the database accepts the agent’s credentials. The breach does not require stealing those credentials. The attacker instead supplies content that persuades a privileged deputy to use them.

Traditional applications narrow this path with fixed fields and predefined queries. Agentic applications introduce values derived dynamically from language, so the system must distinguish agent-controlled parameters from application-controlled parameters. Agent parameters come from model interpretation and should be treated as untrusted. Application parameters encode facts and constraints—such as the authenticated user’s ID—that must remain outside the model’s control.

The first boundary separates three identities. The user needs access to the application. The application’s workload identity may need broader access to supporting services. The agent operating inside that application should receive only the data access required for that user’s journey. Collapsing these identities into one credential makes the agent a proxy for the application’s full authority.

At the start of the secure-tool progression at 12:33, a fully model-controlled database tool is effectively a superuser interface: the model supplies credentials, host, port, connection details, and arbitrary SQL. If the model can be tricked, every one of those controls is exposed through the same vulnerable decision-maker.

MCP Toolbox’s source primitive removes connection configuration from the tool signature. An operator defines it in YAML when starting the MCP server, and the server injects it without exposing those details to the agent. This does not yet control the SQL, but it prevents the model from choosing a different host, port, credential, or database connection.

The source can enforce narrower limits in layers. Read-only access should remove write tools and reach down to the database driver, rather than relying on the agent to avoid writes. Allowed datasets, supported by some cloud-native databases, restrict which data collections are reachable. Output caps limit how much data one call can retrieve, reducing both exfiltration impact and load on the agent and database. These controls do not make malicious instructions harmless; they reduce what those instructions can accomplish.

Connection and source controls still leave one dangerous input: arbitrary SQL generated by the model. Custom tools remove that freedom by defining the exact statement in YAML, along with a name and description that tell the agent when to use it. Typed parameters flow through prepared statements, so values are validated and bound as data rather than concatenated into executable SQL.

A constrained tool can still fail if the agent cannot understand or invoke it reliably. The recommended interface is outcome-shaped rather than a thin copy of atomic REST endpoints. A tool named for the complete action can replace several round trips, while its description should explain when and why to use it instead of repeating parameter definitions already present in the schema.

Several interface choices improve control and reliability:

• Separate reads from writes. Read tools can be automatically approved while write tools can require explicit user confirmation.
• Return actionable errors. A generic 404 gives the agent little recovery guidance; an error that identifies a retryable condition gives it a useful next step.
• Prefer flat, simple inputs. Complex nested maps are harder for models to construct consistently than a small set of typed fields.

The lookup_flights tool now has predefined SQL and accepts only specific fields such as user_id and date. That is safer than arbitrary SQL, but user_id remains sensitive: a prompt could still persuade the model to substitute another account. The final step removes this parameter from the agent-visible interface.

With bound parameters, the application authenticates the user and injects the resulting identity directly into the tool call without showing it to the agent. With authenticated parameters, the tool receives an OpenID-signed JWT, validates the token, and extracts claims such as user ID, email, and issuer. Both mechanisms make conversation text unable to select the account under which the query runs.

What changed from the original superuser tool to the final lookup at 19:08? The diagram follows the shrinking control surface. Connection details move into server configuration; driver restrictions and, where supported, allowed datasets limit the blast radius; custom tools replace generated SQL; and authenticated application state supplies the user identity. The agent is left with a nonsensitive choice such as the date.

The resulting design is described as a zero-trust architecture because meaningful privileges and factual constraints are not granted merely because the model appears cooperative. That label does not make the complete system automatically safe; it describes where enforcement occurs. The model proposes a date, while components outside the model determine the connection, permitted operation, reachable data, output size, and authenticated user.

The closing reminder is operational: tool behavior must be evaluated, not merely declared secure. EvalBench is highlighted as the framework used to check whether agentic tools work well. The recording does not provide evaluation results, but the dependency is clear: SQL and identity guardrails define permitted behavior, while evaluations test whether agents select and use those constrained tools successfully.