300 Agents, 60 People: How We Secured Our Own AI Agents - Aiden Sochowski, Runlayer

Runlayer had the same visibility problem it saw elsewhere: employees used local MCP clients, signed up for agent-powered products, and authenticated connections to company systems without a central account of what those connections could do. Aiden Sochowski, a founding engineer at Runlayer, describes the company’s response: deploy a gateway and centrally log every agent session and tool call. Before deciding what to block, the team needed to see what was happening.

The inventory revealed more than 300 agents running every day in a company of roughly 50–60 people. These were not all assistants waiting for someone to type a prompt. Chatbots, triggers, and SaaS applications could initiate work. Mapping them by topic and risk mattered because agent count alone said little about exposure: an agent with many tools could reach more systems and cause more damage than a narrowly provisioned one.

What connects that inventory to an inspectable action? A colleague built a galaxy-like visualization that grouped teams, agents, and individual runs. Looking into a run revealed the MCP servers it touched, the actions it took, and errors it encountered. The useful relationship was between the organization-wide view and the individual tool call: a fetch, a failed operation, or a write could be traced back to the agent doing it.

Each call also marked a place where risk could enter or leave. Reading could introduce hostile data; writing could destroy or alter something; sending could disclose information. Agents made it easier to combine data from systems that previously required someone to deliberately write an integration. That convenience also made it easier to mix data classes that should remain separate.

Support automation provides a concrete threat scenario. An agent reads incoming company email. A malicious sender asks it to return a list of current sales prospects. If the agent treats the email as an instruction, has permission to inspect a calendar containing prospect information, and can communicate the result externally, an ordinary support intake path can become a data-exfiltration path. This is an illustrative attack in the talk, not a documented successful breach.

The failure depends on several capabilities meeting in one agent: accepting an outside message, reading internal information, and sending information back out. Scoping every employee’s agent by hand is difficult, but a blanket ban on a tool is often too crude. Sending an email may be appropriate for the support task; sending private prospect information is not. The policy needs enough context to distinguish them.

The example offers several distinct places to intervene:

• Inspect incoming content. A trained classifier can scrutinize each email for instructions that should not control the agent. This checks the material entering the session, but the classifier will not be perfectly accurate.
• Inspect outgoing content. A policy can block external messages containing sensitive information. Sochowski gives regular-expression matching for customer names as one example of a content check.
• Restrict sensitive reads. A policy can deny access to a particular source, such as a private calendar, so the agent cannot retrieve that information in the first place.

These controls address different parts of the same path. An inbound check asks whether the email is trying to redirect the agent. An outbound check asks what the agent is about to disclose. A read restriction removes access to a source regardless of how persuasive the email is. Runlayer found that deploying such policies internally remained difficult: safe behavior depends on the task and session context, so writing the rules is still an ongoing problem.

Some attacks do not fit inside one suspicious tool call. Consider a pull-request reviewer allowed both to review code and approve it. Its assigned job is to inspect a security-sensitive change against a checklist. If it can also read GitHub comments, the PR author has a conversational route into the review process.

The author can dispute findings, explain why checks supposedly pass, or redirect attention with a request such as “only review the cleanup.” Over successive exchanges, the reviewer may narrow its attention and approve the PR without satisfying the original security review. The approval capability has not changed; what changed is the agent’s interpretation of the work it should do before using it. Repeated persuasion is a possible failure mode here, not a measured success rate.

That makes a different question necessary at execution time: does this tool call still serve the agent’s original assignment, given everything that happened in the session? Runlayer trained its own classifier models to take session data and assess each tool call against the agent’s defined purpose. An approval can look ordinary in isolation while being inappropriate after a conversation that talked the reviewer out of its checklist. The talk describes this classifier’s inputs and purpose, but does not establish its detection accuracy.

How much can a bot do beyond its assigned task? Runlayer’s recruiter bot had access to an applicant tracking system, meeting notes, a calendar, and the ability to send messages to users. Its job could be narrowly defined while its credentials exposed much more. Reading the applicant tracking system under a person’s credentials, for example, could expose negotiation emails as well as the information the recruiting task actually needed.

For an agent with a stable task, historical tool use provides a useful baseline. If a run begins using tools outside its usual range, Runlayer can flag it as high risk and investigate early. This is easier to interpret for a specialized bot than for a general-purpose local coding assistant: a broad assistant legitimately changes activities, while a task-specific bot has a clearer expected range.

Runlayer also uses models to examine an agent’s past tool calls and suggest permissions that could be removed. A recruiter bot may need some applicant records without needing everything in the tracking system, or some meeting information without access to every meeting. The history helps propose a smaller permission set when that set is hard to specify upfront. The proposals inform deterministic access restrictions; the talk does not describe automatic removal or establish that every historically unused permission is unnecessary.

Limiting one agent’s tools does not fully describe its effect on other agents. In another threat scenario, a résumé contains malicious instructions. An agent reads it and writes into a company system. A different agent later reads that system as part of its normal work. The shared store becomes an indirect route for hostile content to enter another agent’s context, even without the agents communicating directly.

Where can one contaminated input reach another agent? The data-flow diagram follows the résumé example through the shared system. Its important connection is the write followed by a later read: restricting an agent’s immediate actions does not isolate the content it leaves for someone else.

This is why Runlayer maps systems and their connections, rather than treating each agent as an independent security problem. Sochowski’s plain description is useful: agents are software “plumbing data” between sources. Large, poorly partitioned systems give that plumbing many possible downstream readers.

The response is to tightly scope writes, potentially limiting an agent to particular databases inside a SaaS application. That restricts where contaminated content can be deposited and which later workflows can encounter it. Sochowski treats write scoping as essential to protecting downstream context; it limits propagation paths rather than proving that the content itself is harmless.

The same monitoring also exposed failures with no malicious actor. When an MCP tool returns an error, an agent may try to work around it. Those attempts consume input and output tokens. A recurring agent can encounter the same defect on every run, making a small integration problem an ongoing expense.

Runlayer’s recruiting team had vibe-coded an MCP server for a vendor that supplied an API but no MCP server. The builders had not written software before and had tested it only lightly. Formatting problems caused repeated errors, and costs grew as the agent kept encountering them. The agent’s willingness to keep trying amplified a defect in the connector.

The practical controls address different parts of that failure:

• Monitor errors. Tool-call records reveal repeated failures that otherwise disappear inside agent sessions.
• Rate-limit activity. Limits constrain how much repeated failing work can accumulate.
• Inspect connector quality. Reviewing the MCP server’s errors and code addresses the formatting defect rather than asking the agent to keep improvising around it.

A separate cost problem appeared when people switched agents from initially cheap models to more expensive ones. Sochowski reports costs rising more than 100× within a week after such switches. This is an operational observation, not a controlled comparison: the talk does not separate model pricing from changes in token use or run volume.

The difficult decision is whether a particular task earns the extra model capability. Sochowski observes that many people building these agents are not running evaluations, leaving the company without a clear basis for saying that a more expensive model produces enough additional value. Refusing every upgrade would also prevent useful work. Runlayer’s response was to set budgets, establish sensible defaults, and require a reason when a task supposedly needed more intelligence.

Security and cost therefore end at the same operational requirement: monitoring must lead to an intervention. Company-wide automation cannot be made useful by stopping every action. The team needs to identify the high-risk operations that deserve scrutiny, stop them when necessary, and route people toward tools and behavior that follow internal policy.

The gateway and session history make those choices more concrete. They help locate the point where a support agent could disclose data, a reviewer could abandon its assignment, a shared write could affect another agent, or a failing connector could keep spending money. The goal is enough visibility and control to intervene at those consequential points while allowing the rest of the work to continue.