We Gave an Agent Production Code Access and Then Tried to Sleep at Night

Dependency patching resembles vacuuming: finish the job, and it soon needs doing again. Across thousands of repositories, the backlog never empties. Moritz Johner illustrates the treadmill with ten issues closed today and twenty more arriving next week. Automating the repetitive work sounds straightforward—Dependabot and Renovate already create dependency updates—but the vulnerable component may sit outside the dependency declaration the automation knows how to change.

An OS package can arrive inside a container’s base image without appearing explicitly in the Dockerfile. A build can also download a binary from a URL: seeing a version in that URL does not establish how to remediate the binary’s vulnerabilities. Johner characterizes the existing workflow as patching manifests while leaving the rest invisible. That characterization needs a narrow reading: current Renovate custom managers can extract dependencies beyond ordinary manifests. The missing capability here is reasoning from a vulnerable shipped artifact back to the changes that fix it.

Even a visible dependency rarely changes in isolation. Upgrading the Go runtime to another minor version can require upgrading the Go linter. The newer linter can introduce rules that invalidate previously accepted code. A bot that bumps only the runtime and opens a pull request leaves an engineer to resolve that chain of consequences. The task is to locate the CVE across the artifact’s full surface, then determine everything else that must move to reach green CI.

The team built PatchPilot to do that reasoning and put it into production. InfoSec then asked whether they had created useful automation or a supply chain incident waiting to happen. Johner accepts the pushback: a useful coding agent is a supply chain actor. Once an application has production credentials and can change code, it needs guardrails comparable to those applied to an engineer. The practical question becomes which capabilities the agent itself needs, and which the surrounding application can perform on its behalf.

PatchPilot starts with the artifacts customers actually receive: OCI images. A deterministic Go application discovers and scans those images, then maps each image to the repository that builds it. This mapping connects a vulnerability in a shipped artifact to a place where the system can make a repair. The controller is deliberately simple; agents run inside its orchestration when the work requires diagnosis.

A failed CI run illustrates the division. The failure might be caused by the patch, a flaky test, an infrastructure problem, or a timeout. Those cases do not all call for another code change: a timeout may need only a retry. The controller manages the lifecycle; the agent interprets the evidence and decides what kind of repair, if any, is appropriate. In the walkthrough diagram, green denotes deterministic controller steps and yellow denotes agent steps.

The first agent invocation has a concrete input and a deliberately restricted output:

1. Prepare the assessment. A JSON manifest lists images and the CVEs found in each image.
2. Prepare the workspace. The controller clones the repository and creates a context directory containing the assessment, a substantial prompt, and material used to communicate with the agent.
3. Remediate the findings. The CVE remediation agent makes the smallest effective change set. It targets the version that fixes the identified CVE rather than upgrading everything to the latest release.
4. Verify locally. The agent is instructed to build the Dockerfile and rescan the resulting image.

The scope restriction matters because unrelated upgrades introduce risk without helping resolve the assessed findings.

The agent’s output is a modified filesystem. It does not commit, push, create a pull request, or watch CI. After it returns control, the controller vets the changes for accidental debris: an incorrectly assembled shell pipeline can create empty files, and a build can leave a compiled binary in the repository. This check catches a different class of mistake from whether the vulnerability was fixed.

After vetting, the controller commits and pushes the changes, opens a pull request, and watches CI. Green CI leads to a Slack notification. A failure leads to another agent invocation, this time with CI logs, workflow metadata, and the other context needed to diagnose the PR.

The PR remediation agent must make minimum forward fixes. Johner found that models tend to revert their preceding changes to recover a passing build; for a security patch, that can undo the reason for the PR. The repair instructions therefore require preserving the remediation and verifying the new work. The controller repeats the publish-and-watch cycle until CI passes or a configured retry limit sends the issue to a human.

The ownership split can be expressed with a small Go control loop. Here, publishAndWatch belongs to the controller; repairLocally receives failure context and edits and verifies the workspace. Each repair returns through vet before publication.

package patching

import "errors"

type Failure struct {
	Logs     string
	Workflow string
}

var ErrNeedsHuman = errors.New("CI repair limit reached")

func Reconcile(
	maxRepairs int,
	vet func() error,
	publishAndWatch func() (*Failure, error),
	repairLocally func(Failure) error,
) error {
	for repairs := 0; ; repairs++ {
		if err := vet(); err != nil {
			return err
		}
		failure, err := publishAndWatch()
		if err != nil {
			return err
		}
		if failure == nil {
			return nil
		}
		if repairs >= maxRepairs {
			return ErrNeedsHuman
		}
		if err := repairLocally(*failure); err != nil {
			return err
		}
	}
}

The function shape makes the retry boundary explicit. Credential separation must enforce the corresponding authority boundary: passing a narrowly named function is not itself a security control.

At the end of every invocation, PatchPilot asks the agent for a short retrospective: what went well, what went wrong, which tools were missing, and what context would help next time. These reports give the team a practical view into agent behavior while agent observability remains an open problem. Aggregating and condensing them across pull requests exposes recurring problems that are harder to see in individual runs.

Two categories recur:

• Infrastructure failures: network problems or insufficient permissions to clone a repository.
• Repository complexity: codebases that are difficult to reason about without additional context.

The second category can lead to changes in the system prompt or repository-specific instructions. The retrospective is useful because it directs attention toward the missing tool, permission, or explanation rather than treating every unsuccessful run as the same kind of model failure.

The production example is heavily redacted because it comes from a finance environment, but its important behavior is visible in the walkthrough. The patch updates a golang base image—an ordinary dependency-update operation—and also remediates libcrypto3 and libssl3.

The diff explains how those changes fit together. The packages had previously been pinned explicitly. Once the updated base image supplied their fixes, the agent removed the obsolete pins. The repair therefore depended on understanding what the new base image already provided, not merely replacing one version string with another. Johner says he might have missed that cleanup himself. The result is a working pull request awaiting human review, approval, and merge; green CI does not make those decisions disappear.

The complete PatchPilot application needs substantial access. Its GitHub capabilities cover cloning repositories, committing and pushing changes, opening PRs, downloading CI logs, and triggering CI. OCI registry credentials let it enumerate images and determine which are latest. Its execution environment includes Go and Python runtimes, linters, static libraries, a shell, and network access.

Those capabilities are not all handed to the agent. The architecture’s two layers also divide credentials:

The write credentials and CI-triggering authority stay in code whose actions the team can reason about directly. If untrusted context redirects the agent, it does not thereby acquire those credentials. The application can be powerful without making every part equally powerful.

Johner reports roughly 70,000 changed lines in the demonstrated PR after only a few dependency bumps. That is the size of one dependency-remediation diff, not a count of code independently authored by the agent. It illustrates how much material can enter the workflow through a seemingly small update, and why the credential boundary matters.

PatchPilot does not assume prompt injection is solved. It uses prompt steering to identify known sources of untrusted context, including the vendor directory and files containing CI logs. The instruction is to treat that material as evidence to inspect, not authority to obey.

The team also runs end-to-end evaluations against crafted repositories. One contains a deprecated function that tries to recruit the agent into malicious behavior; another route leads through a migration guide referenced by a deprecated function. These tests exercise the workflow an agent would actually follow while repairing code. The slide expands the attack surface to source code, migration guides and README.md, GitHub issues and release notes, poisoned CVE advisories, and CI logs. Known cases can be tested, but unknown injection vectors remain, which is why steering and evaluations sit alongside limits on what the agent can do.

Local verification creates another security problem. An agent repairing a Dockerfile needs to build an image and may need to run a container to inspect installed package versions. Giving it a Docker socket seems like the natural way to make that possible.

But access to a powerful host Docker daemon—particularly a rootful daemon—can turn a restricted workspace into host-level authority. The agent can ask the daemon to launch a privileged container and reach outside its original sandbox. Johner describes the resulting risks: reading other processes’ environments or memory and planting SSH keys. The Docker security boundary depends on the daemon’s privileges; this warning is not a claim that every rootless or restricted socket has identical power. In the diagram, docker.sock crosses the sandbox boundary to a host labeled “root.”

The team had run the host-socket configuration in production and moved away from it. They reconsidered Linux isolation mechanisms, including Landlock, Bubblewrap, seccomp-related mechanisms, and Fence, as well as container build options such as Kaniko and BuildKit. Johner’s objection was that the evaluated approaches did not compose adequately with their container workflow or contain access to the host’s Docker daemon. Restricting the agent process is insufficient if that process can command a more privileged service outside the restriction.

The emerging replacement uses a Firecracker microVM. Both the agent and its Docker daemon live inside the guest, with a separate guest kernel. A container escape into that guest no longer implies the same direct route into the host that the host-socket arrangement provided. Johner describes this design as still in its infancy: the microVM adds an isolation boundary, rather than making escapes categorically impossible.

Network policy follows the same controller–agent split. The controller has relatively predictable destinations, such as GitHub. The agent’s requirements depend on the repository: Java, Python, and Go need access to different ecosystems. Applying one undifferentiated network policy to both layers would ignore that distinction.

In the proposed plumbing, a DNS and TCP forwarder inside the microVM sends traffic through vsock to a host process. Other host connectivity is cut off, making that process the point where outgoing traffic is checked. Policies can match hostnames, destination ports, and CIDR ranges. This forwarding and enforcement layer belongs to the surrounding design; Firecracker itself is not the egress firewall.

A further option is to install a custom certificate authority inside the guest and intercept TLS traffic for finer inspection. Johner flags that as difficult, particularly around Docker, and does not develop it further. The core design already separates two responsibilities: the guest boundary contains the execution environment, while the host-side forwarder governs where that environment can communicate.

For the configurations discussed in the recording, Johner considers the sandboxes bundled with Codex and Claude inadequate once a host Docker socket is exposed; he describes OpenCode and Pi as lacking built-in sandboxes. These are his assessments of those configurations, not a current feature matrix. His desired contract is an externally enforced environment in which the agent has the tools and freedom needed to finish the assigned job.

A VM with the necessary forwarding is not an exotic invention, and sandbox services can supply parts of it. Their suitability depends on details such as Docker containment and network access controls, which vary across providers. Johner names Microsandbox as the project he would consider if rebuilding PatchPilot, particularly for its included network controls. He describes it as open source and YC-funded, while questioning whether it will attract enough community traction to remain sustainable. His recommendation is exploratory: he treats the project as young, and continued availability alone does not answer the operational questions.

The remaining gap is getting those pieces into an enterprise deployment. In Johner’s assessment, much of the ecosystem is still beta, with missing features and limited enterprise adoption. Even a suitable sandbox needs orchestration above it. He points to vendor offerings, Kubernetes Agent Sandbox, a project under SIG Apps, and OpenSandbox as efforts in that direction. Managing sandbox lifecycles is a separate responsibility from the runtime that provides isolation; selecting one does not eliminate the need for the other.

An agent’s blast radius is an architectural decision. PatchPilot needs to push changes, open PRs, and trigger CI, but its reasoning agent does not need to hold the credentials for those operations. Moving them into the deterministic controller preserves the workflow while limiting the agent’s direct authority. Deciding what is deterministic and what is agentic therefore determines more than code organization: it determines the security model.

Johner closes by inviting comparison with other production systems, including discussion at the following day’s sandbox panel. The open work is practical: how teams provide agents enough freedom to complete real tasks, which boundaries they enforce outside the model, and what orchestration makes those boundaries usable in production.