Don't Vibe Code Your Agent Infrastructure - Akshat Bubna, Modal

A financial analyst wants to run the same analysis on every company in a portfolio. Each company has its own data sources, and each analysis may require Python. Hebbia put that work behind a spreadsheet interface, with a different agent backing each cell. For the analyst, running the query is one action. For the infrastructure underneath, it can mean starting thousands of execution environments fast enough for the spreadsheet to feel responsive, then shutting them down when the work finishes.

In the architecture Akshat Bubna described, each agent gets a Modal sandbox, a secure environment for running untrusted code. A sandbox can contain multiple containers or microVMs. That lets the developer put the harness, the loop that calls the model and dispatches tools, in a separate container from the code those tools execute. The harness can keep directing the analysis while the platform isolates the execution of generated code.

Compute isolation still leaves a choice about which agents should see each other’s work. Bubna offered one possible arrangement for the spreadsheet. Give every cell in a row access to the same storage volume, and those agents can exchange files while running in separate environments. A main container and its sidecar can likewise share files when a tool needs access to the harness’s inputs.

The contents of a workspace depend on the job. Ramp’s internal coding agent, Inspect, needs the repository, its dependencies and the development services it uses. Basis, which builds accounting agents, needs source documents, accounting workpapers and scripts. For Basis, the workspace also has to survive an unpredictable pause. Documents arrive, an agent works through them, and then it needs an accountant’s response. The reply might come days later. Keeping compute allocated through that wait costs money, yet discarding the working state makes it hard to resume.

Bubna described three persistence choices. A snapshot of the entire root filesystem preserves the environment as a unit. A directory snapshot preserves a narrower piece, such as node_modules, so the base image can be upgraded separately. A volume is useful when several participants need ongoing access to the same files. The practical question is what the next participant needs when work resumes. Reopening an agent’s environment and giving a colleague access to its workpapers are different storage requirements.

Ramp let Modal share CPU and memory observations from Inspect. Demand varied across sampled sandboxes and over time within a single sandbox. A compilation could drive up CPU use, while an analysis might need more memory. Bubna estimated that sizing every sandbox for the observed peaks would mean allocating roughly 15 CPU cores and 15 GB of RAM to each one. Those figures describe a peak-provisioning decision. An agent would incur the cost of that allocation during the quieter parts of its run as well.

Modal’s approach is to let a sandbox acquire resources when the operation needs them and charge for that interval. The same reasoning applies to accelerators. A scientific workflow can preprocess data on CPUs and start a GPU sandbox for the model run. In a model-training experiment Bubna described, an agent could switch GPUs between runs and execute ablations in parallel. That gave the harness two independent decisions to make for each experiment, which hardware to use and which trials could run at the same time.

Once a human, an agent and its subagents work on the same files, filesystem consistency becomes part of the application’s behavior. A subagent needs to know when it can read another participant’s update. Bubna sees a related pattern in agents that maintain memory through background tasks. The task that revises a memory and the agent that later consults it both depend on how quickly and predictably a write becomes visible.

Network access calls for finer distinctions. A sandbox may need to reach GitHub or Datadog to do its job, but allowing the whole domain can grant access to more operations than that job requires. Bubna described a pattern Ramp has used successfully. Developers put a lightweight proxy in a sidecar and write rules for the paths and operations the agent can use. Credentials can be injected into a permitted request without giving the agent a secret it could read and send elsewhere. Inbound access has its own controls, including which IP addresses may connect and how callers authenticate.

Consider a training run that fails. A failure signal tells the harness that execution stopped, but offers little basis for choosing the next set of parameters. If the agent can read the training logs, it can inspect what happened before launching another iteration. While the job is running, GPU utilization gives it evidence to investigate a possible stall. Access to spend data lets it weigh the cost of continuing.

Bubna’s point is that the agent dispatching the work is now one of the consumers of the telemetry. A human may still use a dashboard to investigate a run. The harness needs access to those same observations while it can still act on them, whether that means inspecting an idle GPU or changing the parameters for a retry.

Bubna’s case against improvising a sandbox platform rests on three operating responsibilities. Someone has to reason through rare failures and how the system responds to them. Someone has to keep reviewing the security boundaries as the runtime evolves. And the platform has to deliver capacity at the rate the application consumes it. A prototype that successfully starts a sandbox exercises only a small part of that last requirement.

One personal-assistant workload he described called for 10,000 sandbox starts per second and close to a million running sandboxes. Each would live for only two or three minutes and needed to become interactive in under a second. Storage has its own scale. Across Modal, Bubna reported 20 million snapshots a week, about eight petabytes of snapshot data processed per day, and almost 2.6 million new volumes a week. He said Modal aggregates CPU capacity from multiple cloud providers because it cannot meet its capacity needs from one alone.

Bubna expects the harness and the runtime to become increasingly independent. Today a developer might deploy many copies of one agent environment. In the model he anticipates, the harness chooses environments as the work unfolds, starts several when experiments can proceed in parallel, and retains selected snapshots for later. The accounting agent waiting days for a reply and the spreadsheet agent needed for a single query illustrate why that separation matters. The work has a lifetime of its own, and the harness can decide when it needs compute again.