Your AI Agents Shouldn't Care Where They Run - Nikhil Harithas, Factory

The workflow Blackstone wanted was easy to describe. An engineer assigns a Jira ticket to Droid, Factory’s coding agent, and gets a pull request back. The engineer could also delegate from Slack. Either way, somebody has to give Droid a machine and enough authority to do the work. For Blackstone, a financial institution with roughly a trillion dollars in assets, those were two separate decisions.

Nikhil Harithas leads forward deployed engineering at Factory. Part of his job is to find the places where a deployment forces a product decision, and Blackstone ruled out nearly every default. Agents couldn’t run on developers’ laptops. Blackstone wanted to supply the compute, inference, and identity, with single tenancy throughout. Knowing who assigned the ticket still left two questions open. Whose permissions would Droid use, and where would it exercise them?

A convenient way to start a coding agent is to launch a sandbox with the developer’s identity attached. The agent can immediately use whatever permissions come with it. Harithas saw that arrangement as temporary. A slightly nondeterministic application is still an application, and an enterprise already has ways to decide what authority to give one. Delegating a task to Droid shouldn’t require lending it the identity of the person at the keyboard.

The choice of machine could change even within one deployment. Factory was considering a sandbox provider for Blackstone when a change in constraints made EC2 worth pursuing instead. Other customers wanted Kubernetes or Google Cloud. Factory already supported installing Droid on a customer-supplied computer, but that left the customer responsible for starting machines, keeping track of them, and recovering them when something went wrong.

Factory built Open Compute to take on that lifecycle. The customer could choose where the agent ran and enforce security controls there, while Factory continued to operate its product control plane. Splitting out the compute plane gave customers much of what they wanted from an on-premises deployment without requiring them to host the whole product.

Follow a request from the CLI to start a session. Factory’s backend asks its compute gateway for a machine. The gateway can use a sandbox service or route the request over an existing connection to an Open Compute controller inside the customer’s virtual private cloud, or VPC. The controller calls a driver that knows how to provision the selected kind of compute. An EC2 driver launches an instance. A Kubernetes driver would provision a pod. A driver could also wrap a sandbox provider such as Daytona or Modal.

Once the EC2 instance comes up, the agent daemon on the machine opens a WebSocket to a relay in Factory’s control plane. Now send Droid a message from the CLI, Slack, or the web app. The relay forwards it to the sandbox over that socket, and the daemon can send a response back. Provisioning the instance and carrying the conversation are distinct jobs, which is why the architecture has both a gateway and a relay.

Connection direction mattered to Factory’s customers. They were generally more comfortable allowing an outbound connection than opening their cloud to inbound connections. After setup, the customer side initiates the connections. The socket carries traffic in both directions, so Factory can deliver a user’s next message without opening a new connection into the VPC.

Provisioning the sandbox settles where Droid executes commands. There is still a model call inside the agent loop. A company with an existing inference agreement can bring its own key. Harithas used a Bedrock setup as an example of how the company could keep that call within its VPC boundary. He saw this option as particularly relevant to mature engineering organizations that already buy inference directly.

The resulting division of responsibility is fairly precise. Client messages still travel through Factory’s relay. The customer controls the environment where the model-driven loop runs, can observe what the agent does there, and can constrain the reach of its actions. Harithas reported that this arrangement helped Factory gain customers’ trust. He expects some large enterprises to want control over more of the stack over time, including the model and eventually the product deployment itself.

At the time of the talk, Factory was running Open Compute in private preview and planned to open source it in the following weeks. Its interface gives a caller a computer it can start, pause, resume, or destroy. A customer-controlled template specifies what to install, down to a particular revision or choice of container. The gateway can request a known build without knowing how each provider constructs it.

A Factory session carries more meaning than that. It represents a back-and-forth with Droid. Building that concept into the controller would make customers host a piece of Factory’s application logic and make the software harder to reuse with another agent. Keeping the contract at the level of a computer also gives driver authors a concrete target. Factory could hand the specification to Droid and ask it to implement a driver, then judge the result by whether it satisfied the same contract the gateway calls.

Suppose Droid reaches a point in a task where it can delegate part of the work. It can snapshot its current machine, fork the snapshot, and hand the resulting environment to a subagent. The child starts with the state the parent had already assembled. If the child needs to delegate again, the same operation applies at the next level.

Snapshot ownership was another part of Factory’s plan. Harithas described letting customers save snapshots in S3 or another store of their choice and, eventually, populate a restored machine with secrets from their own secret store. That gives the customer a say in both where saved state lives and how a new machine acquires the credentials to use it.

A provider can acknowledge a provisioning request without ever returning a usable sandbox. A request can also disappear halfway through a session, leaving the client waiting for an acknowledgment. The compute layer has to track whether a machine actually started, what the provider called it, and which sandbox should receive a given user’s messages. Retrying a request requires enough bookkeeping for the client and sandbox to agree on where they left off.

Factory’s Missions product makes the distinction between a machine and a session tangible. A mission can orchestrate several agents for multiple days. Harithas cited an eight-hour session cap for an AWS microVM environment. To keep the mission going across that limit, Factory has to snapshot the expiring sandbox, restore its state into a fresh one, and direct the ongoing interaction to the replacement.

Even distributing the controller took work. During private preview, Factory versioned and shared Terraform packages so customers could stand it up in their own environments. For the engineer waiting on that Jira ticket, the desired outcome is still a pull request. Behind it, the driver may provision more than one machine over the life of the assignment, while the engineer continues working with the same Droid session.