The execution contract and its trusted components
A sandbox is a controlled execution environment that restricts an application's permissions and access to resources. Generated code needs the same treatment as other untrusted code. The restriction must exist in the execution system; an instruction asking the program to avoid sensitive files does not prevent access.
A trust boundary separates different assumptions about control or authority. The trusted computing base comprises the hardware, firmware and software relied upon to enforce policy. For execution, that can include the kernel, runtime, management service and credential gateway. The protected assets determine which components belong in this set; AI Security explains the surrounding threat model.
A repository-build service provides a useful execution contract: receive one tenant's source, install dependencies, run tests and return selected changes for review. The harness—the runtime requesting and coordinating execution—needs observable results from this environment. Scheduling and continuation belong to Harness Engineering; executing code and incorporating its changes remain separate operations.
| Contract dimension | Required specification |
|---|---|
| Assets and operations | Name protected resources, permitted accesses and the enforcer for each crossing. |
| Consumption | Specify resource budgets, their scope and the response when a limit is reached. |
| Surviving state | Identify disposable work, retained outputs and independently persistent effects. |
Isolation limits where execution can act. Authorization determines whether a particular action on a particular resource is permitted. An allowed connection to a repository service therefore does not itself authorize a push. The control-placement distinction keeps execution restrictions from being mistaken for business approval.
Process, container and virtual machine boundaries
A process is a running program with an address space: the memory addresses it can use. Separate address spaces do not eliminate inherited access. On Linux, fork creates a child with initially copied memory and inherited file descriptors—handles referring to already-open resources. Parent and child can consequently retain access to the same underlying files.
The kernel is the operating-system component that mediates hardware and privileged operations. A container groups processes under operating-system isolation controls while sharing a kernel. A virtual machine runs a guest operating system, including its own kernel, behind a virtualization layer. Its virtual machine monitor, or VMM, manages guest execution and virtual devices.
Different paths to host services
Isolation changes the interfaces exposed to workload requests.
Read the diagram as text
- Container code.
- gVisor workload.
- Sentry.
- Guest application.
- Guest kernel.
- VMM device handling.
- Host kernel.
- Container code → Host kernel: System calls.
- gVisor workload → Sentry: System calls.
- Sentry → Host kernel: Restricted host interface.
- Guest application → Guest kernel: System calls.
- Guest kernel → VMM device handling: Device servicing boundary.
- VMM device handling → Host kernel: Host operations.
| Choice | Boundary and remaining dependency | Workload implication |
|---|---|---|
| Ordinary container | Scoped resource views; application system calls still reach the shared host kernel. | Broad Linux compatibility, with host-kernel exposure to assess. |
| gVisor | Its Sentry implements application system calls and uses a restricted host interface; mapped files and allowed connections remain accessible. | Check application compatibility and the actual filesystem/network configuration. |
| MicroVM | A guest kernel and a reduced VMM device surface; host infrastructure still supplies resources. | Check guest images, device support and host/guest maintenance obligations. |
| Language isolate | A constrained language execution context exposing selected interfaces. | Useful for functions; package installation, subprocesses and application servers require a compatible richer environment. |
A VM exit transfers guest execution to host-side handling. Device-heavy workloads can incur repeated servicing costs; CPU work and cached guest operations follow different paths. These differences require workload measurements, not a universal overhead percentage.
Startup optimization also extends beyond the isolation runtime. Modal's 2025 architecture account pairs gVisor with custom scheduling and storage infrastructure. This illustrates why selecting a runtime does not settle image distribution, readiness or fleet behavior.
Operating-system controls and inherited access
A system call is a program's request for a kernel operation. Confinement combines controls over visibility, access, privilege and consumption. These properties are independent: hiding another process from a process listing does not establish that every route to its resources has been denied.
| Control | Enforced property | Important exclusion |
|---|---|---|
| Namespaces | Separate views of resources such as process IDs, mounts and network stacks. | A restricted view is not a resource-specific permission policy. |
| Identity-based permissions | Kernel access checks use the requesting identity and applicable permissions. | Privileged powers can bypass particular checks. |
| Linux capabilities | Divide traditionally root-only powers into separately enabled privileges. | Removing one power leaves ordinary permissions and other grants intact. |
| seccomp | Filters system-call numbers, argument values and metadata. | Cannot dereference a pointer to inspect pathname contents. |
| Landlock | Adds kernel-enforced restrictions on permitted resource operations. | Supported rights vary by kernel interface version; previously opened files need separate handling. |
| Resource accounting and limits | Measure or constrain consumption by grouped processes. | Do not themselves determine which business data a process may read. |
seccomp reduces exposed kernel interfaces; it is not a complete filesystem policy. Landlock supplies additional mandatory kernel restrictions that constrained code cannot simply waive. Inspect both supported rights and inherited handles before relying on the resulting policy.
Changing the apparent filesystem root with chroot does not close descriptors or change the current working directory. An already-open resource outside that tree can remain reachable. Path presentation and actual authority must therefore be checked separately.
Rootless Docker runs both its daemon—the management service—and containers without host root privileges. User-namespace remapping differs: its daemon retains root privileges. Rootless operation reduces those privileges, but mounts, networking, ordinary host permissions and kernel interfaces still need an explicit contract.
A host Docker socket is a management interface, not an ordinary workspace file. Access to a privileged daemon can permit requests to mount host storage into another container. Restricting the original container's visible paths does not remove authority granted through that socket.
Capabilities, credentials and mediated operations
An object capability is an unforgeable reference that identifies a resource and conveys authority to use it. Naming a resource is insufficient: a caller-supplied pathname does not grant permission to write there. The Confused Deputy explains how a service can accidentally apply its own stronger authority to a caller's request.
Linux capabilities use the same word for a different mechanism: dividing operating-system privileges. Ambient authority means broadly available access, such as access inherited from an execution identity, rather than a narrowly supplied resource reference. Minimizing it makes the workload's usable powers easier to identify.
Keep credentials behind checked operations
ExampleWithholding a token still leaves broker authority to constrain.
Read the diagram as text
- Workload.
- Trusted execution identity.
- Broker grant checks.
- Broker-held credential.
- Authorized service operation.
- Rejected request.
- Workload → Broker grant checks: Data: requested operation.
- Trusted execution identity → Broker grant checks: Control: authenticated context.
- Broker-held credential → Broker grant checks: Secret: retained outside workload.
- Broker grant checks → Authorized service operation: All checks pass: attach credential.
- Broker grant checks → Rejected request: Any check fails.
A credential broker is a trusted service that performs authenticated operations for a workload. A useful proposed grant binds trusted execution identity, target resource, permitted operation and validity period. Identity must come from trusted context, not a generated argument. The broker implements grants; AI Security's execution-authority discussion owns the underlying business policy.
Cloudflare's Dynamic Workers gateway demonstrates secret withholding: trusted loader code can attach credentials to selected outgoing requests. Its outbound restriction does not disable separately supplied bindings. Every binding therefore remains an independent grant to inspect.
Direct credential delivery exposes the credential to the receiving process. For example, the MCP TypeScript subprocess client filters its default environment, but explicitly supplied environment entries reach the launched server. Filtering defaults cannot protect a secret deliberately passed to untrusted execution.
| Event | Meaning |
|---|---|
| Lease expiry | The recorded validity or renewal period ends. A lease is lifecycle metadata, not necessarily intrinsic expiry at the external service. |
| Revocation request | The issuer attempts to withdraw authority. This may require contacting another system. |
| Confirmed invalidation | The relevant service no longer accepts the credential. Vault's database-unavailability example shows why expiry alone cannot establish this. |
Credential copies also need lifecycle controls. Environment data can reach child processes; snapshots can retain tokens; logs can preserve secrets after execution ends. Avoid placing broad credentials in workload memory, and exclude tokens from routine diagnostic records.
Workspace paths and underlying storage
A mount makes storage visible at a filesystem path. A bind mount exposes existing host storage rather than copying it. Writes through a writable bind mount change that underlying storage.
OverlayFS combines lower and upper layers into one view. A write requiring modification of a lower-layer file copies it into the upper layer; the upper copy then takes precedence. This copy-on-write behavior separates shared base files from private changes.
Writes follow underlying storage
ExampleA private write layer and a live host mount have different owners.
Read the diagram as text
- Read-only base.
- Workload.
- Private upper layer.
- Live host storage.
- Read-only base → Workload: Read.
- Read-only base → Private upper layer: Copy-up when required.
- Workload → Private upper layer: Workspace writes.
- Workload → Live host storage: Writable bind-mount grant.
| Workspace component | Mutation and retention rule |
|---|---|
| Read-only base | Common tools are readable; changes belong in private writable state. |
| Copied task inputs | The execution edits its own copy, separated from other tenants' working files. |
| Private scratch and candidate outputs | Keep intermediate work separate from files explicitly selected for retention. |
| Live host mount | Changes affect host storage; read-only access is not a snapshot. |
A symbolic link redirects pathname resolution; path traversal uses names such as parent-directory components to reach outside an intended tree. Trusted readers must constrain resolution, not merely check a string prefix. Linux openat2 can require containment beneath a directory and reject symlinks or mount crossings. These checks protect individual opens, not every later processing step.
Device files and management sockets expose operations beyond ordinary file contents. Existing descriptors can preserve access even after a path becomes invisible. The workspace specification must enumerate these grants alongside data paths; access governance determines which data access the environment should permit.
Network reachability and service authority
Egress is outbound communication; ingress is incoming communication. A network namespace supplies a separate view of network devices and stacks. Connection filters decide which address-and-port connections may occur. An application proxy receives requests on behalf of another program and can apply checks to their application-level contents.
Deny outbound communication unless the task requires it, then mediate the necessary paths. Identity-aware connections can help a destination identify the calling workload, as the Tailscale talk proposes. The receiving service must still decide which resource and operation that identity may use.
Cloud metadata endpoints expose instance information and potentially service credentials through special network paths. EC2 documents IPv4 and conditionally available IPv6 access, plus token-based IMDSv2 requests. Blocking public Internet access does not establish that these local paths are blocked. Token requirements alone do not exclude arbitrary code that can reach the endpoint.
DNS resolution translates hostnames into addresses; redirects instruct a client to contact another destination. Either can change where a validated request ultimately goes. SSRF prevention guidance therefore addresses destination membership, resolved addresses and redirects together. Server-side request forgery means inducing a service to contact an unintended destination using its own reachability or authority.
A reported repository-fetching exploit illustrates the consequence: changing a repository destination caused a tool to send its Git credentials elsewhere. The failure was in destination and credential handling; it did not require escaping the execution environment.
Network-policy declarations require effective enforcement. Kubernetes NetworkPolicy needs a supporting plugin; ingress and egress are independent, applicable allow rules combine, and policy distribution can lag creation. Changes to established connections are implementation-dependent. Readiness must establish the required restrictions before untrusted work starts, rather than merely confirm that a policy object exists.
Preview servers need separate inbound authentication and tenant routing. Arrakis's port-forwarding example explains connectivity into a guest, not permission to use the exposed service.
An allowed destination can still receive private data. Filesystem and network restrictions must therefore be assessed together, including every granted service interface. Disclosure paths explains how private-data access, attacker-controlled content and external communication can combine.
Resource budgets and limit outcomes
A control group, or cgroup, groups Linux processes for hierarchical resource accounting and enforcement. New children inherit membership, but moving a parent later does not move its existing descendants. Establish the execution group before setup begins, and verify that required controllers are enabled.
| Budget | Scope and mechanism | Limit outcome |
|---|---|---|
| CPU rate | Group CPU time per period; cpu.max. | Throttles execution. |
| Accumulated CPU time | Per-process CPU seconds; RLIMIT_CPU. | Soft limit signals; hard limit kills. |
| Elapsed runtime | Service-manager active runtime deadline. | Manager initiates termination under its configured rules. |
| Group memory | memory.high and memory.max. | Pressure/reclaim; possible out-of-memory killing at the ceiling. |
| Virtual address space | Per-process addressable space; RLIMIT_AS. | Memory operations can fail; this is not resident-memory usage. |
| Process count | Group pids.max. | Rejects further process creation. |
| Device I/O rate | Throughput or operations per second; io.max. | Limits rate, not stored bytes. |
| Workspace allocation | Filesystem block and inode quotas. | Hard limits reject excess allocation; soft limits allow a grace period. |
| Individual file size | Per-file growth; RLIMIT_FSIZE. | Signals or failed writes; does not cap total workspace size. |
Resident memory occupies physical memory; virtual address space describes addresses a process can map. An inode represents a filesystem object, so many tiny files can exhaust object capacity before byte capacity. A quota must apply to the execution's intended user, group or project identity on a supporting filesystem.
Sleeping or waiting for a service consumes elapsed time without equivalent CPU consumption. An external deadline therefore serves a different purpose from CPU limits. For example, systemd separates startup, active-runtime and stopping timeouts; unit types and permitted deadline extensions affect whether a configured duration is truly a hard bound.
Output bytes, network volume and remote-operation spending require separate enforcement contracts. A local CPU ceiling cannot cap charges from permitted API calls. Specify the measuring component, concurrent-request accounting and overflow response before claiming a bound; retries can multiply downstream work even after the original caller stops waiting.
Per-execution ceilings also differ from service capacity. Admission control decides whether another execution may begin; supervision and cleanup require capacity of their own. Resource abuse supplies this availability context. Resource configurations affect task outcomes, so guaranteed allocations, hard limits and time budgets belong in execution records.
Environment creation and execution readiness
An image packages executable software, filesystem contents and configuration for environment creation. An image digest identifies content, whereas a tag can be reassigned. Recording a requested tag does not establish which image actually ran.
Dependency preparation is execution. npm's version 8 documentation describes installation lifecycle scripts and Git dependencies whose prepare scripts run before packaging. This versioned example establishes the ordering requirement: confinement must precede enabled hooks, not begin only when the advertised application command starts.
Confinement precedes untrusted setup
ExampleA running environment is not necessarily ready for untrusted work.
Read the diagram as text
- Select base and execution identity.
- Prepare storage and controls.
- Verify effective restrictions.
- Run confined dependency setup.
- Admit workload.
- No workload admission.
- Select base and execution identity → Prepare storage and controls: Trusted preparation.
- Prepare storage and controls → Verify effective restrictions: Check establishment.
- Verify effective restrictions → Run confined dependency setup: Restrictions effective.
- Verify effective restrictions → No workload admission: Missing or failed control.
- Run confined dependency setup → Admit workload: Setup and readiness pass.
- Run confined dependency setup → No workload admission: Setup or readiness fails.
The creation gate should establish execution identity, private storage, effective access restrictions and budgets before admitting setup. Missing enforcement is a no-execution outcome. Readiness then includes the required tools and policy checks, not simply a running process. This is a proposed integration contract whose components must be verified together.
Image reuse, dependency caching and memory snapshots preserve different state. A memory snapshot can duplicate machine identifiers, randomness and cryptographic tokens. Repeated restoration therefore needs an explicit uniqueness mechanism. Giving a clone a new external name does not by itself refresh the state inside it.
Initialization captured before tenant inputs arrive avoids capturing those inputs. A later snapshot can retain tenant data in both memory and storage. Identity refresh addresses duplicated identity; it does not erase arbitrary retained contents. A clean base and fresh private writable state provide a clearer starting contract than undocumented reuse.
Record the actual image, runtime, policy, grants, limits, dependency inputs and cache conditions. Reproducing these conditions reduces environmental variation but does not freeze remote repositories or guarantee deterministic program behavior. Execution provenance explains how to retain this context without indiscriminate data capture.
Termination, cleanup and environment reuse
A caller timeout means the caller stopped waiting. A cancellation request asks execution to stop. Neither establishes observed termination. Temporal's activity documentation illustrates the distinction: cancellation is cooperative, and a timed-out attempt can continue. The execution service needs independent evidence about the actual unit it controls.
Stopping only the main process can leave descendants active. Whole-unit stopping must target the execution's group. Linux cgroup.kill targets a subtree, including concurrent forks; cgroup.events can report whether live processes remain. Process absence still says nothing about retained storage or credentials.
Stopped can still mean unclean
ExampleConfirmed termination does not establish completed cleanup.
E owns active execution.
Read the diagram as text
- Environment E.
- Running.
- Stop requested.
- Termination confirmed.
- Cleanup unresolved.
- Quarantined.
- Environment E → Running: Initial state.
- Running → Stop requested: Stop initiated.
- Stop requested → Termination confirmed: Whole-unit absence observed.
- Termination confirmed → Cleanup unresolved: Cleanup incomplete.
- Cleanup unresolved → Quarantined: Reassignment denied.
- Running. E owns active execution. Active: Environment E, Running. New: Environment E, Running.
- Request. A stop request is not completion. Active: Environment E, Running, Stop requested. New: Stop requested.
- Confirm. The execution unit is now absent. Active: Environment E, Running, Stop requested, Termination confirmed. New: Termination confirmed.
- Inspect. Retained-resource cleanup remains unresolved. Active: Environment E, Running, Stop requested, Termination confirmed, Cleanup unresolved. New: Cleanup unresolved.
- Hold. E remains unavailable to another tenant. Active: Environment E, Running, Stop requested, Termination confirmed, Cleanup unresolved, Quarantined. New: Quarantined.
An execution lease is a proposed time-bounded permission to remain active. Its enforcer must survive loss of the requesting controller. Manager timeouts are one building block, not evidence of complete orphan discovery, storage cleanup or authoritative status reporting. Those failure paths require an implementation and tests.
Cleanup also releases references and retained resources. A namespace can survive its last process through an open descriptor or mount reference. Network-policy changes may leave established connections intact. Destruction therefore needs checks appropriate to storage, references, connectivity and authority, beyond a process exit code.
Use finally-style cleanup for ordinary completion and exceptions, plus maximum lifetimes for abandonment. Neither can establish cleanup after every controller failure. Return exit reason, termination evidence and cleanup outcome separately; an unresolved cleanup state must remain unavailable for reassignment.
Pooling trades repeated initialization for a reset obligation. Before reassignment, establish that processes, memory, writable storage, credentials, connections and caches cannot expose prior-tenant state. A successful task is not reset evidence. When reset is unverified, creation from the recorded clean baseline is the defensible alternative; continuation policy remains in Harness Engineering.
Artifact export and retained copies
Artifacts are retained outputs: files, patches, reports and logs. A proposed export manifest names selected relative paths, their execution origin, size/count limits and intended use. Selection separates useful results from disposable workspace contents; recording origin does not validate those results.
Trusted exporters must resolve paths under an approved directory. openat2 can reject symlinks throughout a path and prevent mount crossings; O_NOFOLLOW only addresses the final component. These constraints do not validate contents or secure a later multi-step export against every race.
Export is another trust boundary
ExampleAccepted extraction does not establish safe downstream use.
Read the diagram as text
- Selected workspace files.
- Trusted exporter.
- Path, type and resource checks.
- Retained artifact.
- Rejected or partial output.
- Downstream-use checks.
- Selected workspace files → Trusted exporter: Selected data.
- Trusted exporter → Path, type and resource checks: Controlled reading/extraction.
- Path, type and resource checks → Retained artifact: Export checks pass.
- Path, type and resource checks → Rejected or partial output: Check or extraction fails.
- Retained artifact → Downstream-use checks: Still-untrusted content.
Archive extraction adds link, special-file and resource risks. Use fresh destinations, reject unnecessary links, limit sizes and counts, and handle concurrent modification. Extraction filters are partial defenses; failed extraction can leave partial output.
Program-reported success, process exit status, accepted export and verified correctness are different observations. A patch can export successfully and still fail tests. Verification in a separate environment receives only transferred artifacts; verification inside the workload environment may inherit modified tools and state.
Parsing, rendering or executing an exported artifact opens another exposure. PatchPilot illustrates a useful authority split: the agent edits local files, while a deterministic controller handles repository writes and review handoff. Basic file checks support that split but do not establish code safety.
Logs are outputs too. Exclude access tokens and other unnecessary secrets, sanitize untrusted event fields, restrict readers and define disposal. Exported files, snapshots, caches and telemetry can outlive the workspace. Lifecycle fulfillment across derivatives explains why deletion at one location does not establish deletion everywhere.
Snapshots, rollback and external effects
A checkpoint is a saved recovery point; a snapshot captures specified environment state. Disk checkpoints can preserve installed packages and files for restoration on another node. Incremental snapshots retain changes between checkpoints, reducing repeated copying while making restoration depend on the saved lineage.
Running-state recovery needs more than files. Arrakis describes pausing the guest while capturing memory and writable filesystem state, then resuming. That coordinated capture has broader scope than disk persistence alone.
Local restoration leaves remote state
ExampleRestoring disk state cannot resolve a missing remote acknowledgment.
C captures S's disk before changes.
Read the diagram as text
- Sandbox S.
- Service record R.
- Disk checkpoint C.
- Local edit.
- Remote update committed.
- Acknowledgment missing.
- Local disk restored.
- Remote reconciliation pending.
- Sandbox S → Disk checkpoint C: Capture disk.
- Sandbox S → Local edit: Local write.
- Service record R → Remote update committed: Service commits.
- Remote update committed → Acknowledgment missing: Response lost.
- Disk checkpoint C → Local disk restored: Restore S.
- Acknowledgment missing → Remote reconciliation pending: Outcome unconfirmed.
- Save. C captures S's disk before changes. Active: Sandbox S, Service record R, Disk checkpoint C. New: Sandbox S, Service record R, Disk checkpoint C.
- Change. Local editing and remote mutation both occur. Active: Sandbox S, Service record R, Disk checkpoint C, Local edit, Remote update committed. New: Local edit, Remote update committed.
- Lose response. The caller lacks confirmation of R's update. Active: Sandbox S, Service record R, Disk checkpoint C, Local edit, Remote update committed, Acknowledgment missing. New: Acknowledgment missing.
- Restore locally. S returns to C; R still needs reconciliation. Active: Sandbox S, Service record R, Disk checkpoint C, Local edit, Remote update committed, Acknowledgment missing, Local disk restored, Remote reconciliation pending. New: Local disk restored, Remote reconciliation pending.
| State or effect | What recovery can change |
|---|---|
| Private workspace edits | Discard private state or restore a captured filesystem version. |
| Running guest state | Restore captured memory and machine state with the required backing resources. |
| Host bind-mount writes | Recover the affected host storage separately. |
| Exported patch | Control its separate retained copy and any subsequent incorporation. |
| Remote record update | Inspect the receiving service; local rollback cannot establish its outcome. |
An uncertain outcome occurs when an operation may have completed without confirmed acknowledgment. A remote record can change before the response disappears. Restoring local state neither reverses that change nor supplies the missing observation. Retain the operation identity and reconcile—compare local expectations with authoritative service state—before deciding what further action is needed.
An idempotent retry repeats one logical operation without duplicating its intended effect, when the receiving service enforces that contract. A caller-generated key in a local log is insufficient. A compensating action is a separately authorized corrective operation, not restoration of a sandbox snapshot. Uncertain effects and safe retries develops the recovery mechanics.
Boundary failures and remaining containment
An escape crosses an intended isolation boundary through a failure in implementation or enforcement. A crash inside the boundary does not establish escape: the relevant evidence is an effect outside the protected region. The V8 discussion illustrates this distinction without supplying a container-escape mechanism.
| Failure class | Example path | Relevant repair |
|---|---|---|
| Implementation escape | A flaw lets code cross an intended runtime boundary. | Repair and reduce exposed interfaces; reassess the compromised boundary. |
| Excessive grant | A privileged Docker daemon accepts a request exposing host storage. | Remove or narrow management authority; no exploit is required. |
| Harm within granted authority | Code reaches an allowed API and performs an unwanted operation. | Constrain service authority and permitted data flows. |
The attack surface is the set of interfaces an attacker can influence. Memory-safe VMM code and narrowly jailed device backends can reduce exposure. Operators still need to maintain the monitor and guest and host kernels.
Blast radius describes the resources and effects reachable after compromise. PatchPilot's production account identifies host Docker access as a dangerous grant despite surrounding sandboxing. Separating repository-write credentials into trusted orchestration limits direct agent actions, while leaving generated changes subject to downstream review.
Remaining controls help only while their enforcers remain trustworthy. A host supervisor can stop a compromised workload; a supervisor controlled by a compromised host cannot independently establish host safety. Linux process-memory access rules illustrate that the host kernel controls inspection permissions. Sandbox placement does not establish confidentiality against compromise of that same kernel.
Residual risk is the exposure remaining after controls. Shared-hardware side channels—information leakage through shared resource behavior—are one concern beyond ordinary access checks. gVisor explicitly leaves hardware side-channel defenses to the host and platform. Stronger execution boundaries therefore need stated assumptions rather than claims of immunity.
Stopping computation, cutting connectivity and invalidating credentials address different future effects. None recalls data already disclosed or proves that restoration is safe. Containment and verified restoration owns incident coordination and the evidence needed before resuming ordinary operation.
Boundary evidence and workload-based selection
Compare execution designs using the same repository, required tools, access contract and retained output. Keep environment failures distinct from patch failures. SWE-bench's evaluation reports distinguish unresolved tasks from missing results and ambiguous infrastructure-versus-patch failures; a missing result is not evidence that a patch was tested and failed.
| Claim to test | Independent observation |
|---|---|
| Forbidden files and destinations are inaccessible | Inspect protected host/service state as well as denial responses. |
| Granted authority stays within scope | Check the actual target and operation against the intended resource boundary. |
| Setup and descendants remain bounded | Exercise setup hooks, exhaustion and child-process survival; record enforcement outcomes. |
| Outputs cross safely | Exercise traversal, links, concurrent changes, oversized archives and partial extraction. |
| Trials start from the intended state | Inspect the recorded baseline and residue between executions. |
| Controller loss is contained | Inject requester failure; independently observe termination and unresolved cleanup. |
Use a separate, narrowly privileged verifier where independent observation is required. It should read authoritative evidence without gaining mutation or permission-granting powers. A passing check supports its tested property and configuration; unavailable observations remain unavailable. Controlled offline comparisons explains how to keep comparison conditions interpretable.
A cold start creates an environment without a running instance to reuse. Warm reuse retains initialized state. Tail latency describes the slower portion of a latency distribution; noisy-neighbor interference is delay or failure caused by competing workloads. Measure readiness, execution, memory use, interference, teardown and cleanup failures under stated concurrency and resource conditions.
Guest boot and useful task readiness are different endpoints. Firecracker's historical evaluation measured startup to guest init under specified minimal configurations; it did not include application dependency installation. Preserve measurement boundaries and distributions before comparing those numbers with an interactive build service.
Startup techniques remove different work. Content-addressed storage deduplicates image blobs; lazy loading avoids unused reads but can leave sequential access delays. Prefetching anticipates reads from earlier runs. Memory restoration can bypass repeated initialization. Modal's account demonstrates these complementary mechanisms without establishing a universal speed ranking.
Selection requires compatible functionality, demonstrated enforcement and acceptable lifecycle costs together. Extra resource headroom can prevent infrastructure failures, but can also enable different solution strategies. Record both guarantees and hard limits, and retain the execution provenance needed to explain the result.
Open questions
Verified reuse remains difficult because useful initialization and tenant state can occupy the same memory or storage. Progress requires reset tests that detect retained data and duplicated authority across failure paths, not merely successful startup under a new environment name.
Revocation needs observable completion when external services are unavailable. The unresolved design problem is bounding continued credential use while invalidation is delayed; progress would include outage tests and an explicit maximum exposure period backed by service behavior.
Execution budgets remain fragmented across local resources and remote operations. Concurrent calls and retries make a single hard spending bound difficult. Progress would demonstrate shared accounting, defined overflow behavior and bounded in-flight exposure during failures.
Startup optimization needs comparisons that include setup and retained-state constraints. Lazy reads and restored initialization move costs to different phases. Progress would report matched readiness and cleanup distributions for clean creation and pre-tenant snapshots, including changing dependency access patterns.












