Harness Engineering

The harness and its execution contract

An agent harness is the application runtime surrounding an agent’s decisions. It dispatches operations, maintains records, schedules work, enforces limits, handles interruptions, and checks results. The agent loop is one component; an evaluation harness instead runs cases and judges outcomes. [1]

The model proposes an operation; application code executes it. A generated function name and arguments neither perform the action nor establish permission. Agent Engineering explains this decision boundary. The diagram below follows one dispatch through the surrounding runtime. [2]

Durable execution preserves enough recorded progress to continue after process loss. Session storage, the harness process, and its execution environment can have separate lifecycles. Replacing a failed harness need not erase its session, while replacing an environment need not preserve every file it contained. [3]

The running example is a constructed task: investigate a failed data-processing job, perform a permitted corrected rerun, validate required output partitions, and deliver a report. A partition is a separately identified subset of data. The test fixture supplies the expected partition set and checks resulting state independently of the agent’s account. [4][5]

The fixture’s job service atomically deduplicates submissions by task-scoped operation key throughout each test, returning the same job for matching requests and rejecting changed arguments. Other APIs may differ. [6]

A real submission interface illustrates the separate controls: EMR Serverless accepts an idempotency token, execution role, job configuration, retry policy, and timeout, then returns a job identifier. Submission acceptance establishes that work was submitted; it does not validate its output. [7]

Isolation restricts execution access and resources. Those restrictions must be enforced outside the requesting program’s control. Sandboxes and Execution Isolation owns the enforcement mechanisms; the harness must select and use an environment whose boundaries match the task’s authority. [8]

Task identity and execution states

Task identity follows the user’s objective across interruptions. An attempt identifies a particular execution pursuing it. Temporal’s distinction between Workflow ID and Run ID illustrates stable business identity across executions, although a Temporal run is not a worker-process attempt. The following identifiers describe the example application. [9]

A state machine names states and permits transitions only under specified conditions. Keep worker activity, operation knowledge, and task outcome separate. A worker can stop while a job continues; an operation can have an unknown outcome while its task awaits reconciliation. Protocol task states may summarize a richer application lifecycle. [11][12]

Unknown outcome describes missing knowledge, not remote failure. External reality, recorded observations, and the information supplied to the model remain distinct, as explained in task-state representation. Task-versus-attempt identity prevents an interrupted attempt from silently becoming a new user request. [16]

Checkpoints and authoritative records

A checkpoint is a saved recovery point with an explicit coverage boundary. Graph checkpoints can retain values, pending nodes, interrupts, and execution metadata. They do not automatically capture arbitrary process memory or remote-service state. Application-specific budgets and artifact references survive only if the application records them. [17]

A journal is an ordered record of transitions. A current-state row supports immediate decisions; a journal supports investigation and reconstruction. Both can coexist. Event sourcing goes further by deriving current state from recorded events, but a harness does not need that architecture merely to retain useful history. [22][21]

A database transaction commits related updates together. When job J is observed, one local transaction can save the observation, change K’s recorded status, and create the next pending check. A crash must not expose only part of that transition. The transaction does not include an unrelated job-service request. [23]

Conversation history records interactions; model context selects information for a particular decision. Summaries can omit details, and a smaller context does not imply less durable task state. Context Engineering explains that selection boundary. Recovery should consult authoritative records rather than treating a progress summary as their replacement. [24][3]

Filesystem snapshots preserve their specified files and installed dependencies, not necessarily running processes or later writes. Disk checkpoints can restore a workspace on another node. Environment snapshot limits explain why that restoration neither establishes nor reverses external effects. [25]

Checkpoint granularity and persistence timing are separate choices. Smaller execution units reduce work repeated after interruption but create more recording boundaries. Synchronous persistence waits for recording before proceeding; asynchronous persistence may still be outstanding when failure occurs. Checkpointing only at exit exposes more intermediate work. No interval is optimal independently of workload and storage cost. [26]

Recorded intent and external effects

A retry repeats one logical operation. Idempotency prevents additional effects from repetition. Receiver enforcement matters: identical arguments can also represent separate intended operations. [10]

A transactional outbox commits dispatch intent with the initiating database change. A relay sends committed entries. At-least-once delivery permits duplicates; consumers must deduplicate. [27]

Retain K’s immutable request, authority reference, delivery attempts, and returned evidence. The lost-response boundary remains: a remote effect can commit before its completion is recorded locally. [28][29]

AWS’s receiver contract defines key scope, rejects parameter changes, coordinates deduplication with mutation, returns equivalent results, and limits retention. Unprotected check-then-write permits concurrent duplicates. [6]

Retention limits matter. Stripe, for example, may prune keys once they are at least 24 hours old; subsequent reuse can initiate a new request. A key cannot justify indefinite retries, and changing it to escape an ambiguous response can create another effect. [30]

Scheduling and ownership

Eligibility records whether work may run; a due time records when it may run. A queue notification tells a worker to inspect that state. It does not establish ownership. Even SQS visibility timeouts can coexist with duplicate delivery, so notification suppression cannot substitute for a conditional claim. [31]

The outbox retains notifications across crashes. Duplicates compete for one conditional claim, without creating new tasks. [27][13]

A lease grants temporary ownership until expiry unless renewed. A heartbeat reports liveness or progress; it does not prove task success. A fencing token identifies an ownership generation that a mutation recipient checks before accepting work. Chubby’s sequencers illustrate recipient-enforced rejection of obsolete ownership. [32][33]

Require the expected ownership generation and state version when updating progress. If reassignment changes either, the old worker’s update no longer matches. PostgreSQL rechecks an UPDATE condition after a conflicting update commits. Every relevant writer must use the guard; protecting this row does not fence a remote API. [13]

Backpressure limits admission when downstream capacity is unavailable. Concurrency limits bound simultaneous work; rate limits bound its arrival rate. A circuit breaker temporarily stops calls to a repeatedly failing dependency and later probes recovery. These controls contain load; they do not establish unique external effects. [34][35]

Durable waits and continuation

A continuation records where execution should resume and what remains pending. Conversation history supplies prior interactions; control metadata identifies the suspended operation and required input. Persisting that information allows the worker to exit instead of remaining alive throughout a human response or external job. [36][14]

A signal supplies external input to a pending workflow. A correlation identity connects it to the intended task and operation. A durable timer retains a wakeup deadline independently of a process. A lost wakeup occurs when an event arrives before a listener is ready and no retained record bridges the gap. [37][38]

For the example, persist incoming events in an inbox. Serialize wait registration and event matching, then commit event consumption with newly eligible work. An early completion event remains matchable; duplicates cannot advance the same wait twice. This is a proposed local protocol whose crash boundaries need testing. [37][23]

Azure’s durable external events buffer early events for existing instances but discard events targeting nonexistent instances. Buffering therefore has a lifecycle boundary, not merely a connection boundary. [37]

Resumption must recheck changing observations and applicable authority. An approval belongs to a specific actor, action, arguments, scope, and lifetime; preserving a remembered yes is insufficient. Approval freshness determines whether a suspended action remains permitted. [39]

Restoration, replay, and diagnostic evidence

Restoration loads saved state. Recorded replay reconstructs control flow using saved outcomes. Fresh execution makes new calls. These mechanisms can coexist: a runtime restores task metadata, replays completed activities, then dispatches genuinely pending work. An activity is a separately executed unit whose recorded result can be reused. [40]

Deterministic replay concerns workflow control flow, not deterministic model generation. Temporal compares commands against recorded history; reordering an activity and timer can make replay incompatible. Model calls, database queries, and other external operations belong outside that replay path. Automatic resumption should stop when new code cannot interpret the existing history safely. [43]

Record state-schema, runtime, model, tool, and environment versions with the execution. Recreate workspace resources separately from logical progress, and keep unresolved operations pending reconciliation. Recorded evidence and fresh execution explains why reproducing configuration does not turn a new call into historical evidence. [42][20]

A trace connects execution records; a span records one timed operation. Observability owns this vocabulary. The harness should correlate task, attempt, operation, owner, state version, wait reason, usage, and verification results so diagnostic records can be joined to the execution they describe. [44][45]

Recovery records and diagnostic exports serve different purposes. In a test that suppresses a trace export, a committed waiting state still establishes the pending obligation. Missing telemetry does not establish nonexecution; a finished span or recent heartbeat does not establish task success. Identity propagation and diagnostic coverage explain these limits; capture policy governs payload collection. [46][44]

Budgets across concurrency and restart

A task budget distinguishes configured limits from recorded consumption. Track elapsed time, model calls, tool operations, concurrency, and spending in their own units. Tokens are counted model input and output units; complete-request token counts explain request accounting. A post-response spending counter can overshoot when concurrent requests pass admission before either reports usage. [19][47]

Admission permits work to start. A reservation holds allowance for an outstanding request. In one accounting unit, let $C$ be consumed usage, $R$ outstanding reservations, $r$ the requested reservation, and $L$ the task limit. Admission must atomically enforce: [48]

Settlement records actual usage once and releases the unused reservation. A replacement worker must not reset counters or count restored responses again. Unknown usage remains an unresolved liability; worker loss or reservation expiry does not establish that the provider consumed nothing. [19][48]

Retain the original deadline across retries and restarts. Attempt time, retry delay, and local overhead must fit the remaining interval. Monetary reservations provide a hard ceiling only when enforceable request bounds cover actual charges. Environment limits constrain controlled resources; an accounting ledger alone cannot constrain a remote provider. [49][48]

Stopping and observed termination

A stop decision must survive the process that made it. Persist the reason, prevent new dispatch, notify running work, and retain unresolved operations and useful partial outputs. Stop, pause, and transfer decisions define the intent; the harness must implement it through shutdown and recovery. [50]

Cooperative cancellation requires running work to observe and respond to a stop signal. Temporal activity cancellation illustrates the dependency: heartbeats can carry notifications, throttling can delay them, and activity code must react. A request to cancel, its acknowledgment, and observed termination are separate facts. [15]

The caller can stop waiting while remote execution continues. A timeout may arrive after the server has performed the mutation; cancellation does not roll it back. Keep the operation identity and resolve its outcome before treating repetition or reversal as safe. [51]

Provider states can expose this distinction. EMR’s job states distinguish CANCELLING, when stopping and resource release are being attempted, from CANCELLED, when they succeeded. SUCCESS establishes job completion, not report validity or delivery. [52]

Graceful shutdown stops admission and gives controlled work a bounded opportunity to finish or clean up. Escalate when that interval expires, while recording any remaining external work. A task canceled by the user may still contain a completed external operation. Termination and cleanup covers process-level enforcement. [50][12]

Failure classification and bounded retries

Retry policy depends on error type and effect knowledge. Invalid-call loops waste resources. Changed intent requires a recorded decision; do not replace an unresolved operation’s key. [53][10]

Backoff increases delay between retries; jitter randomizes that delay to avoid synchronized retry waves. Follow provider guidance, cap attempts, and stop when the remaining deadline cannot accommodate another attempt. These controls reduce overload; they do not make duplicate mutations safe. [54]

Choose the retry layer deliberately. If an outer layer permits three total attempts and each invokes an inner layer permitting three, one operation can produce nine downstream attempts. This is an upper bound when both policies exhaust, not a measured multiplier. Existing SDK retries count toward the design. [57]

Persist the next eligible time instead of occupying a worker during backoff. At dispatch, recheck current authority, ownership, deadline, and allowance. Exhausted attempts, permanent rejection, or unavailable safe recovery must produce a recorded stopping or escalation outcome rather than perpetually runnable work. [38][58][8]

Reconciliation and compensating actions

Reconciliation compares recorded intent and observations with authoritative state. Within the fixture’s retention guarantee, K’s unchanged request recovers J. [6]

The current owner records J and resumes status checking. Late evidence from an earlier worker can be reconciled without granting that worker authority to update progress. A receiver’s duplicate protection and the local ownership guard solve different problems; recovery requires both where both races are possible. [32][13]

EMR’s submission documentation does not specify lookup by token or the token’s retention window. A production integration needs those destination guarantees before adopting the fixture’s recovery path. [7]

Transport status alone may remain inconclusive. Stripe documents cases where an HTTP 500 accompanies possible side effects and later reconciliation. Correlating provider objects with the local operation can resolve the uncertainty. Replaying a cached response and confirming the business outcome are different observations. [59]

Compensation is a new action intended to counter an earlier effect. A saga organizes committed steps with application-defined compensating steps. Compensation does not erase history: other systems may have observed the original effect, some actions are irreversible, and compensation can fail. Record its identity, authority, budget, and verification separately; unresolved compensation may require manual recovery. [60]

Durable artifacts and validation

An artifact is a retained output such as a report, patch, or file. An artifact manifest identifies that output and its production evidence. Provenance records where it came from; a content digest helps identify its bytes. Neither establishes semantic correctness. SLSA provenance provides a concrete vocabulary for inputs, outputs, and producing executions. [61]

Validate separate properties: existence, readability, format, and domain correctness. For R1, verify that its job reference is J and compare observed partition coverage with the fixture-owned expected set. A well-formed report can still describe the wrong job or omit required data. Verifier limitations define what passing checks can establish. [5]

Stage an exact version in durable storage, validate it, then publish a guarded reference to that version. S3 conditional writes support create-if-absent and replacement conditioned on an ETag. Those checks neither validate content nor transact with a task database; an ETag is not a universal content digest. [62]

A crash after storage can leave a valid but unreferenced artifact. Recovery should reconcile the exact stored version and its validation evidence before finalization. A temporary workspace path is insufficient once that environment disappears. Artifact export covers the boundary between disposable workspace contents and retained outputs. [25][62]

Verified outcomes and durable delivery

A postcondition is a required state after success. Agent Engineering explains that boundary. Worker exit establishes termination; job SUCCESS establishes processing completion; a readable report establishes readability. The requested partition results and exact report version still require their own checks. [52][4]

Commit the verified outcome, exact artifact references, exceptions, and delivery intent together before notification. The transaction preserves pending delivery across crashes. [23][27]

Keep work outcome and delivery state separate. If delivery is required, overall fulfillment waits for evidence at that boundary. Channel acceptance, rendered output, and human attention establish different facts. [63]

Retry only the saved delivery intent and artifact reference. Lost acknowledgments can cause duplicates requiring consumer deduplication; verified work remains available without rerunning processing. [27]

Verification at interruption boundaries

Fault injection deliberately interrupts execution to test an invariant—a property that must remain true. The cases below are test specifications, not reported results. Their structure follows controlled failure testing: establish initial state, inject a failure, restore conditions where recovery is possible, and inspect authoritative outcomes. [46]

Eventual recovery is conditional: restore required dependencies and preserve sufficient authority, allowance, and deadline. Otherwise the expected result is justified stopping or explicit uncertainty. A one-effect assertion requires the fixture’s receiver guarantee. Executable evaluation cases supply the broader method for connecting controlled inputs to state-based judgments. [46][49]

Retain input revisions, submitted changes, runtime version, test commands, timeouts, and logs. Report infrastructure failures separately from task failures. Reset each trial to its recorded environment baseline; dependency versions, cache state, resource allocation, and concurrency can affect execution even when the model is unchanged. [65][66]

Resource guarantees and hard enforcement limits are different experimental settings. Additional headroom can prevent infrastructure failures, but it can also enable a different strategy. Record both before attributing outcome changes to a harness improvement. [67]

A successful retry demonstration covers only its exercised path. One durable-agent demo recovered from injected exceptions but also encountered an unexplained stall. Tests must target the actual interruption boundaries rather than infer comprehensive recovery from a process continuing once. [68]

Convert recurring failure classes into durable checks with actionable remediation. Repository checks can detect missing timeout policies, while runtime tests establish behavior under interruption. Add machinery when it protects a required invariant; a small durable state machine can be sufficient when its ownership, effect, budget, and completion boundaries are explicit. [69][70]

Open questions

1. 01

   Hard spending bounds remain difficult when requests outlive workers and usage arrives late. Progress requires demonstrated limits on individual provider requests plus crash tests showing that shared reservations survive and settle without resetting allowance or double counting. [19]

2. 02

   Long-lived executions need safe upgrades without abandoning useful progress. Changing command order can invalidate history, while preserving it may retain obsolete behavior. Progress includes compatibility tests that identify which histories resume safely and which require explicit migration or intervention. [43]

3. 03

   End-to-end artifact finalization spans storage, validation, task records, and delivery. Independent commits make partially completed handoffs unavoidable. Progress requires crash tests showing that every fulfilled task references the exact validated version and that delivery recovery never regenerates already verified work. [61][62]

4. 04

   Compensation remains difficult when another system has already acted on an effect or the reversal itself fails. Reliable progress requires domain-specific recovery procedures that preserve history, identify remaining obligations, and reach either a verified acceptable state or accountable manual handling. [60]