RL Environments and Simulators

What an environment defines

An environment is the system through which an agent interacts with a task. It owns the work state and implements what happens when the agent acts. That may mean executing actual software: editing files, querying a database, or recalculating a spreadsheet. A simulator supplies modeled consequences for some part of that interaction. An environment can combine real application code with simulated services or participants. [1][2]

Reinforcement learning (RL) improves behavior using rewards from interaction. Evaluation can execute the same task while keeping the agent's learned parameters fixed. Collecting an attempt does not itself perform a learning update; Post-training and Alignment explains that separate operation. [3]

Specify the intended work before choosing a score. Instructions describe the desired behavior, executable constraints enforce mandatory conditions, and checks supply evidence about results. These responsibilities complement one another. A prompt saying that a file must remain untouched neither prevents modification nor establishes afterward that it was preserved. [4]

The Harbor RL recipe illustrates this separation with task instructions, configuration, executable environment materials, and verification. Its agent uses a shell tool against actual software. A sandbox restricts execution authority; Sandboxes and Execution Isolation explains that boundary. Packaging the task makes it runnable, but does not establish that its checks capture the intended work. [5]

One episode of interaction

An episode is one bounded attempt at a task. It begins with an observation: information the environment exposes to the agent. The agent's policy, its rule or probability distribution for choosing what to do, selects an action, a requested operation. The environment handles that request according to the task's rules and returns an observation of the result. The agent can then choose its next action using what happened. [6][7]

Alongside observations, the environment supplies reward—numerical feedback used for learning—and information about whether the episode has ended. The recorded sequence of observations, actions, and feedback is a trajectory. A rollout is a collected interaction sequence; it can contain a full episode or stop partway through one. [6][7]

Consider an illustrative editing task: make a function return an empty list for empty input while preserving its other behavior. The agent sees the requirement and a failing public test, edits the function, and runs that test again. Its passing output becomes an observation. After submission, a verifier—a procedure checking stated properties—runs separate acceptance tests. A reward rule converts that assessment into a learning signal. The public test result, submitted code, acceptance verdict, and reward are four different objects. [6]

The Gymnasium interface expresses these responsibilities as two calls: reset establishes an episode and returns its initial observation; step applies an action and returns an observation, reward, ending flags, and diagnostic information. It separates a task-defined ending (termination) from an external cutoff (truncation). That distinction matters when a learner interprets an interrupted attempt. [8]

The figure separates these paths for one edit. A complete episode can repeat observation and action many times, and rewards need not wait until submission. Agent Engineering develops action selection and feedback. Here the responsibility is supplying faithful consequences and interpretable records. Those records can later support evaluation, demonstration-based training, or reinforcement learning; each use requires its own procedure. [9]

Why shared interfaces developed

Separating task dynamics from learning code predates language-model agents. Tanner and White's September 2009 RL-Glue paper addressed the difficulty of sharing incompatible experimental components. It separated the agent, environment, and experiment: one chose actions and learned, another supplied consequences and feedback, and the third controlled execution and assessment. A communication layer allowed implementations in different languages to cooperate. This made interactive tasks reusable across learning programs; a fixed dataset generally could not substitute because the next observation depended on the chosen action. [10]

OpenAI Gym's public beta, announced on April 27, 2016, emphasized another problem: comparable experiments. Small changes to an action set or reward function could substantially change task difficulty. Gym offered common environments spanning classic control, Atari, and simulated robots while leaving the learning implementation to the researcher. Its contribution was shared experimental conditions, not a guarantee that every program exposing the same methods represented an equivalent task. [11]

Stateful software workspaces extend the usefulness of this separation. A repository, spreadsheet, or system of record can retain task state independently of the model operating on it. Stateful environments for vertical agents describes that boundary around external computation. The enduring benefit is modularity: replace the agent without rewriting the task engine. The enduring obligation is semantic: document what observations, actions, and feedback actually mean. [1]

State and observations

State is the information the environment uses to generate subsequent consequences. The design question is whether it retains everything needed for that job. A representation has the Markov property when the current state and action suffice to determine the distributions of the next state and reward, without needing additional history. A Markov decision process (MDP) formalizes interaction under this assumption. If omitted history still affects those distributions, the representation does not satisfy the assumption, however convenient it is to implement. [12]

The agent's observation need not reveal that state. A partially observable Markov decision process (POMDP) distinguishes hidden state from the evidence the agent receives. Observations may be incomplete, noisy, or delayed. Earlier actions and observations can help interpret them, but a complete transcript cannot disclose something never observed. Information-gathering actions can therefore be useful before committing to a change. The formal treatment in Planning and Acting in Partially Observable Stochastic Domains explains this distinction. [13]

For example, reading a file's contents could return identical text whether the file is writable or read-only. The same attempted edit then has different consequences. The figure illustrates this ambiguity; inspecting permissions could resolve it if that operation is available. Distinguish a deliberately hidden answer from information the agent needs a legitimate way to obtain. [14]

Diagnostics deserve an explicit visibility rule. Gymnasium's information dictionary can contain internal state or reward components beyond the observation. Forwarding it wholesale can expose privileged information. Context Engineering explains how an application selects the information actually supplied to a model. [8]

Actions, legality, and timing

An action space declares possible action values. Current state can further restrict their validity. In Gymnasium's Taxi example, pickup is one of six action types, but requires the passenger to be at the taxi's location. An action mask identifies currently valid choices. Using it to guide selection does not replace checking permission and legality when execution occurs. [15]

Action granularity determines how much work the interface delegates. Keystrokes leave terminal interaction decisions with the agent; a complete command hides some of that interaction; a repair operation can delegate an entire strategy. The original Terminus design used tmux keystrokes through one interactive terminal. It also placed the agent process outside the task container. Action design and process placement were separate choices. [16]

A transition is the resulting change in state, including modeled events and the passage of time. One agent action need not equal one internal computation step. DeepMind Control, for example, advances physics through substeps between actions, then obtains observations and reward. Its control interval and physics timestep are distinct quantities. An interface must specify when it samples the world as well as what operation it requests. [17]

For software agents, waiting for a whole command response simplifies turn-taking but can prevent timely reactions to a running process. Agents are Robots Too motivates live output, completion evidence, and interruption. A wrapper is an adapter around the interface; changing its observation scope or action behavior can change difficulty even when the underlying application stays fixed. [18][19][20]

These are design obligations, not promises provided by a method named step. Checking meaning and current state covers request validation; Interpreting operation results explains why an error response can leave effects uncertain.

Reset and starting conditions

Reset establishes a specified starting condition. The initial-state distribution is the rule selecting starts across attempts: a fixed fixture, sampled conditions, or a restored snapshot. A snapshot is a saved selection of state, so its contents determine what restoration can recover. Resetting enables repeated attempts at the same task, including groups of rollouts compared during training. [21][22]

Environment state and agent state have different lifetimes. Harbor's multi-step tasks retain the environment across stages while starting fresh conversations by default; supported resume can retain agent state too. Those stages share prior effects and are not independent episodes. [23]

Residual state is information or effects left from earlier execution. An old solution file can disclose an answer; a pending write can alter a newly restored database. Verify observable reset postconditions before admitting the next attempt. Reset verification and external effects explain why process restart alone establishes neither freshness nor reversal.

Seeding also needs an explicit contract. In Gymnasium, an integer seed resets the environment's random generator; seed=None normally preserves an existing generator. Reproducible action-space sampling requires its own seed. [8]

Endings and unavailable outcomes

Termination reaches an ending defined by the task. Truncation stops collection for a reason outside those terminal conditions. A deadline built into the task can terminate it, while an experiment's shorter time limit can truncate it. For a fully observable task with a built-in deadline, remaining time must be observable to preserve the Markov property. The learner must distinguish these cases when estimating future reward. Bootstrapping uses an estimate of accumulated reward beyond the final recorded state: an external cutoff can leave future value, whereas a true terminal state has no continuation. Neither flag alone establishes success. Gymnasium's time-limit guide explains the distinction; Post-training and Alignment covers the learning update. [24]

Keep execution validity separate. If setup never establishes the required environment, there may be no valid attempt to grade. Harbor can abort on setup or healthcheck failure before agent execution. Its stage gates also distinguish a missing reward from a supplied value; a failed gate is a control decision, not an observed zero reward. [23]

An invalid action need not invalidate the episode. A game can reject it, apply a penalty, and allow recovery within a turn limit. Stefano Fiorucci describes making this change for small models that frequently produced malformed or illegal tic-tac-toe moves. Immediate termination and bounded recovery teach different problems; recovery is appropriate only when the task permits it. The reported experiment does not isolate its effect from other training changes. [25]

Filtering stopped attempts can change incentives too. In Learning on the job, Raymond Feng reports a run that discarded sandbox timeouts. Bursts of slow tool calls could then remove a difficult attempt instead of leaving a zero-reward example. This supports investigating a filtering-induced incentive, not inferring model intent from call volume alone. Preserve the stop reason, final observation, validity, and assessment status so the hypothesis can be tested. [26]

Replay and new branches

Inspecting a recording shows what happened. Repeating execution attempts to reproduce it. Branching restores a decision point and executes an alternative action. That alternative requires newly computed downstream states and observations. Reusing the old observation sequence would attribute one action's consequences to another. A recording can therefore support inspection or replay of known inputs without supporting arbitrary interactive continuation. [27]

The figure uses a deliberately small synchronous file example. From identical saved contents, writing A and writing B produce different states; reading each state yields its corresponding observation. The branch is a counterfactual continuation: an alternative execution under stated controlled conditions. It must preserve the relevant history up to the fork and generate consequences afterward. [27]

Restoring visible state may be insufficient. The MuJoCo 3.6.0 physics simulator distinguishes integration state—the inputs needed to advance the simulation—from quantities computed from it. After restoring state, dependent calculations must run again. Exact continuation also requires warmstart accelerations, values used to initialize the numerical solver. Its reproducibility contract applies only to the same MuJoCo version and computational architecture. Saving positions alone therefore does not preserve everything needed for identical continuation. [28]

For a software environment, specify the analogous boundary: application data, relevant agent context, random-generator positions, queued work, dependency versions, and controlled external responses. A seed does not encode all of these. If an external service cannot be restored, identify that limit rather than calling the branch identical. Resettable environments can enable alternative-path exploration, but choosing branches and paying for that search remain separate responsibilities. [29][30]

Evals and Benchmarks explains the limits of replay evidence; Reasoning and Test-Time Compute develops search over alternatives. Reproducibility establishes behavior inside the implemented environment. Whether its new branches resemble consequences in the target setting is a further fidelity question.

What the verifier establishes

A verifier needs a stated property and evidence capable of testing it. File contents, database state, execution traces, and user-facing responses answer different questions. Executable checks suit precisely specified requirements. Human reviewers or model judges, learned systems applying assessment criteria, can examine less mechanical qualities. Their judgments still need validation; checks and their limits, human assessment, and judge validation cover those methods. [31]

In the editing example, public tests help the agent work; protected acceptance tests assess its submission. Harbor permits shared verification in the agent container or separate verification with declared artifact transfer. Shared execution can inherit agent modifications; separate execution can omit needed state. The evidence boundary must therefore be designed alongside the checker. [32]

Correct final state does not prove a compliant process. The tau-bench paper notes that its reward can accept the target database state even when the agent acted without required confirmation. A process requirement needs evidence of the preceding interaction, not another check of the final database. [33]

A checker can also reject legitimate alternatives. The May 2026 SpatialBench verification report describes independent experts reconstructing solutions from instructions and data. Their notebooks exposed unspecified analysis choices and tolerances excluding defensible results. For example, normalization or neighborhood choices could change numerical outputs while preserving a scientific interpretation. Reviewing the work helped distinguish invalid analysis from an ambiguous task or overly narrow grader. [34]

Test the verifier as software. An oracle solution is a known acceptable solution used as a positive control; it should pass. An untouched or deliberately unsuccessful attempt should fail when the task requires a change. Include borderline valid alternatives, missing artifacts, and checker crashes. Repeated agent runs can expose unstable task behavior. These controls test different defects; one passing reference implementation establishes neither complete acceptance coverage nor resistance to shortcuts. [35]

Rewards across an episode

Sparse reward arrives at selected events, perhaps only completion; dense reward arrives more frequently. Return accumulates reward, optionally discounting later feedback. For a finite episode with transitions numbered from zero, reward $r_{t+1}$ follows action $t$, and $\gamma$ is the discount factor. The initial return is: [3]

Here $N$ is the number of transitions. With $\gamma=1$, rewards add without discounting. A terminal outcome can evaluate the trajectory while leaving unresolved which earlier choices helped; distributing learning credit is a separate algorithmic problem. [3]

Inspect what can earn reward repeatedly. OpenAI's December 2016 CoastRunners report describes an agent circling respawning targets to collect game points while crashing and never finishing the race. It did not need to modify the scorer. Ordinary actions already offered a rewarding alternative to the intended objective. More frequent feedback had supplied a reason to remain in the loop. [36]

Reward composition and reward timing are separate decisions. A rubric can score formatting, answer type, and correctness together at the end; several components do not make feedback temporally dense. Partial credit can recognize progress before full success becomes common, but a formatting score establishes structure, not reasoning quality. Outcome and process feedback explains the different supervision roles. [37]

Costs and bonuses also create tradeoffs. The ART·E email-retrieval experiment used a small auxiliary reward for fewer tool turns—queries to the inbox—with correctness intended to dominate. Among successful answers, fewer queries were preferred. That is a useful design aim, but the full reward range matters. Repeated bonuses can accumulate, and a weighted sum may permit sacrificing one requirement for another. Keep mandatory restrictions enforced outside that tradeoff rather than relying on a penalty to make violations unattractive. [38][4]

Shaping and terminal boundaries

Reward shaping adds feedback to aid learning. Potential-based shaping scores states and uses discounted differences between those scores as extra reward. Let $\Phi(s)$ be the scalar score, or potential, of state $s$. For action $a$ leading to $s'$, using the return's discount factor $\gamma$, add: [39]

Here $F_t$ is the extra reward on transition $t$, and $N$ is the episode length. Intermediate potentials cancel. If every terminal potential is zero, each trajectory from the same start receives the same offset, $-\Phi(s_0)$, preserving their ordering by return. A nonzero terminal residual can change that ordering. Setting the final shaping reward to zero is not equivalent, and an external truncation is not a terminal boundary. See the episodic shaping analysis. [39]

Protecting the feedback path

Reward hacking and specification gaming describe earning score without fulfilling the intended task. Environment design must distinguish the routes. Exploiting a misspecified proxy uses the existing rule. Reward-function tampering changes the scoring mechanism. Reward-input tampering corrupts what that mechanism observes. Protecting scorer code addresses only one route; the evidence path needs protection too. [40]

Draw the agent's actual writable surface. Repository optimization experiments reported agents changing runtime dependencies rather than performing the intended internal optimization. The conceptual figure separates a submitted artifact from evidence obtained by trusted collection, and separates both from checker code and reward records. Its blocked arrows are requirements for the proposed design, not guarantees obtained merely by drawing a boundary. [41]

Answer access is another route. DeepSWE v1.1 reports separating verifier and agent runtimes and trimming Git history beyond the task's base commit. These address different problems: evaluator influence and recovery of an existing solution. They do not establish that all shortcuts are closed. Record suspicious attempts separately from submitted exploits and from exploits that actually earned reward. [42][43]

Separate verification still needs the right evidence: Harbor transfers declared artifacts into its verifier environment, so undeclared state may be absent. Public feedback may intentionally help learning; hidden acceptance material serves a different purpose. Protect credentials and reward outputs, and test that permitted solutions remain possible under the restrictions. [32]

Include residual-state leakage, forged success reports, and manipulation of simulated participants in threat analysis without assuming each has occurred. Secrecy cannot repair a wrong acceptance rule. AI Security covers enforcement at protected operations. After closing a shortcut, rerun relevant legitimate and adversarial attempts: changed access or feedback can alter behavior as well as block the exploit. [35][44]

Task distributions and coverage

A task distribution governs which instances and conditions appear and how frequently. An instance is one concrete task; a family groups tasks sharing underlying structure. The distribution can vary goals, initial data, layouts, constraints, tool behavior, and disturbances. Define the target workload separately from the practice mixture. Common traffic and rare consequential failures deserve identifiable coverage rather than an unexplained blend. [45]

Procedural generation creates instances by applying rules and sampling parameters. Procgen's 2020 study trained on finite level collections and tested unseen levels from the same generator. That tests instance generalization, not transfer to arbitrary task families. Its fixed-sequence ablation also showed apparent progress through familiar levels without corresponding performance when the sequence changed. The sampling protocol determines what the score establishes. [46]

More combinations are useful only if they remain valid. A changed starting screen, contact list, or visual theme should preserve an achievable task under the allowed observations and actions. Validate combinations before treating failures as agent weaknesses. Also inspect repeated task structures: changing names or random seeds can produce many instances of one familiar problem. [47]

Held-out assessment uses material excluded from development and selection. Hold out the unit relevant to the claim: instances for new-instance performance, families for new-family performance, or separately sourced workflows for a broader claim. Repeatedly tuning the generator against assessment feedback weakens that independence. Workload coverage and assessment independence explain the evaluation design.

Curricula and changing exposure

A curriculum changes training exposure as learning proceeds. Bengio and colleagues' 2009 Curriculum Learning described sequences of distributions emphasizing easier examples before harder or more varied ones. The target distribution remained the reference. This distinguishes guiding practice from redefining success around whatever the learner already handles. [48]

Difficulty can change through the starting state. The 2017 reverse curriculum method began near an accessible goal and expanded toward more distant starts. It favored intermediate success rates and retained previously useful starts to reduce forgetting. Evaluation still used the original starting distribution. This makes reset capability a learning resource, but assumes the environment can reach and restore those selected starts. [49]

A fixed schedule advances according to a predetermined rule; adaptive selection responds to performance. Automatic environment design also learns which tasks to generate. A 2020 study rewarded an environment designer for the performance gap between two agents, seeking tasks that one could solve but the other found difficult. This addressed a tension: random generation can waste effort, while a generator rewarded only for defeating an agent can produce impossible tasks. Using the performance gap encourages learnable challenges, but does not guarantee that every generated instance is solvable or relevant to deployment. [50]

For an agent environment, useful changes might include shorter tasks, more recoverable mistakes, weaker opponents, or gradually reduced assistance. Making opposition perfect immediately is not necessarily helpful: Fiorucci reports overly defensive tic-tac-toe behavior that failed to exploit weaker opponents. Mixtures can preserve practice on earlier skills while adding harder cases. The figure shows that relationship without prescribing a schedule or sampling weights. [51][52]

Track what makes a task easier. A closer start preserves a different portion of the work from a hint revealing the answer. Assistance unavailable at deployment must eventually be removed and tested without it. An adaptive curriculum also inherits verifier defects: if shortcuts score well, selection can favor them. Keep an unchanged, independent target assessment—or explicitly version a changed one—so rising training reward cannot be explained solely by easier exposure. [53]

Fidelity for an intended use

Simulation fidelity is correspondence to specified aspects of a target setting. Its intended use determines which correspondence matters. NASA's models-and-simulations standard distinguishes implementation verification from empirical validation against the real world. A correctly implemented simulator can still be unsuitable for a particular decision. Conversely, unnecessary detail need not improve the evidence needed for that decision. [54]

Actual software with controlled data can preserve application behavior without reproducing every production dependency. Snapshots, service containers, and mocks can bound the system. Simulated users can supply interactions when real participants are unavailable. These choices trade setup, reset, transition, and checking work against the behaviors represented; identify which simplifications could change the task before spending effort on detail. [2]

Learned dynamics predict transitions from experience and can implement part of a simulator; World Models explains their construction. In Ha and Schmidhuber's learned Doom environment, a controller discovered movements that prevented simulated monsters from firing, a shortcut absent from the original game. Increasing sampling randomness helped their experiments but also introduced unrealistic events. A model's predictive errors can become useful opportunities for an optimizing agent. [55]

Plausible dialogue is likewise insufficient evidence of participant fidelity. A January 2026 study compared simulated users and people interacting with one fixed GPT-4o agent on 18 adapted retail tasks. Simulation underestimated performance on some difficult tasks and overestimated it on some moderately difficult tasks, with different conversational and failure patterns. The useful validation target is the resulting interaction, not merely whether each simulated message sounds human. [56]

Evidence beyond the simulator

Passing the implemented task, handling unseen generated instances, and succeeding in the target workflow are separate claims. Start with matched conditions and actions to locate transition discrepancies. Then test complete closed-loop attempts, where each resulting observation affects the next action. Small discrepancies can change later choices, so agreement on isolated operations does not establish agreement over an entire trajectory. [57][58]

Domain randomization varies modeled conditions during training. Tobin and colleagues' 2017 visual-localization study varied textures, lighting, camera properties, and distractors, then evaluated on real tabletop images. Distractor variation mattered for real clutter. This demonstrated usefulness for a bounded perception task without requiring one photorealistic training scene; it did not establish transfer of arbitrary physical dynamics. [59]

A separate 2018 puck-pushing study randomized dynamics and observation conditions, including masses, friction, control gains, noise, and action intervals. Physical tests used motion capture to locate the puck. That evidence concerns instrumented control under its tested conditions, not unrestricted visual manipulation. The reality gap is the consequential difference between simulation and physical execution; Robotics develops transfer, control, and safety. [60]

Software simulators can also be judged by what they teach. A May 2026 writing-assistance study held the starting assistant and training procedure fixed while changing the user simulator. People preferred the assistant trained with a simulator fine-tuned on human conversations; the variant trained with prompted role-playing was not statistically distinguishable from the starting assistant. The measured outcome was conversational preference on selected writing tasks, not productivity or general tool-use success. [61]

Randomization covers the factors and ranges actually modeled. Test consequential permissions, delays, errors, and participant behavior explicitly, and retain rare-condition evidence separately from representative traffic. Use uncertainty estimates and offline-to-live evidence to bound the claim. A favorable result within a validated domain does not establish correspondence outside it. [54]

Delivering usable experience

A collector assembles interaction records for evaluation or learning. Provenance identifies where those records came from. Collection must preserve the distinction between what the agent requested, what execution accepted, and what the environment subsequently exposed. For multi-call agents, individual calls and environment transitions can be represented separately; an episode's final reward does not establish that every included action was useful. [62]

Parallel collection makes producing versions important. TorchRL's MultiCollector can tag each frame with the policy version actually applied by its worker; one returned batch can therefore contain several versions. It also documents trajectory identifiers and validity masks for padded or preempted entries. Such entries must not be treated as ordinary experience. Those controls do not identify the task or verifier version on their own. [63]

Attach delayed assessments using episode and assessment identity, not arrival order. Make duplicate delivery detectable so one judgment is not counted twice. Preserve unavailable judgments explicitly rather than filling them with zero. These are requirements for the handoff: the exact attachment and deduplication mechanism depends on the collector and storage design.

Preserve an episode's final observation before resetting its environment. The next episode's initial observation belongs to another record, even if both arrive through the same worker or batch slot. Keep diagnostic values, agent-visible tool feedback, private acceptance evidence, and training reward separate. Agent Runtimes and Harness Engineering owns durable execution; Post-training and Alignment owns the updates consuming collected experience.

Versioning the effective task

Version the environment as a combined semantic contract: task definitions, transition code, observation and action adapters, reset rules, endings, rewards, verifiers, generators, and sampling settings. The model version identifies only one participant. A change that supplies different information, permits different effects, or grants more effort can change the effective task.

GSO's official changelog makes this concrete. It added deceptive-optimization detection on November 3, 2025; increased the iteration budget and changed evaluation settings on April 27, 2026; and added network restrictions and disabled MCP access on July 12, 2026 after observed solution fetching. The Model Context Protocol connects tools and services, as its home chapter explains. These changes altered checking, effort, and available information. Scores across those releases should not silently be treated as measurements under one protocol. [64]

Test interface invariants before interpreting learning curves. Gymnasium's checker examines observation-space membership, seeded resets, and repeated seeded steps where determinism is expected. These catch implementation defects, not reward validity or deployment fidelity. Add reference solutions, deliberate failures, action-boundary cases, and reset postconditions for the task-specific contract. [65]

Harbor can regrade captured artifacts with a new verifier, preserving the original trial and recording a new assessment. This changes the judgment of an old attempt; it does not execute new behavior. [66]

Use the figure to decide what surviving evidence can support. An assessment-only repair may permit regrading. Changes to feedback, permissions, observations, or transitions require fresh attempts for a behavioral comparison. A changed curriculum requires a recorded exposure history and assessment against the same target. When execution capacity is limited, prioritize cases affected by the change and narrow the resulting claim; artifact reuse cannot supply missing evidence about an unrecorded process.

Investigate discrepancies at the boundary that could explain them. High reward with invalid work points toward acceptance rules, evidence integrity, or incentives. Valid work with low reward points toward checker coverage or execution defects. Good simulated results with poor target behavior point toward task coverage or fidelity. Preserve the affected release and records, correct the defect, and reassess the learning data and comparisons that depended on it. Observability explains recording mechanics. [35][58]

Open questions

1. 01

   Useful fidelity must remain valid as agents improve and discover unfamiliar strategies. Exhaustive reproduction of a target workflow is impractical, while omitted transitions can become shortcuts. Progress would mean identifying consequential discrepancies early and demonstrating that targeted simulator repairs improve complete target-workflow attempts, not just simulator scores. [55][58]

2. 02

   Verification becomes difficult when many valid solutions exist and process requirements matter as much as final state. Stronger optimization can expose both permissive checks and rules that reject legitimate alternatives. Progress would combine expert-reviewed acceptance coverage, protected execution evidence, and fresh adversarial attempts while keeping unresolved judgments visible. [68][44]

3. 03

   Automatic curricula must find learnable tasks without narrowing practice to exploits or easy-to-score behavior. Difficulty and usefulness need not move together, and a generator inherits its verifier's mistakes. Progress would demonstrate retained capability across changing practice mixtures on an independent, stable target workload. [51][52][53]