World Models

Learned dynamics and the decisions they support

A game character approaches a junction. An upward command turns it only after it reaches the opening. Published Atari predictions illustrate how a small position error changes that command’s consequence. [2]

A dynamics model, also called a forward model, predicts environment changes over time. The decision rule chooses actions; the actual environment produces their consequences. These responsibilities remain distinct even when one system contains all three. [3]

Passive forecasting predicts evolution without choosing interventions. Planning additionally requires executable alternatives, an objective for comparing them, and predictions reliable enough for that comparison. Predicting several futures does not itself select an action. [4]

Interactive generation demonstrates responsiveness, but responsiveness alone does not establish accurate dynamics. The narrated Genie demonstration generates frames in response to navigation; it supplies no physical-accuracy or planning evaluation. Creative scene generation has its own objectives, developed in Generative Media. [5]

The environment need not be visual. Program execution exposes changing variables and source locations. Code World Model proposes predicting such execution traces, including imagined traces produced without running the program. Those predictions remain distinct from observed execution. [6][7]

State, observations and information through time

Environment state contains sufficient information for prediction under the modeling assumptions. The Markov assumption makes earlier history unnecessary once that state is known. An observation is an available measurement, potentially revealing only part of the state. [8]

Identical visible positions can hide opposite velocities and therefore different futures. History can reveal motion; it cannot necessarily reveal an object hidden throughout the observations. Visual occlusion is missing evidence, not evidence of absence. [9][10]

A latent state is inferred internally. A belief state represents a distribution over possible states. Prediction propagates possibilities through action-conditioned dynamics; correction weights them by the fresh observation’s likelihood and normalizes. Zero total likelihood signals incompatible evidence. A recurrent summary is not automatically a calibrated belief. [8]

A deterministic transition gives one next state for specified state and inputs; a stochastic transition gives a distribution. Neither requires that the agent fully observe the current state. The prediction interval must also be explicit: changing the interval changes the transition being learned. [9][13]

Representations that preserve useful dynamics

An encoder converts observations into features; a transition model advances predictive state, and a readout exposes selected quantities. A reward numerically scores an outcome. PlaNet combines deterministic temporal memory with stochastic latent variables, revised by observations. Its planner predicts latent states and rewards without rendering future images. [14]

TD-MPC jointly trains latent consistency, reward and value predictions without reconstructing observations. A value estimates expected cumulative reward from a state under a given action-selection rule. These representation objectives shape task-oriented prediction, not universal state recovery. [16]

Action-conditioned predictions and their evidence

Action conditioning makes a predicted transition depend on an executable input. Its contract specifies magnitude, duration and delay, not merely a name. Agent-action intervals can contain several internal physics steps; observation timing must match the interval being predicted. [12]

A descriptive prompt can change an environment without specifying a movement command. Hadsell’s narrated Genie examples include prompts that alter an ongoing world. Such editing and navigation are different controls; their prediction targets and permissible effects need separate definitions. [17]

The behavior policy—the rule collecting experience—determines which state-action pairs appear in training. A new policy can choose individually familiar actions in an unfamiliar sequence, reach new states, and require unsupported transitions. This is distribution shift induced by action selection. [18]

Action inputs alone do not establish intervention effects. A demonstrator may see an obstacle hidden from the predictor; that obstacle influences both the chosen turn and later contact. This is confounding. Interpreting recorded behavior causally requires sufficiently specified actions, adequate adjustment for shared causes, and action support in relevant states. Prediction and causal effects develops those conditions. [19]

Learning transitions from trajectories

A trajectory records observations, actions and outcomes in time order. System identification estimates dynamics from measured behavior. Training examples must align each action interval with its resulting target; fitting changes parameters, while inference applies them. [9]

Here, $N$ recorded transitions pair state $s_i$ and action $a_i$ with next state $s'_i$; $\theta$ denotes learned parameters. Minimizing negative log likelihood rewards assigning probability to recorded outcomes. This observed-state objective does not assume hidden states are directly labeled. [20]

Teacher forcing supplies recorded predecessors rather than generated ones. The distinction matters because later predictions then avoid earlier model errors. Sequence supervision uses the same principle: the supplied history determines which conditional prediction is trained. [21]

Multi-step objectives also penalize recursively predicted futures. PlaNet’s latent overshooting matches multi-step predictive distributions to observation-informed latent targets without repeatedly decoding images. It is an additional objective, not a requirement of the paper’s final recurrent agent. [14]

Latent agreement alone permits representation collapse: every input maps to the same vector, so agreement is perfect but information disappears. A concrete countermeasure in VICReg penalizes embedding dimensions whose batch variation falls below a threshold. This illustrates objective design; it is not a required component of the world models described here. [22]

Predicted rollouts and action selection

A rollout recursively advances predicted state to construct a possible trajectory. It begins from observed history, then receives no fresh real observations. Actions may be fixed beforehand or chosen by a policy—a rule selecting actions from available simulated information. Stop at the chosen horizon or a modeled episode ending. [15][3]

The planner minimizes predicted cost $J$ over an action sequence of length $H$. Predicted state $\hat{s}_k$ results from earlier actions; $\ell$ is step cost. Terminal cost $V_f$ estimates consequences beyond the explicit horizon. Constraints separately reject forbidden sequences; a finite penalty merely discourages them. [23]

Assume exact predictions in the grid fixture. Legal cells are (1,0), (2,0), (3,0), (2,1) and (2,2). Start at (2,0); the goal is (2,2). Each move costs one. Terminal cost is remaining horizontal plus vertical distance. The planner rejects candidates attempting blocked moves, although the environment leaves the character in place. [4]

Model predictive control executes only the first selected action, then replans from fresh state measurements or estimates. Model error can undermine safety and continuing feasibility. [23]

More candidates, longer horizons and more sampled futures require more transition evaluations. They can improve search or represent uncertainty, but consume decision time. The useful setting depends on what consequences must be anticipated before the next action. [20]

Imagined experience for learning and simulation

Dyna updates action values: estimates of cumulative reward expected after taking an action and then following a policy. Higher values favor selection. Real transitions update these estimates and fit a model; generated transitions supply additional updates. Here, planning includes learning from simulated experience rather than only decision-time search. [24]

Behavior learning in imagination trains a policy using predicted transitions, rewards and values. Original Dreamer separates this process from dynamics learning and actual interaction. Its deployed action model chooses actions without online trajectory search, while still inheriting errors from the model that trained it. [15]

Imagined experience need not span a complete episode. MBPO initializes many short model rollouts from recorded real states, limiting recursive exposure while using the generated transitions for policy learning. [25]

An environment interface needs initialization, action responses, observations and episode-ending semantics. Reset supplies initial conditions; task termination and a chosen rollout cutoff have different meanings. [12]

Simulator-based training must also preserve relevant failures. Mishra describes digital environments with slow loads, stale tabs and persistent errors, where recovery is an action rather than a silent reset. A convenient reset can otherwise remove the behavior the system needs to learn. [26]

Generated pixels do not automatically provide exportable assets or a complete training interface. The Genie Q&A explicitly separates its image output from game meshes. Robotics develops embodiment and transfer; Synthetic Data develops verification of generated supervision. [27]

Uncertainty across states and futures

A model ensemble contains separately fitted predictors. PETS combines within-model transition variance with between-model disagreement, then propagates sampled transitions through trajectories. These approximate uncertainty representations do not supply certified error bounds or make agreement proof of correctness. [20]

A sampled trajectory is one coherent future, not the entire forecast distribution. Its coordinates across time belong together; selecting unrelated positions independently can erase that temporal relationship. [28]

Calibration compares predicted event probabilities with observed frequencies. For a world model, specify the event and horizon: contact within one tick and contact within ten ticks are different targets. Aggregate agreement can hide poor probabilities in the conditions where actions are consequential. [29]

Compounding error and bounded model use

Compounding error means earlier prediction errors change later prediction inputs. At the Atari junction, a small displacement changes whether an upward command turns the character. No universal error-growth rate follows. [2]

Planner exploitation is selective: optimization favors actions that benefit from model errors. In the reported learned Doom environment, a controller found movements that prevented simulated monsters from firing, an exploit unavailable in the original game. Ordinary forecast drift need not involve such optimized selection. [3]

Pessimistic scoring reduces predicted reward where dynamics are uncertain. MOPO’s theoretical bound requires an estimator that bounds model error; its practical variance-based estimator is explicitly heuristic. An uncertainty penalty is not a safety certificate. [18]

Imagined rollouts supply no fresh observations; replanning cannot undo executed irreversible actions. [23]

Prediction tests across horizons and conditions

Validation must identify which information reaches the predictor. Tests refreshed with measurements and tests initialized once answer different questions. Both should use measurements excluded from fitting and horizons relevant to the intended application. [31]

Record both prediction-step count and elapsed units. For claims about new sessions, group related captures before splitting; adjacent frames can otherwise appear on both sides. This applies independent assessment and visual capture boundaries, rather than assuming every file is an independent case. [31]

Measure consequential events alongside appearance. Physion asks whether marked objects will contact, allowing comparison across pixel, object and physical-state predictors without requiring identical rendered outputs. Its passive scenes test physical prediction, not responses to alternative executable actions or improved planning. [33]

For stochastic forecasts, one case supplies one realized trajectory. Best-sample error can improve merely by offering more samples. A proper scoring rule instead rewards the true distribution in expectation. The proposed trajectory energy score combines distances to the observation with distances among forecast samples, assessing the forecast distribution rather than its luckiest member. [28]

Consequential differences between actions

Action sensitivity concerns whether different commands produce appropriate differences. Action ranking concerns whether those predictions preserve the preferred choice. A proposed evaluation protocol compares separately executed alternatives from matched histories; close returns and scarce trials make rankings uncertain. [29]

A resettable junction fixture can restore the same starting state and observation history before each action. Declare obstacle configuration and disturbance rules, execute each branch, and record contact or progress. For stochastic transitions, repeat the branches. Restoring similar screenshots is insufficient if hidden state differs. [34][35]

After an action changes, recompute downstream states and observations. Feeding the old observation sequence to a changed policy does not simulate its consequences. Holding the same realization of external randomness fixed makes a stronger counterfactual comparison than merely repeating an intervention under similar conditions. [34]

The useful comparison concerns consequential differences, including delayed effects and near-ties. Controlled comparisons require separately initialized executions: matched starting conditions do not require identical resulting trajectories. [34]

Downstream usefulness and operating limits

Forecast quality, action ranking and system usefulness are separate claims. Evaluating an optimized policy only inside its training model can reward the same errors it learned to exploit. Test behavior in the target environment or an independently justified reference environment. [3]

Sample efficiency relates achieved task performance to actual environment interactions. Report other resources separately. MBPO’s equal-update comparison helps distinguish benefits of generated transitions from benefits of simply performing more optimization. [25]

Deployment logs can reveal failures absent from constructed cases. Categorizing them by scenario and failure type makes revision more informative than chasing an aggregate score. The resulting operating claim should name tested actions, conditions, horizons and the purpose the predictions serve. [37]

System usefulness includes unresolved outcomes and consequential failures, not only completed runs. Audit the action trail and effects; a fluent account of success does not establish an acceptable result. [36]

Open questions

1. 01

   Transferable predictive state remains unresolved: task-oriented representations may discard information a new objective needs. Progress would preserve consequential action rankings after an objective change without requiring complete retraining or universal image reconstruction. [16]

2. 02

   Reliability under stronger optimization remains difficult because search can discover favorable model errors absent from ordinary forecast tests. Progress would maintain reference-environment performance as the planner explores more alternatives, rather than merely increasing imagined return. [3]

3. 03

   Efficient intervention coverage remains difficult when hidden conditions influence recorded actions and some alternatives are rarely attempted. Progress would identify which additional controlled trials establish useful action comparisons without treating unsupported extrapolation as causal evidence. [19]

4. 04

   Long-horizon distribution assessment remains difficult because each case reveals only one future and best-sample metrics reward extra samples. Progress would improve distribution-sensitive scores across held-out conditions while also preserving consequential event predictions. [28]