Code World Model: Building World Models for Computation

How can a model learn to reason about an action before taking it? Code offers a constrained place to study that question: programs have rules, actions change their state, and execution reveals consequences. Jacob Kahn and the FAIR team built Code World Model, or CWM, around this connection between computation and prediction. The goal extends beyond writing programs to learning representations that support reasoning, planning, and decisions.

A world model predicts future observations from past observations and actions. If those predictions are useful, an agent can compare possible actions by their expected consequences before choosing one. This framing does not require an alternative to language models: world modeling defines the prediction problem, while an autoregressive LLM can implement it. The question is what observations and actions to represent in its tokens.

What does it mean to model code itself? A conventional token-based model receives source syntax and predicts more tokens. That representation does not explicitly expose the runtime state those tokens produce. The opening comparison puts a Python character-counting function beside a description of its execution: the same program can be represented both as source and as an account of what happens when it runs.

CWM makes execution an explicit prediction target. A systematic description of a running program can be ingested by a model and emitted autoregressively, just like source text. Rather than learning only how a program should look, the model learns a transition function: given the current program state and the next operation, predict the state that follows.

The concrete example counts occurrences of r in strawberry. A compact Python version makes the state changes visible:

def count_r(text: str) -> int:
    count = 0
    for character in text:
        if character == "r":
            count += 1
    return count

result = count_r("strawberry")

The useful supervision is not just the final result, 3. It is the sequence of executed lines and local-variable values along the way. After s and t, count remains 0; when the first r reaches count += 1, it becomes 1. A trace can separate execution steps with frame separators and record the locals at each step. Those entries map back to source lines, while repeated loop iterations expose how the same line acts on different states. Memory information could also become part of the trace.

This representation need not stop at a single function. Repository execution, distributed systems, and complex programming-contest solutions all offer larger sequences of state changes to model. Natural-language tracing is another possible representation of those dynamics. The common requirement is to describe execution in a form that connects what happened with the program responsible for it.

A program and an agent can both be described through state transitions. For a program, the state includes its data, the action is an execution step, and the result is a new state. For an agent, the state is its current environment, the action might be a command, and the result is the environment after that command.

An ordinary coding agent reasons about a problem, acts in an execution environment, reads feedback, and tries again if necessary. A world model adds the possibility of imagined feedback: predict what an action would do without first executing it.

The proposed benefit is fewer real interactions while exploring alternatives. The agent can defer execution until it is ready to act; predicted feedback remains a model output, not an observed test result.

An autoregressive LLM can learn this process through serialized traces. The sequence contains a state, an action, the next state, and another action. Predicting its continuation teaches the model to generate the transition token by token. Execution traces thus provide a structured reasoning sequence whose intermediate steps describe program behavior rather than only discussing it in prose.

To move beyond small functions, CWM's data pipeline starts with GitHub events and repository-level pull requests. Kahn outlines a progression from collecting PRs to mutating them and predicting changes. Repositories with known passing tests or CI provide an executable starting point: running that code can produce traces grounded in repository behavior. The connection to tests matters because the training material can include what the software does, not only the patch text describing a change.

CWM is a 32-billion-parameter dense transformer. Kahn presents it as a research model that people can experiment with, contrasting its size and dense architecture with a much larger mixture-of-experts model. The team performs both pretraining and post-training, with long-context training included to support reasoning tasks.

The training sequence builds increasingly specialized behavior on top of the base model:

1. Pretrain on a few trillion tokens.
2. Mid-train on domain-specific data, including a long-context phase.
3. Fine-tune for instruction following and reasoning.
4. Post-train with reinforcement learning in a joint setup that includes agentic reasoning.

Execution-aware behavior therefore sits within an end-to-end training program, rather than being supplied solely through a debugger prompt at inference time.

The agent-training interface follows the same transition structure. A prompt leads to reasoning and an action, such as a tool call or code sent into the environment. An environment step produces feedback; the training setup handles tokens, rewards, log probabilities, and potentially compiler output. CWM emphasizes Bash and a relatively small tool surface, making terminal competence central to solving its tasks.

In the software-engineering RL setup Kahn calls SWE-RL, a GitHub issue places the agent inside a repository. Bash commands let it inspect and mutate the environment and change file contents before submitting a solution. Bash is the center of the interface, not its only possible tool: editing, content creation, and submission also appear in the described workflow. The intent is to train in an environment close to the one an engineer actually uses.

Before RL, supervised fine-tuning can bootstrap useful behavior. Kahn describes finding failure modes and using rejection sampling around code tasks the agent struggles with. The companion technical report clarifies the selection step: SFT uses high-quality, test-passing traces, rather than indiscriminately teaching the model to reproduce failed attempts. The example here concerns instantiation logic: the agent reasons about what it needs to find, then searches for the code through an explicit grep function. Even with an emphasis on Bash, targeted search remains part of the working tool interface.

Scaling post-training makes throughput a central systems problem. Samplers run the model against terminal environments and produce reasoning trajectories. Trainers consume those trajectories, score them, compute gradients, and update the authoritative model. The updated model must then reach the samplers so that subsequent experience reflects newer behavior.

This is a producer–consumer pipeline with traffic in both directions: trajectories move toward training, while checkpoints move toward sampling. Weight synchronization can make one side wait for the other. CWM uses a highly asynchronous design, sending checkpoints and completed sampling work eagerly rather than making the whole system advance together.

Queues allow multiple checkpoints and trajectories to be in flight. The aim is to keep both sampling and training occupied despite their different completion times. Kahn describes this arrangement as achieving strong throughput while remaining relatively on policy; the talk supplies no numerical throughput comparison or formal policy-lag bound.

The asynchronous design goes further than updating workers between tasks. A sampler can start a trajectory with one checkpoint, execute Bash commands or code, receive output, and then continue the same interaction using a newer checkpoint. Model weights can change before the trajectory ends. That changes the policy generating subsequent actions without requiring the ongoing environment interaction to restart.

A trajectory produced by multiple checkpoints is not entirely on policy with respect to one fixed model. Kahn acknowledges this tradeoff and attributes the system's practical robustness to its throughput and data volume. That is an account of why the design works in this training setup, not a general guarantee that arbitrary checkpoint changes are safe. Queuing models and trajectories removes the need to wait for every ongoing rollout to finish before making progress.

Kahn reports processing roughly 200-plus billion tokens during post-training, over a relatively small number of steps at large scale. He describes the resulting model as versatile and effective with Bash tools. These are qualitative capability claims here; the talk does not attach benchmark scores to them.

What does execution training make possible at inference time? Given a function, CWM can predict a line-by-line trace and the values of local variables at particular points. Kahn describes these predictions as highly accurate, without supplying a numerical evaluation in this passage. Their value is that they expose intermediate behavior: the model can discuss where a computation is going, not merely produce a final answer.

That leads to a neural debugger as an interface for writing code. Suppose a partial program leaves the assignments to left and right unspecified. With a conventional prose prompt, the programmer must explain the ambiguity and describe the intended values. With execution-aware assistance, the surrounding code can carry much of that intent: an existing loop, a condition, and an unassigned variable constrain the missing pieces.

The model can reason through the loop and condition to infer how a proposed assignment would behave. Simulated execution connects the structure the programmer has already written with the completion they need. This supports composing alongside code: the programmer fixes the shape of the computation while leaving selected details open for the model to infer. The trace is still a prediction, but it gives the interaction a more precise object to reason about than an underspecified request to finish the program.

Termination is a more ambitious test of the same idea. Running a program until it stops can establish termination for that execution, but if it never stops, waiting cannot finish the check. The general halting problem admits no universally correct decision procedure. A learned model does not remove that limit.

The narrower research question is whether predicted execution can reveal patterns that support useful termination judgments on particular programs. A model might recognize the high-level dynamics of a loop without literally generating every execution step. Such a judgment is an approximation, not a proof that works for all programs.

The same motivation applies when execution is possible but expensive. Debugging a large distributed system may require costly interactions; even one function can be expensive to run. An internal model of the computation could help explore likely behavior before paying that cost. Kahn closes this part of the talk by proposing learned simulation as a way to approximate difficult execution questions. The broader distributed-system applications remain research ambitions, rather than demonstrated reliability results.

CWM is offered as an artifact for researchers and builders to extend. The model weights on Hugging Face and the official GitHub inference code provide the starting points for experimenting with its behavior and lower-level inference details. Availability does not mean unrestricted use: the current model card specifies gated access and a noncommercial research license for the weights.

The technical report expands the training description, including the asynchronous post-training system, data used for execution training, and possible applications. Those details make the release useful for investigating the question that drives the work: what becomes possible when a model learns not only to write code, but to predict the computation that code produces?