AI Engineer Code 2025
Code World Model: Building World Models for Computation
Read the talk
Code World Model: Learning to Predict What Programs Do
Execution traces turn code into a sequence of observable state changes, giving a language model a way to reason about programs before running them.
From a talk by Jacob Kahn
Before you start: Basic familiarity with Python loops, local variables, language-model training, and reinforcement learning will help.
What should a model learn about code?
How can a model learn to reason about an action before taking it? Code offers a constrained place to study that question: programs have rules, actions change their state, and execution reveals consequences. Jacob Kahn and the FAIR team built Code World Model, or CWM, around this connection between computation and prediction. The goal extends beyond writing programs to learning representations that support reasoning, planning, and decisions.
A world model predicts future observations from past observations and actions. If those predictions are useful, an agent can compare possible actions by their expected consequences before choosing one. This framing does not require an alternative to language models: world modeling defines the prediction problem, while an autoregressive LLM can implement it. The question is what observations and actions to represent in its tokens.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From source tokens to changing variables
What does it mean to model code itself? A conventional token-based model receives source syntax and predicts more tokens. That representation does not explicitly expose the runtime state those tokens produce. The opening comparison puts a Python character-counting function beside a description of its execution: the same program can be represented both as source and as an account of what happens when it runs.
CWM makes execution an explicit prediction target. A systematic description of a running program can be ingested by a model and emitted autoregressively, just like source text. Rather than learning only how a program should look, the model learns a transition function: given the current program state and the next operation, predict the state that follows.
The concrete example counts occurrences of r in strawberry. A compact Python version makes the state changes visible:
python
def count_r(text: str) -> int:
count = 0
for character in text:
if character == "r":
count += 1
return count
result = count_r("strawberry")
The useful supervision is not just the final result, 3. It is the sequence of executed lines and local-variable values along the way. After s and t, count remains 0; when the first r reaches count += 1, it becomes 1. A trace can separate execution steps with frame separators and record the locals at each step. Those entries map back to source lines, while repeated loop iterations expose how the same line acts on different states. Memory information could also become part of the trace.
This representation need not stop at a single function. Repository execution, distributed systems, and complex programming-contest solutions all offer larger sequences of state changes to model. Natural-language tracing is another possible representation of those dynamics. The common requirement is to describe execution in a form that connects what happened with the program responsible for it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Predicting feedback before acting
A program and an agent can both be described through state transitions. For a program, the state includes its data, the action is an execution step, and the result is a new state. For an agent, the state is its current environment, the action might be a command, and the result is the environment after that command.
An ordinary coding agent reasons about a problem, acts in an execution environment, reads feedback, and tries again if necessary. A world model adds the possibility of imagined feedback: predict what an action would do without first executing it.
| Loop | Action | Feedback |
|---|---|---|
| Environment interaction | Execute a command or program | Observe the actual result |
| World-model reasoning | Imagine the command or execution | Predict the resulting state |
The proposed benefit is fewer real interactions while exploring alternatives. The agent can defer execution until it is ready to act; predicted feedback remains a model output, not an observed test result.
An autoregressive LLM can learn this process through serialized traces. The sequence contains a state, an action, the next state, and another action. Predicting its continuation teaches the model to generate the transition token by token. Execution traces thus provide a structured reasoning sequence whose intermediate steps describe program behavior rather than only discussing it in prose.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Building supervision from repositories
To move beyond small functions, CWM's data pipeline starts with GitHub events and repository-level pull requests. Kahn outlines a progression from collecting PRs to mutating them and predicting changes. Repositories with known passing tests or CI provide an executable starting point: running that code can produce traces grounded in repository behavior. The connection to tests matters because the training material can include what the software does, not only the patch text describing a change.
CWM is a 32-billion-parameter dense transformer. Kahn presents it as a research model that people can experiment with, contrasting its size and dense architecture with a much larger mixture-of-experts model. The team performs both pretraining and post-training, with long-context training included to support reasoning tasks.
The training sequence builds increasingly specialized behavior on top of the base model:
- Pretrain on a few trillion tokens.
- Mid-train on domain-specific data, including a long-context phase.
- Fine-tune for instruction following and reasoning.
- Post-train with reinforcement learning in a joint setup that includes agentic reasoning.
Execution-aware behavior therefore sits within an end-to-end training program, rather than being supplied solely through a debugger prompt at inference time.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Learning in an engineer's terminal
The agent-training interface follows the same transition structure. A prompt leads to reasoning and an action, such as a tool call or code sent into the environment. An environment step produces feedback; the training setup handles tokens, rewards, log probabilities, and potentially compiler output. CWM emphasizes Bash and a relatively small tool surface, making terminal competence central to solving its tasks.
In the software-engineering RL setup Kahn calls SWE-RL, a GitHub issue places the agent inside a repository. Bash commands let it inspect and mutate the environment and change file contents before submitting a solution. Bash is the center of the interface, not its only possible tool: editing, content creation, and submission also appear in the described workflow. The intent is to train in an environment close to the one an engineer actually uses.
Before RL, supervised fine-tuning can bootstrap useful behavior. Kahn describes finding failure modes and using rejection sampling around code tasks the agent struggles with. The companion technical report clarifies the selection step: SFT uses high-quality, test-passing traces, rather than indiscriminately teaching the model to reproduce failed attempts. The example here concerns instantiation logic: the agent reasons about what it needs to find, then searches for the code through an explicit grep function. Even with an emphasis on Bash, targeted search remains part of the working tool interface.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keeping samplers and trainers moving
Scaling post-training makes throughput a central systems problem. Samplers run the model against terminal environments and produce reasoning trajectories. Trainers consume those trajectories, score them, compute gradients, and update the authoritative model. The updated model must then reach the samplers so that subsequent experience reflects newer behavior.
This is a producer–consumer pipeline with traffic in both directions: trajectories move toward training, while checkpoints move toward sampling. Weight synchronization can make one side wait for the other. CWM uses a highly asynchronous design, sending checkpoints and completed sampling work eagerly rather than making the whole system advance together.
| Queue | Producer | Consumer |
|---|---|---|
| Model checkpoints | Trainer | Sampling system |
| Reasoning trajectories | Samplers | Scoring and training system |
Queues allow multiple checkpoints and trajectories to be in flight. The aim is to keep both sampling and training occupied despite their different completion times. Kahn describes this arrangement as achieving strong throughput while remaining relatively on policy; the talk supplies no numerical throughput comparison or formal policy-lag bound.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Changing checkpoints during a trajectory
The asynchronous design goes further than updating workers between tasks. A sampler can start a trajectory with one checkpoint, execute Bash commands or code, receive output, and then continue the same interaction using a newer checkpoint. Model weights can change before the trajectory ends. That changes the policy generating subsequent actions without requiring the ongoing environment interaction to restart.
A trajectory produced by multiple checkpoints is not entirely on policy with respect to one fixed model. Kahn acknowledges this tradeoff and attributes the system's practical robustness to its throughput and data volume. That is an account of why the design works in this training setup, not a general guarantee that arbitrary checkpoint changes are safe. Queuing models and trajectories removes the need to wait for every ongoing rollout to finish before making progress.
Kahn reports processing roughly 200-plus billion tokens during post-training, over a relatively small number of steps at large scale. He describes the resulting model as versatile and effective with Bash tools. These are qualitative capability claims here; the talk does not attach benchmark scores to them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Expressing intent through unfinished code
What does execution training make possible at inference time? Given a function, CWM can predict a line-by-line trace and the values of local variables at particular points. Kahn describes these predictions as highly accurate, without supplying a numerical evaluation in this passage. Their value is that they expose intermediate behavior: the model can discuss where a computation is going, not merely produce a final answer.
That leads to a neural debugger as an interface for writing code. Suppose a partial program leaves the assignments to left and right unspecified. With a conventional prose prompt, the programmer must explain the ambiguity and describe the intended values. With execution-aware assistance, the surrounding code can carry much of that intent: an existing loop, a condition, and an unassigned variable constrain the missing pieces.
The model can reason through the loop and condition to infer how a proposed assignment would behave. Simulated execution connects the structure the programmer has already written with the completion they need. This supports composing alongside code: the programmer fixes the shape of the computation while leaving selected details open for the model to infer. The trace is still a prediction, but it gives the interaction a more precise object to reason about than an underspecified request to finish the program.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reasoning about execution that is hard to observe
Termination is a more ambitious test of the same idea. Running a program until it stops can establish termination for that execution, but if it never stops, waiting cannot finish the check. The general halting problem admits no universally correct decision procedure. A learned model does not remove that limit.
The narrower research question is whether predicted execution can reveal patterns that support useful termination judgments on particular programs. A model might recognize the high-level dynamics of a loop without literally generating every execution step. Such a judgment is an approximation, not a proof that works for all programs.
The same motivation applies when execution is possible but expensive. Debugging a large distributed system may require costly interactions; even one function can be expensive to run. An internal model of the computation could help explore likely behavior before paying that cost. Kahn closes this part of the talk by proposing learned simulation as a way to approximate difficult execution questions. The broader distributed-system applications remain research ambitions, rather than demonstrated reliability results.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Building with CWM
CWM is offered as an artifact for researchers and builders to extend. The model weights on Hugging Face and the official GitHub inference code provide the starting points for experimenting with its behavior and lower-level inference details. Availability does not mean unrestricted use: the current model card specifies gated access and a noncommercial research license for the weights.
The technical report expands the training description, including the asynchronous post-training system, data used for execution training, and possible applications. Those details make the release useful for investigating the question that drives the work: what becomes possible when a model learns not only to write code, but to predict the computation that code produces?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Training methods, execution representations, infrastructure and evaluations for Code World Model.
Official setup instructions, inference tools and benchmark reproduction code.
Post-trained checkpoint and usage guidance, with gated access under a noncommercial research license.
Further reading
CLI and notebook examples for stepping through predicted execution and inspecting predicted local variables.
- SWE-RLPaper
Earlier research on reinforcement learning from software evolution data and patch-similarity rewards.
Read the complete timestamped transcript
- 0:00
[upbeat electronic music] Great to be here, everyone.
- 0:22
I'm Jacob Kahn. I'm a researcher at, at FAIR at Meta AI. I'm gonna talk today about the Code World Model, which I'll abbreviate as CWM, and what it means to build world models for computation.
- 0:33
This is work done by an incredible team at FAIR, uh, extends all over the world. I'm very grateful to be collaborating with them.
- 0:42
So what's our goal with CWM? Our primary goal is to build models that reason, plan, and make decisions, and we start with code because it's an interesting sandbox in which to think about reasoning, right?
- 0:53
It's constrained. Uh, there are certain rules with code. And so our, our goal is to predict future observations given past observations and actions. That's maybe what it means to build a world model in some sense, and we wanna do this because we can learn good representations of things if we learn some sort of mapping between observations and
- 1:10
the future. And eventually, that leads us to planning and reasoning, and we can consider different actions and see if we like the results for decisions we make. I think there's a bit of a false dichotomy right now between world models and large language models.
- 1:24
World models are just a parameterization of a problem, as I'll discuss. LLMs are a, a way to, to view and use that parameterization, and I'll, I'll dive into more of what that means in a bit.
- 1:36
So one of the fundamental questions we're asking with CWM is what does it mean to model code? Is code literally the syntax in your editor, or is it something else?
- 1:48
And if you think about it, all a model sees that is operating on code is just syntax, right? We tokenize the input, it goes into the model, and we predict more code as the output.
- 1:58
This is the starting and ending point for an analysis of a program with a token-based autoregressive model. Uh, it's just the syntax. But what if we instead modeled execution more explicitly?
- 2:09
And what if we created a, maybe a natural language systematic description of programs, and neural models could ingest a more structured representation of what it means to execute code.
- 2:20
And then maybe we could emit autoregressively this representation too.
- 2:25
So that's one of our goals for CWM. We wanna predict program execution because we believe it might lead to us better modeling things about code, writing code, analyzing code, and beyond.
- 2:36
And so what we're gonna implicitly do is predict a transition function of program states as we go about executing.
- 2:44
So this is what execution tracing might look like in action. We have a program. We're gonna count the number of Rs in strawberry. And at each step, maybe we'll have some frame separator which will denote distinct lines of execution, and we'll actually explicitly have local variables.
- 3:02
We could in- introduce things about memory in that trace, and that will delineate line by line what's happening as our program executes. And this is something we could ostensibly feed to a model because each line of our execution trace maps to a corresponding line in a program.
- 3:18
We don't have to stop at functions. We could think about entire repository-level execution traces. We could think about distributed system-level execution traces. We could think about modeling execution for code contest solutions or something more complex, programs with high complexity.
- 3:33
We could also then transition that into, as I said, natural language tracing, and we'll see what that means in a moment.
- 3:39
But what does it actually look like to model that transition function at a high level as we start to parameterize the problem? Well, we have programs or we have data.
- 3:47
That's some state. We have an action executing the next line, and that results in the next state. And so both, both the program execution and the model's decision-making in an agentic sense, uh, can be modeled as a transition function.
- 4:03
So where are we? This broader approach, world modeling, we could say in an agentic reasoning setting, we have a problem, we have a model that thinks about the problem, it takes an action in the world, we get some feedback, maybe we fail, we think again, and we iteratively continue this process with feedback from the environment.
- 4:21
Maybe in the sense of code, that environment is just an execution, uh, in a, in a code setting, right? But with a world model, maybe we can actually simulate, we can imagine that action.
- 4:32
We can get feedback in our imagined environment, so we could actually generate execution traces about a program without executing it. And this gives us the ability to be far more efficient with how we actually structure our agentic execution.
- 4:45
We don't have to interact with the real world unless we're ready to.
- 4:51
So let's couple this with autoregressive large language models. Right now, we have a state of a program, we have an action, maybe the next line, and then we get to a new state, we take another action, et cetera.
- 5:03
And so we can sort of turn this with the execution tracing format I mentioned into almost a chain of thought that a model can just interpret. A model can learn to predict the next state of an execution trace.
- 5:15
And so an LLM can autoregressively generate token by token the state and action to state function with program executions as the starting point. Okay.
- 5:27
Let's talk about data for a second. Let's talk about for CWM, we gathered a huge amount of GitHub data. We take GitHub events, and as I said, um, we're interested in modeling things at the repo level if we can, at the systems level if we can.
- 5:42
We wanna have execution traces go outside of the scope of simple programs. And so we'll take a bunch of PRs, we'll mutate those PRs, predict changes, and we'll eventually have a raw PR dataset, and we can actually run tests or CI on those GitHub repos when we know they're passing, and then generate execution traces from that repo-level
- 6:02
data if we want.
- 6:04
So here we are at the artifact, the Code World Model itself. I'll talk a bit about what we did with it, how we trained it, and then what we can do with some of these interesting execution trace capabilities.
- 6:15
But first, it's a thirty-two billion parameter dense transformer. This is a model for research. This is not a huge MoE you can't play with. Uh, you can play with it right now.
- 6:25
It has a, a nice long context length for some reasoning tasks, and we train it end-to-end. We do all the pre-training and post-training ourselves.
- 6:34
Processes, we pre-train on a few trillion tokens. We mid-train on some more domain-specific data. We do some long context mid-training. We fine-tune further, uh, on some instruction following and reasoning tokens, and then we do this joint RL and agentic reasoning setup.
- 6:52
So let's parameterize the problem even more broadly with CWM. We have a prompt, we have an agent, we do some reasoning, we take an action. We can use a tool, we can emit text, which is code that goes into the environment.
- 7:06
We take a step, and from that environment, we get a few things back. We get tokens, we get rewards, we get log probabilities. We might get compiler output. So with CWM, we're also taking a, a big step back with how we interact with the environment.
- 7:19
C- CWM is a very Bash-oriented model. It has fewer tools than do other models, and it has to learn how to use the terminal pretty well to solve a lot of the tasks we give it.
- 7:32
And this starts with SWE-RL, and with SWE-RL, we take a GitHub issue, we feed it to the agent, starting with that repository level data set f- from before, and we just use Bash, right?
- 7:43
We learn commands, uh, in Bash, and that lets us mutate our environment, that lets us mutate the state of files. We can maybe use an edit tool eventually or create content and then submit things.
- 7:55
But ultimately, we're trying to put the model in an environment that's very, very similar to what an engineer would be in and, and learn end-to-end in a Bash-based setting.
- 8:05
Okay. So we can bootstrap this setup further. We can do some SFT before RL, and we can find some failure modes for the model. We can rejection sample, so we can take a bunch of agentic reasoning traces on code tasks that failed, and we can basically feed those back into the model.
- 8:24
So in this example here, we have a thinking trace where we're thinking about instantiation logic for some code, and I can look for that code. I can call an explicit grep function, and this is something we did with CWM, again, with fewer tools and a larger emphasis on Bash as a starting point.
- 8:43
Let's talk about post-training for a moment. We wanna scale post-training quite a bit. This is the trend we see, and we're getting a lot of excellent returns out, out of, uh, from a reasoning perspective when we post-train.
- 8:54
So part of solving this for CWM, because we have a small model, this is an opportunity to really scale up how we do post-training and in particular to improve the throughput of the system.
- 9:06
And we're doing an asynchronous RL-based setup. We have samplers. We have an environment where we can execute in the terminal and get output. We have a bunch of trajectories, reasoning trajectories we output.
- 9:16
We have a trainer where we compute gradients and score trajectories. We have a source of truth for the model, and then that loop repeats.
- 9:26
So what's the challenge here? We have this loop, right? We have samplers predicting trajectories. We have scoring trajectories we're executing in the environment. As we're doing this, we're gonna update a model eventually.
- 9:35
We have a produce-consume pipeline problem, and so samplers are producing lots of trajectories that are consumed by those trainers. We need to synchronize weights. And so we solve this in CWM with a very, very asynchronous model.
- 9:50
So of course, we have a trainer that's sending a model checkpoint to a sampler very, very eagerly. We have trajectories which are being sampled and then sent back to trainers very eagerly.
- 10:03
But in particular, we have queues. So we actually will have many models queued up to be input into a sampling system. We'll have many trajectories queued up to be scored and then added vis-a-vis gradients to the trained model.
- 10:19
And so this setup stays relatively on policy, even though it's highly asynchronous, and we're not really waiting for much with this setup. We're able to achieve very, very strong throughput, uh, because of the asynchronicity.
- 10:34
So one interesting feature of this, which is increasingly common, is that we're actually updating models mid-trajectory. So I have a model which we're sampling from. It's interacting with the environment.
- 10:46
It's generating data. It's executing Bash commands. It's executing code. It's getting output. And I might actually update that model while it's interacting with the environment. So mid-trajectory, I could totally swap out the model with a new checkpoint, and the trajectory will change a little bit.
- 11:05
Uh, theoretically, that trajectory is a bit off policy, but the guarantees we have with this system are quite strong still, and that's because of the throughputs and because of the amount of data we see.
- 11:17
We're able to make a lot of guarantees around and take a lot of risk with updating the model on the fly. And this gives us really a system with very, very few bottlenecks overall.
- 11:28
Because we're queuing models, we're queuing trajectories, we don't have to wait until anything is done.
- 11:34
Okay. So overall, we post-train on still a relatively small number of steps at pretty large scale, and we process about two hundred and some billion tokens, and this scale works really well.
- 11:49
It produces a strong model, a strong open model. It's a pretty small model. It punches above its weight. It's very nice. It's pretty versatile. It uses tools in Bash very well.
- 12:01
But what can you, what can you actually do with, uh, with this model, right? What can we do with a model that understands program execution traces, that maybe has a good understanding of how-
- 12:10
How a program will run and predicting future state of a program.
- 12:16
CWM traces code really well, right? We know that. We've showed it execution traces. And I can actually give it a function, and then it can go and trace line by line that function with very, very high accuracy.
- 12:28
It can show me the values of local variables at certain points, again, with a lot of precision.
- 12:35
And this gives us some pretty interesting capabilities. I can think about a neural debugger on top of a model.
- 12:44
Traditionally, right, I have a piece of code. I don't know what I wanna write. I put some question marks. Historically, I might prompt a model with natural language. I wanna set the valuable, uh, the variable left and right to be something in particular.
- 12:58
I don't know what it is. Uh, now I need to specify very fully the ambiguity that I'm experiencing with how to complete my program. With CWM, I can express those things very naturally in line with code, and I can actually express the shape of the program I want with code, and the model will fill in the rest.
- 13:17
And the model fills in the rest by understanding that the user wrote a for loop here. The user wrote a condition here. The user left a variable unassigned. Well, if I were to go execute that, I could simulate the execution of that loop and understand better what it is the user is really after.
- 13:35
And so a neural debugger is something that helps you compose with code side by side. It's not just generating code. And it allows you to, again, express the semantics of code very, very loosely but also very, very precisely.
- 13:49
So if I have a piece of code where I, I want a certain structure, I can ensure that the model understands that structure and, and can implicitly trace the execution.
- 14:01
This will make theoreticians bristle, but I can also think about some really ambitious things in computer science. The halting problem we know is this very fundamental problem where we don't know if a c- if a program is going to, to halt, to stop executing, to terminate.
- 14:18
And in particular, this is tough because in order to know if a program halts, we would have to simulate the entire execution of the program, which, if it didn't halt, would take forever.
- 14:28
So the halting problem is in some sense a difficult problem to simulate or decide. And so the question we can ask with CWM is, "Can I approximate some of these things?
- 14:39
Can I concretely reason about program execution dynamics in this sense?" So can I say, "Here's a program. Does it halt?" Maybe the model, by simulating execution, can understand really, really high-level patterns.
- 14:56
In the same way, the model can understand high-level patterns in broader systems, right? I could use this to debug a huge distributed system where executing code is very, very expensive or even an expensive function on a single machine, right?
- 15:11
But the ability to have an implicit world model internally where I'm simulating what's happening with a piece of code or a broader system gives me the ability to reason about it without executing otherwise expensive things.
- 15:25
So we can make some progress with the halting problem by building a model that simulates it, that simulates execution, and from there we can
- 15:35
simulate and approximate what it means to solve otherwise impossible problems in computer science. So this is pretty interesting.
- 15:43
With that, I wanna encourage everyone to go build on CWM.
- 15:50
Uh, this talk does halt. This talk does terminate. [audience laughs]
- 15:54
Um, and the model's available on Hugging Face. We have some code on GitHub which will help you get started with inference in a fashion where you can twiddle bits a bit more.
- 16:03
We also have a technical report, again, where we really try to be as open as possible with all of these details around training. This post-training setup I mentioned is explained in even more excruciating detail as well as some of the data that we use for execution training and some of what we imagine a model with these capabilities
- 16:19
could be used for. Thanks for your time. Have fun. [audience applauds] [upbeat music]