World Models Need Causality, Not Pretty Pixels — Christopher Manning, Moonlake AI

Christopher Manning opens with a goal for Moonlake AI: simulation infrastructure for practical physical AI. Embodied intelligence means an intelligence that can operate in an environment, rather than only describe it. But the route to that goal begins with a deliberately slow historical build. Language, control and robots have been part of AI’s story from the beginning.

The 1956 Dartmouth summer research project supplies the familiar starting point: John McCarthy coined the term artificial intelligence and gathered researchers including Claude Shannon. Earlier cybernetics work had already connected communication, control and feedback in living things and computers. That tradition matters here because acting intelligently requires a loop: observe an environment, act on it, and use the resulting change to decide what comes next.

Natural language processing also predates the name AI. Manning revisits a 1954 public demonstration of Russian-to-English machine translation, accompanied by predictions that computers would replace most human translators. The demonstration establishes an early ambition that would take decades to mature: language processing as useful machine intelligence, rather than a peripheral application.

McCarthy founded the Stanford Artificial Intelligence Lab starting in 1963. Although his own work centered on mathematical logic, he supported a wider effort to build embodied intelligence. The Stanford Cart and, more directly relevant to this talk, Shakey brought perception and action together. Shakey could perceive its environment, move around and move boxes. Understanding the world was already tied to doing something in it.

A playful Stanford-versus-Berkeley detour places this history in its institutional setting. Manning emphasizes Stanford’s early AI work and network connections, while acknowledging Berkeley’s important systems work and its later growth in AI. He then returns to his own field: natural language processing, where language models had long been a central technology even when much of the rest of AI paid them little attention.

The lineage runs from character-level models of language to Shannon’s word and character n-gram models in the late 1940s, then to probabilistic text models developed at IBM in the 1970s. These models gave speech and NLP systems a way to judge which sequences of words were plausible. Their value was practical: they helped power speech recognition, spelling correction and machine translation, including Google’s translation work around 2007.

Useful did not yet mean general. In Manning’s recollection, language models were components for particular tasks; researchers still expected broader intelligence to require separate memories, knowledge representations, planning systems and reasoning systems. That expectation makes the later rise of language models surprising even to someone who spent his career building them.

Manning’s language-model history separates three ingredients that are easy to collapse into one story: data, compute and model flexibility. An early neural language model around 2000 used a 32 million-token corpus and a 31,000-word vocabulary. Compute constrained what it could do. By contrast, he cites Google’s 2007 language model built on two trillion tokens: substantial data scale existed well before today’s neural models. 14:58

The missing ingredient in that large earlier model was flexibility. More text could not supply the expressive power of neural architectures that had not yet arrived. Transformer-based language models appeared in 2018, but early versions again used comparatively small datasets. In this account, the ingredients came together with GPT-3 in 2020 and subsequent models: enough data and compute, joined to an architecture capable of using them.

The result was a surprising victory for NLP. Language models moved from a relatively marginal part of AI to the technology many people now mean when they say AI. Manning’s surprise is personal: after 30 years in the field, he still finds it remarkable that a model can work through difficult mathematics using test-time thinking and reach answers he could not produce himself. That success sets a demanding starting point for the next question: what does intelligence still need to act in the physical world?

Text-based descriptions have supported far more intelligence than many researchers expected. Physical intelligence adds another demand: an agent must operate in the world around it. One route is to learn directly from hardware experiments, recorded robot behavior or real-world video. Manning points to roughly 10,000 hours of human teleoperation as an example of the data collection burden. Robots move at physical speed, and people must spend time guiding them.

Shakey supplies an older alternative: keep an internal representation of the world and use it to consider actions before executing them. Its representation recorded locations and object attributes in a logical grid world. The modern version keeps the same purpose while seeking richer environments. An action-conditioned world model starts from a representation of the current state and predicts the state that will result from a chosen action.

The important distinction is between an observation and the state behind it. A picture shows what a camera can see. A semantic state represents things the agent needs to reason about: objects, their attributes and how actions affect them. Planning needs predictions about those changes, so the quality of generated images alone is an insufficient test.

A fluid Genie 3 generative-video example by Riley Goodside makes the distinction visible. Manning credits its visual appeal, then criticizes the demonstrated approach for relying on simulated observations without the semantics needed for dependable planning. This is his assessment of the demonstrated approach, rather than a measured comparison of planning performance. Moonlake’s proposed advantage is causal structure: a simulator should explain how acting on something produces a change that an agent can anticipate. 23:56

Moonlake’s starting input is an image or a short video: a partial observation of an underlying world. The desired output is a model in which an agent can act and inspect the consequences. A room reconstruction provides a concrete test. Manning describes a Marble reconstruction that supports walking around and seeing the scene from different angles. For that task, it works well. But the room also contains a kettle, a box and a cup, and the demonstrated reconstruction does not let the user manipulate them.

The first change is to separate the scene into background and foreground objects. The background can retain a representation suited to viewing the environment. Foreground objects become separate things that can move. The observable difference is simple: the tea box stops being part of the room’s appearance and becomes an object the agent can act on.

Opening the box exposes a harder problem. The original photo shows it closed, so its contents cannot be recovered from visible pixels alone. Moonlake adds web retrieval in a retrieval-augmented fashion: find images of the product open, read its description and obtain dimensions. That information supports a reconstruction with tea bags inside. It describes the product’s expected contents; it cannot establish which contents are actually present in this particular closed box. 29:04

The next change is equally important: the contents must become objects too. Tea bags that merely appear inside the box do not yet support the task. They need their own size and manipulable representation so an agent can lift one and take it out. Each step adds a capability required by the intended action, rather than indiscriminately adding detail to the whole room.

What changes between viewing the tea box and using it? The diagram follows the added representations. A movable box needs a separate identity; an openable box needs hidden structure; removing a tea bag needs the contents to have their own identities and behavior. Visual reconstruction supplies the scene, while these additions supply the actions.

The controllable parts of Moonlake’s worlds have code underneath them. This uses the ability of language models to work with symbolic material—including mathematics and code—as well as natural language. Generated code supplies an editable representation of objects and behavior. Neural generation and symbolic control meet in what Manning calls a neurosymbolic world model.

The construction process uses an iterative coding loop. Code renders objects, diffusion-based generation supplies textures, and the rendered result is assessed against physical reality. Differences guide revisions to the code, producing a new render for the next comparison. The practical advantage of the symbolic layer is that people and other applications can interface with it, control it, edit it and maintain it. 31:40

How does a reconstruction improve after its first attempt? The loop below makes the feedback path explicit: reality provides a comparison target, the comparison identifies deviations, and code revisions change the next simulated result. This is the mechanism Moonlake proposes for improving its generated worlds, rather than treating the first reconstruction as final.

Manning extends the idea that software will eat the world to physical processes. Software already manages records, suppliers and other virtual representations of activity. His hope is that verifiable simulations powered by code will let it reach further into the physical activity itself. The link is prediction: a useful simulation gives an agent a place to test what an action will do before it acts outside the computer.

The tea example continues beyond opening the box. Making tea calls for removing a bag from its foil container, putting it in a cup, boiling water and pouring it. These actions demand additional structure and behavior. A reconstruction adequate for moving a closed box may be inadequate for handling its packaging or liquid. Simulation quality therefore depends on the task that must transfer to reality.

A simulator need not reproduce every detail of the world. Human world models also leave most things unmodeled while representing what matters to the current purpose. The engineering choice is where to spend fidelity: include the objects, properties and dynamics that affect the intended action, and retain control over which parts of the world receive that detail.

The closing application is an industrial process, later identified in Q&A as a conveyor belt system. A short video becomes a 3D simulation with objects and movement represented in sufficient detail to generate training data. A robotic system can then explore the simulated environment and learn how to operate in it. This is the proposed replacement for collecting 10,000 hours of real-world teleoperation.

Manning describes obtaining 10,000 hours of simulation “for free.” The useful economic claim is that repeated simulated experience can avoid the corresponding human teleoperation effort. The presentation does not quantify simulation compute costs or establish measured real-world transfer performance; cheap, effective training remains the intended payoff of building a sufficiently accurate model.

The first questions extend the approach beyond the industrial example. Several applications and representation choices share the same need for an agent to practice actions in a modeled environment:

• Games: Millions of players provide abundant behavior data, and their goals are often fairly clear. A reinforcement learning loop can use those goals to learn effective actions. Moonlake has explored gaming, although Manning says its recent emphasis is physical infrastructure.
• Symbolic interfaces: Code brings back some of the function of older knowledge representations without requiring a return to old-style ontologies. A purely neural latent representation may be powerful, but people and other applications have difficulty connecting to it. Moonlake’s bet is that symbolic representations offer a practical interface for physical AI.
• Rare situations: Humanoid robotics, space robotics and other new forms of automation need experience beyond routine cases. Collecting that experience physically is costly, especially in the long tail of unusual scenarios. Good simulation provides a way to explore those cases.

The symbolic choice is a practical bet about integration, rather than a claim that neural representations cannot work. A code-based world gives neural systems something they can reason over and plan with, while giving humans an accessible way to change the model. The older ambition of knowledge representation returns in executable form.

Does the simulation learn all physical behavior from video? Manning answers that it uses physics engines and knowledge of physics to generate and control movement. Water is his example: starting from an image does not mean the system must infer all fluid behavior from that image. A physics model supplies predictions about how it will behave. The neurosymbolic approach therefore combines generated representations with established physical modeling.

The next question presses on the simulation-to-reality gap: mechanical details and tolerances can make real behavior diverge from a simulation. Moonlake’s proposed response extends the earlier coding loop from appearance to behavior. Compare simulated behavior against real-world video, then use neural optimization to improve the simulation. The aim is to shrink discrepancies automatically rather than depend entirely on people hand-writing and adjusting a physics simulation. 45:52

Physical dynamics are only one layer of a working environment. An audience question asks about modeling people, processes and the decision makers who control objects. Manning says Moonlake has not really been addressing that sociotechnical layer. He sees potential in language models as human-behavior simulators because they absorb large amounts of human behavior data, but presents that as a neighboring area of development, rather than a capability established by the physical-world examples.

The final question asks whether exploration in latent space can reveal connections humans have not already identified. A sufficiently good simulation can support surprising discoveries, Manning replies, but correctness inside the simulated world does not ensure that every relevant fact about reality has been captured. Reality can also contain relationships absent from the model. The value of simulation is wider exploration with a useful approximation; its discoveries still depend on what the approximation represents. 49:16