You Can't Tell What's Real Anymore. Here's What Comes Next - Cristóbal Valenzuela, Runway

In The Orange, Runway’s first AI film, an orange falls from a tree and walks through different parts of the world. How much of that story could a model draw in 2019? Its frames measured 128 × 128 pixels. The team enlarged them for display. Cristóbal Valenzuela looks back on the result with the affection he feels for early cinema. The lab was still trying to convince itself that making the models bigger would get it somewhere.

The later footage makes the scale of that bet easier to appreciate. Runway generated the characters and the locations, and the scenes look convincing even to Valenzuela after nearly a decade working on the technology. His prediction is that in a couple of years this footage, too, will look like The Orange. To see what could change besides image quality, it helps to ask how a model gets a still image to move.

Cinema supplies the playback trick. Show a sequence of still frames fast enough and the eye sees movement. The video model has a different job. Start with Runway’s generated image of two people. Given the frames so far, the model predicts what the next one should look like. That frame becomes part of the history for the following prediction. Runway’s bet on larger networks and more compute was that this simple task would reward learning how a scene changes over time.

One test is whether a viewer can tell that the people and their movements were generated. Valenzuela says Runway tested viewers and takes the results to mean its videos can pass that test. He gives no accuracy figure or viewing conditions, so how readily people can distinguish other clips remains open.

A stranger scene poses a more revealing question. What has to stay consistent when a camel stands on a ball? The position of the feet only makes sense relative to the curved surface supporting them. Water in a container poses a similar problem because its shape depends on what holds it. A single frame can depict either relationship. Each subsequent frame has to respect it as things move.

This is the sense in which next-frame prediction can teach a model physics. Gravity constrains where an object can move. Occlusion determines which parts are visible, and object permanence means accounting for an object even while something else hides it. Runway evaluates geometry and object behavior alongside the creative judgment of whether someone likes the clip. A beautiful frame can still be followed by a physically implausible one.

One generated clip looks like someone playing a video game. The oddity is that the game itself doesn’t exist. The model produces the appearance of a player moving through a 3D environment without anyone specifying physical laws for its objects. It can depict what gameplay looks like. But the clip took seconds or minutes to make, and by playback time the player’s path was already settled.

Real-time generation changes when a decision can enter the picture. Let the model continue and it predicts from the preceding frames. Touch something in the image while it is generating and the action can affect its next prediction. The altered frame then supplies context for what follows. That is how the same scene can have different futures depending on what the viewer does.

The possibility interests Runway beyond filmmaking. Its New York office, in Valenzuela’s description, is half Hollywood studio and half robots walking around. The company was starting a robotics lab to explore whether models that predict the effects of movement could help embodied systems reason about their surroundings and move through them.

A car image gives that action a familiar meaning. Runway calls the system a world interface model. It trains a video model to interpret actions and clicks in an interface, so a user can operate the car lights or turn the car around. The starting image can be a photograph or a generated picture.

Where does the view under the hood come from? Opening it would ordinarily reveal a part of a 3D asset that was hidden from view. Here there is no underlying car model in Unity, Unreal or Blender. The video model uses the car’s appearance and the action to generate a plausible view inside. The original picture alone could never tell you exactly what was under that hood.

The room example asks for the same kind of inference in a different setting. Start with an image of a room and move an object within it. The model has to depict the object changing position in a way that fits the surrounding space. It can draw on spatial relationships learned from video even if nobody has built a 3D version of that particular room.

Continuum begins with an operating-system question. If larger models and more data let a simulator predict what belongs anywhere on a screen, what part of a computer’s display could come from a video model? Runway’s early research prototype tries this within an ordinary task: reading a website to find out when San Francisco gets fog and how it forms.

How does the paragraph about fog become something the reader can explore? Selecting a passage makes it the source for what Runway calls a portal. Continuum uses the selected content to generate a real-time video about it. The passage sets the subject, while the reader can decide where to go next within the visualization.

Two other portals show what remains under the reader’s control after that initial selection.

• Return to batteries. A history on the right keeps earlier portals available. Reopening the battery visualization restores something the reader can click, move around in and question. The video changes in response to those interactions.
• Resize a lesson on photosynthesis. Changing the window size changes the experience the model generates. The reader can also click through parts of a cell to explore photosynthesis. Here even the dimensions of the window can influence what the lesson presents.

The prompt behind the interaction can change, too. That leaves Valenzuela unsure what to call the result. It plays like a video, explains like a visualization and responds to exploration like a game. A saved portal lets the reader pick up that exploration again.

Runway also connected Continuum to a coding agent for an experiment it calls Visual Thinking. Imagine asking the agent to create a 3D model of your bike. The agent can start a Continuum session and use it to explore the problem visually while working on the request. The visualization is an intermediate step toward the bike model, one the user can inspect before the work is finished.

What does the person gain by seeing the same visualization as the agent? Both can interact with it. The person can refer to what is already in view when guiding the agent, without first having to describe the whole imagined bike in text. That shared point of reference is the proposed advantage of Visual Thinking.

The fog portal started with a passage someone had already chosen. The last example starts one step earlier, with a search for Mars. Continuum infers what the user might want to explore and suggests an interface for that purpose. In this case it generates a Mars outpost. The user can move from place to place in it as in a 3D environment. The outpost is intended to reflect data about robots on Mars or robots projected for 2030. The demonstration shows navigation and leaves the accuracy of those details open.

Valenzuela expects Continuum itself may come to look like The Orange: an early experiment whose limitations become obvious once you have seen what follows it. At the time of the talk, Runway planned to release most of what it had shown over the following eight to twelve months. The distance between the two experiments is already tangible. With The Orange, the audience could watch where the fruit went. At the Mars outpost, the viewer gets to choose where to go.