# Robots Will Be Trained on Video Games - Pim de Witte, General Intuition

Pim de Witte, General Intuition

[Talk page](https://ai.engineer/talks/MpkARD5pq-M-robots-will-be-trained-on-video-games) · [Video](https://www.youtube.com/watch?v=MpkARD5pq-M)

## Why General Intuition Trains Robot Operators on Video Games

Medal records what players see and which controls they use. Pim de Witte explains how those paired records help General Intuition train a model that can play a game or steer a robot, issuing several inputs at once as new frames arrive.

### What Olaf can see on the way down the mountain

[0:12](https://www.youtube.com/watch?v=MpkARD5pq-M&t=12s)

Pim de Witte asks the audience to imagine training a model on information from across the internet and using it to solve scientific problems whose answers never appeared there. General Intuition is making a comparable bet on a different body of human experience. By learning which controls players use in response to what they see on screen, a model may learn to operate a physical machine from its camera feed.

To see what kind of intelligence that would take, de Witte asks the audience to picture Olaf, the snowman in *Frozen*, rolling down a mountain. A little stone pops up. Olaf spreads his pieces apart, clears the obstacle and comes back together. The important detail is when he gets to see the stone. He is already moving. He can take in new information while carrying out an action, then change what he does in time for it to matter.

An autoregressive language model can produce a sequence describing the whole maneuver. For an operator on the mountainside, the problem is deciding what to do with the next observation while the maneuver is underway. That timing requirement is de Witte’s reason for building an architecture specifically for action.

### Medal records the view and the input that went with it

[1:45](https://www.youtube.com/watch?v=MpkARD5pq-M&t=105s)

The starting point was Medal, which de Witte describes as the world’s largest video game recorder. People use it to record themselves playing tens of thousands of games. From a spectator’s perspective, the product is a clip. From the perspective of someone training an operator, the valuable part is the association Medal can preserve between the frames on screen and the inputs the player supplied. A human has already made a decision about what to do in that situation, and the recorder has access to that side of the interaction.

De Witte sees a route from those records to robotics in the equipment people already use. The same person who plays a game can look at a camera feed and teleoperate a robot, in many cases with the same controller. He says robots below a certain number of degrees of freedom routinely ship with game controllers and that much of the supply chain already works this way. He gives no numerical cutoff. The overlap is in the actions available to the operator, even when the machines receiving those actions move quite differently.

General Intuition labels what each action means and gives a vision-based model a set of available controls. De Witte’s API example combines three pieces of information. A developer sends frames, supplies the instruction to wave back when somebody waves, and makes an action labeled `greet` available. The instruction tells the model which event to watch for. The frames tell it when the event occurs. Its output is the input bound to `greet` on that device. For the example to work, the device has to expose a control that performs the greeting.

![A camera view shows a kneeling person waving. Arrows lead through the Action API to a second view with a raised robot limb and the caption “Wave back!”](https://ai.engineer/api/talk-essay-media/16f689fb7fe8d1010b5c88b2eb1c9f08f19e895497c8091455be057631042c14)

Predicting a wave-back action

### A screen or camera keeps supplying the next observation

[3:26](https://www.youtube.com/watch?v=MpkARD5pq-M&t=206s)

De Witte uses coding to explain why a different interface makes sense here. A language model trained to predict the next text token can write software in turns. The developer asks for a change, the model works, and the developer can inspect the result before asking for something else. Much useful work fits that rhythm. A machine moving through space can be somewhere else by the time an operator finishes a turn.

General Intuition’s model continuously receives frames, either individually or in batches, from a screen or a robot camera. It predicts actions in the terms a game player would use, including keyboard and mouse inputs or game-controller commands. Several controls can be active together. A text prompt can specify the task, but the stream of observations keeps arriving as the model issues inputs and the scene changes.

![Two rows compare how language and action models train, what they predict, and how they are used. One example highlights the word “mat.” The other highlights the W and D keys and two mouse offsets.](https://ai.engineer/api/talk-essay-media/736ae85dc912fe9f9769af89a6476a70fd8dc09a69f2a42b17a0e5e2374556da)

Next-word and next-action prediction

De Witte compares the strategy to his reading of Anthropic’s focus on coding. Getting very good at code generation would give people a reason to express more problems in code. His team wants developers to bring more problems into pixel space. A screen already provides a visual representation, a camera provides one for a robot, and de Witte argues that other sensor information can often be represented visually too. A developer who can present the relevant observations that way and expose a set of controls has the two sides of the interface the model learned in games.

### The controller already handles the motor math

[5:26](https://www.youtube.com/watch?v=MpkARD5pq-M&t=326s)

A human teleoperator can steer a machine without knowing what torque to request from a motor. The manufacturer has already provided a device and a control system that turn an input the operator understands into the movement the machine needs. De Witte contrasts this with robotics efforts that train a model end to end to predict exact poses or motor values. His team places the learned policy at the point where the human would pick up the controller.

![Dashed lines connect one game controller to four drawings: a video game on a monitor, a four-legged robot, a quadcopter, and a wheeled robot.](https://ai.engineer/api/talk-essay-media/dc7fcbea92a840ead51b0bdebc37f7767d019a47657f06c5b19706bb77961628)

Shared controls for games and robot teleoperation

That choice determines what can transfer from Medal. A game clip provides evidence about choosing controls from visual observations. It would be a poor source for the exact motor values of a particular physical robot. Working through the controller lets the team train the decision for which it has demonstrations and leave machine-specific execution to the system built for that machine.

### A billion uploads a year, measured again in action tokens

[6:04](https://www.youtube.com/watch?v=MpkARD5pq-M&t=364s)

De Witte puts Medal’s upload volume at about a billion videos per year and says it was on track for three billion that year. He reports more than 1.5 trillion action tokens per month. The upload count measures how many clips reach Medal. The token count measures the volume of player input available alongside the frames. A player can hold one control while adjusting another, so a single maneuver can involve several inputs at once.

![A blue line shows monthly clip uploads rising sharply in 2025 and reaching 195.2M in August 2026. Vertical markers show cumulative uploads reaching 2B that month.](https://ai.engineer/api/talk-essay-media/5e629d2ff275bae23e8405c261d48a4a1027b7207c3318005f90325ce4662301)

Medal’s monthly clip uploads and cumulative milestones

For a sense of scale, de Witte estimates that Medal collects in half a day as much data as frontier labs use to train robotics models. He supplies neither a named comparison dataset nor a common counting method, so the half-day figure is his estimate of relative volume. The bet behind it is that scaling data and compute on this particular prediction problem will produce an operator useful across computer interfaces and physical machines.

### When to generate an environment and when to use the original view

[7:34](https://www.youtube.com/watch?v=MpkARD5pq-M&t=454s)

General Intuition also builds world models, including Mirai. Their role here is to generate additional environments for the policy to learn in. The distinction is easiest to see from the outputs. The world model can supply a setting and the pixels the agent observes there. The policy takes an observation and chooses what to do. Producing another environment gives the policy another place to exercise that ability.

Making a simulation a prerequisite for learning a new activity would introduce two costs. First comes the compute needed to simulate the setting, which rises with the realism required. Then comes the work of establishing that the simulation preserves what matters about the original. De Witte traces a possible detour in which data passes through an LLM, someone creates a digital twin, the model is deployed into it, and verifiers check whether the two situations are equivalent. Only then does the robot get to learn the activity.

Consider a person trying to communicate something to a robot partner. De Witte wants the partner to be able to observe the very thing the person is looking at. If the team already has that view and the person’s actions, it has a demonstration in the operator’s own input modality. It can use that evidence directly and reserve generated worlds for the cases where additional places to practice are valuable.

### Why a drone operator benefits from a gamer’s actions per minute

[8:54](https://www.youtube.com/watch?v=MpkARD5pq-M&t=534s)

De Witte calls attention to an example in which the model predicts multiple actions at the same time. For a drone operator making a succession of small adjustments, both the frequency and the overlap matter. One correction can already be underway when the camera view calls for another. Learning which controls people combine is part of learning the maneuver.

This is where he favors game recordings over starting with teleoperation footage. In his account, gamers supply more actions per minute, or APM, giving the policy more examples of fast and simultaneous inputs. The semantic labels also make a prediction easier to inspect. A developer can identify the key the policy selected and the behavior that key represents in the current environment. De Witte anticipates a speed advantage over LLMs on these workloads, though he gives no measured APM or latency comparison.

### One policy across games and more than ten robot types

[9:54](https://www.youtube.com/watch?v=MpkARD5pq-M&t=594s)

De Witte reports that a single model can play video games at what he calls the highest level and control more than ten types of robots in real time. Across those embodiments it uses the same primitives: receive pixels, emit actions. He withholds the release videos from the livestream, saying the team wants to save them for the launch. The claim establishes the breadth the team reports, with individual games, robot tasks and performance scores left unspecified.

At the time of the recording, some customers already had private access and the New York team had begun giving live demonstrations. De Witte invites attendees to join a waitlist through a QR code. He says signup order will help determine access as the team rolls the models out over the following months.

### An orchestrator sets the task and the policy explores the interface

[11:34](https://www.youtube.com/watch?v=MpkARD5pq-M&t=694s)

One proposed application starts with an environment generated by an LLM. Someone still has to try it. An action model could explore the result through the controls available to a person. De Witte extends the idea to UX work. If a designer wants to know whether an interface is intuitive, watching an agent attempt to use it from the screen offers a concrete interaction to inspect. He presents this as a use to investigate, without reporting a usability study.

For longer jobs, the expected integration retains an orchestrator. De Witte names Gemini and Astra as possibilities for long-term planning. A planner can supply a text instruction while the action policy handles the succession of visually informed inputs needed to carry it out. His prospective science-lab example depends on both capabilities. Someone must organize what the robots should do, and the robots must reliably execute those instructions in space and time. He also points to a recursive opportunity for robotics businesses: making more robots is itself one of the problems that better robot operation could help solve.

### A player keeps the mouse while the model takes the keyboard

[13:35](https://www.youtube.com/watch?v=MpkARD5pq-M&t=815s)

In an accessibility application the team is exploring, a player who can use one arm operates the mouse. The model supplies the keyboard inputs the game also requires. The player’s own actions are part of what the model observes alongside the screen. That gives the policy context for choosing the complementary keys while leaving the mouse under the player’s control.

This is a different allocation of work from handing an entire task to an agent. The person continues making decisions throughout play, and the model’s share of the controls is limited to what the person needs help operating. De Witte expects another early use to be in simulation, where someone generating environments with Astra or Gemini could put a policy to work exploring them instead of having to play through each one personally.

### Three teams and a first hands-on window

[14:54](https://www.youtube.com/watch?v=MpkARD5pq-M&t=894s)

Medal and General Intuition are based in New York City. De Witte describes three teams. World modeling works on generating worlds. The policy team works on the agent that chooses actions. The third team works on data generation and environment generation, the entry point he suggests for engineers interested in joining. He values experience shipping an AI product end to end and an interest in learning how the models work. He mentions his own engineering path when encouraging candidates without a research background.

The release window he gives is late November or early December. He also announces plans for three hackathons and identifies locations for two, New York City and London. The first was planned for mid-November in New York, with details to go out through the partner waitlist. The attraction for participants was hands-on access of the kind the team already had internally.

### The JEPA question turns on a rudder the camera cannot see

[17:16](https://www.youtube.com/watch?v=MpkARD5pq-M&t=1036s)

A question asks how General Intuition’s approach differs from JEPA. In de Witte’s comparison, a JEPA approach relies largely on inferring the actions behind observations. He allows that this may be the better approach in the limit and says his team borrows some of its ideas. His near-term objection is that it introduces a second scaling question. Along with learning a better policy, one has to count on getting better at recovering the actions that should supervise it.

Imagine a pilot moving the rudder at the back of a plane. From the available video frames, it can be extremely difficult to determine that the pilot used that control. Even if a viewer can see the scene change, the camera may reveal little about the specific input responsible. A synchronized control record gives the learner the pilot’s choice directly, including in sequences where attempting to reconstruct it from the image would be guesswork.

That example explains where General Intuition chose to concentrate its training effort. An archive of recorded inputs gives the team a particular advantage in predicting sequences of actions. De Witte sees less of a corresponding advantage in predicting the next pixels. The team therefore focuses its foundation models on action prediction while using a range of techniques for world modeling and welcoming environments built by others.

## Key takeaways

- Preserve the operator’s inputs alongside the frames. A control such as a rudder can matter even when the camera gives little evidence that the operator touched it.
- Expose the controls a human would use and label what each one does. That connects a text instruction such as waving back to an action the device can execute.
- For tasks that need frequent corrections, examine actions per minute and simultaneous inputs as well as the number of recordings.
- Assign long-term planning to an orchestrator and visually guided control to the action policy. For shared control, include the person’s inputs and give the policy only the controls it needs to supply.

## Slides

### Predicting a wave-back action

[2:19](https://www.youtube.com/watch?v=MpkARD5pq-M&t=139s) [A camera view shows a kneeling person waving. Arrows lead through the Action API to a second view with a raised robot limb and the caption “Wave back!”](https://assets.ai.engineer/sites/aiecode2025/sha256/16f689fb7fe8d1010b5c88b2eb1c9f08f19e895497c8091455be057631042c14.webp)

Our action models see the world, and predict actions

Someone waves
X  dance +1
D2  discord
C  down +1
G  greet +1
J  jump +1
D4  next_episode

Action API

X  dance +1
D2  discord
C  down +1
G  greet +1
J  jump +1
D4  next_episode
“Wave back!”

4

### Next-word and next-action prediction

[3:27](https://www.youtube.com/watch?v=MpkARD5pq-M&t=207s) [Two rows compare how language and action models train, what they predict, and how they are used. One example highlights the word “mat.” The other highlights the W and D keys and two mouse offsets.](https://assets.ai.engineer/sites/aiecode2025/sha256/736ae85dc912fe9f9769af89a6476a70fd8dc09a69f2a42b17a0e5e2374556da.webp)

LLMs predict the next word, our models predict the next action

PRE-TRAINED ON     LEARNS TO PREDICT     NOW USED FOR

Language models
Trillions of words from the internet
The cat sat on the mat
The next word
Writing software
Research & analysis
Agent run workflows

Same recipe,
different data

Action models
Millions of hours of action-labelled gameplay
KEYS
W       A       S       D
forward   left   back   right
MOUSE
dx −75   dy +4
10 times a second
The next action: Keyboard strokes and mouse movements
Playing video games and operating robots, in real time

5

### Shared controls for games and robot teleoperation

[5:18](https://www.youtube.com/watch?v=MpkARD5pq-M&t=318s) [Dashed lines connect one game controller to four drawings: a video game on a monitor, a four-legged robot, a quadcopter, and a wheeled robot.](https://assets.ai.engineer/sites/aiecode2025/sha256/dc7fcbea92a840ead51b0bdebc37f7767d019a47657f06c5b19706bb77961628.webp)

Humans play video games and operate robots with same primitives

Humans play video games and (tele)-operate robots using the same inputs and outputs, namely
vision & controller actions

6

### Medal’s monthly clip uploads and cumulative milestones

[6:26](https://www.youtube.com/watch?v=MpkARD5pq-M&t=386s) [A blue line shows monthly clip uploads rising sharply in 2025 and reaching 195.2M in August 2026. Vertical markers show cumulative uploads reaching 2B that month.](https://assets.ai.engineer/sites/aiecode2025/sha256/5e629d2ff275bae23e8405c261d48a4a1027b7207c3318005f90325ce4662301.webp)

Medal is currently the fastest growing company in games

Clips uploaded per month
Cumulative uploads     Monthly uploads
JUN 2022 TO AUG 2026

500M     Aug 2025
1B       Jan 2026
1.5B     May 2026
2B       Aug 2026
195.2M

200M
150M
100M
50M
0
2023     2024     2025     2026

The total amount of data frontier
robotics models are trained on today
Dataset growth

Medal is on track to 3 billion uploaded videos per
year

8

### Real-time action models for games and robots

[7:18](https://www.youtube.com/watch?v=MpkARD5pq-M&t=438s) [A list describes action models that turn pixels into actions across different bodies and environments. Beside it, a robot marked “02” has one jointed limb raised.](https://assets.ai.engineer/sites/aiecode2025/sha256/84ecb4a8aeec8191c93d787ac3bba98ee77fa44c6db0f43bc699d1c508b2ea50.webp)

A new class of foundation model emerges

Our action models:
• Capable of handling robust autonomy
• ... from pixels to action
• ... across a broad range of embodiments
• ... across a broad range of environments
• ... in real time
• ... in games and in the real world

02

MORE SOON

9

## Full transcript

[0:12](https://www.youtube.com/watch?v=MpkARD5pq-M&t=12s) Hello. Um, I want you to, uh, zoom out for a second and think about if somebody were to tell you today that we would take all the information on the internet, and then we would train a model on it, and we would use that in order to solve scientific problems that were not on the internet. I think a lot of people would tell you that that's crazy. Um, and, uh, what we're doing, um, has a lot of parallels to that story. Um, and so if you think about how

[0:42](https://www.youtube.com/watch?v=MpkARD5pq-M&t=42s) intelligence works,

[0:46](https://www.youtube.com/watch?v=MpkARD5pq-M&t=46s) imagine for a second that you're in the movie Frozen and you're Olaf, the, the snowman. Um, if, if Olaf was an LLM and it-- there, there's a scene which is one of my favorites, which is where he rolls down a mountain and there's a little stone that come-- that pops up, and Olaf spreads out, temporarily becomes a disassembled snowman, and then becomes an assembled snowman. And Olaf knows how to do that because Olaf can observe as he is acting. Olaf can,

[1:15](https://www.youtube.com/watch?v=MpkARD5pq-M&t=75s) unlike LLMs which autoregressively predict tokens, um, constantly part of that behavior is that it's constantly observing as it is acting. And so, um, this is the big difference between, you know, when you see LLMs try to solve robotics problems or when you look at native architectures like we're working on. Um, it's very, very easy to think of LLMs and the demos you see on Twitter as the thing that will solve everything. There's fundamental limitations that we see, uh, where purpose-built

[1:45](https://www.youtube.com/watch?v=MpkARD5pq-M&t=105s) models are a much better fit. Now, today I'm gonna talk a little bit about what the other side of that coin looks like. Um, and so for us, it all started with Metal. We built the largest video game recorder in the world, um, and we think of that as the internet. So people use Metal to record themselves playing tens of thousands of different video games. And if you think about it, uh, as a human, we can teleoperate a robot with the

[2:15](https://www.youtube.com/watch?v=MpkARD5pq-M&t=135s) exact same brain that we played that video game, and in fact, um, in many cases, we also can

[2:25](https://www.youtube.com/watch?v=MpkARD5pq-M&t=145s) use the same controller. So what this means is that the action space of video games, often because of game controllers, and the action space of teleoperations or the real world, in many cases is the same. Um, and so every single robot built below, um, certain degrees of freedom ships with a game controller. A majority of the supply chains already work this way. And so we thought, what if you just take all of the action data that we get through the video recorder, um, which associates the inputs and the actions with the actual frames. We label the actions, so

[2:55](https://www.youtube.com/watch?v=MpkARD5pq-M&t=175s) we unders- understand semantic meaning behind every action, and we use it to give it to a vision-based model. And so what you see here on the left is just an intuitive way of how the model works. So you give the model a set of keys to choose from, essentially a computer use model. It, it, it gives frames as the input, so you can give it a text prompt as well. And you send it through API, and you say, "Somebody waves at you, wave back." In the input space, the model understands that you have a greet action available. It will output the action

[3:26](https://www.youtube.com/watch?v=MpkARD5pq-M&t=206s) associated with that. Um, one way of intu- of intuitively understanding how this works, so, um, I've obviously simplified here because it's predicting the next token, not the next word, but bear with me here. Um, LLMs took large scale internet data, then trained it to predict the next text token, and now use it for writing software. Um, and these are all workloads that are really, really well fit for futures that are not simultaneously unfolding as you are predicting, right?

[3:55](https://www.youtube.com/watch?v=MpkARD5pq-M&t=235s) Code is turn-based. You can say something, the model will do something. You can say something again, the model will do something again. And, um, you can use it for lots of valuable work, but at the end of the day, because it's turn-based, um, it's very, very difficult for it to generalize to environments, especially that in- that include, um, space and time. And so our models, the way that they work is they're continuously receiving frames or receiving batches of essentially,

[4:26](https://www.youtube.com/watch?v=MpkARD5pq-M&t=266s) um, information from the camera of the robot, or it could be a, a screen, and it predicts everything as, as if it were just a video game. So you can think of the way that Anthropic focused on just nailing coding. So, you know, they were never frontier in just base text generation. They always focused on being the best at generating code. And then what the, the bet that they made is that more people would start bringing more problems into coding space. If you think of general intuition, our bet is that you can

[4:56](https://www.youtube.com/watch?v=MpkARD5pq-M&t=296s) s- as humans, we are generally pixel-to-action converters plus sensors, which in many cases you can convert into pixels as well, and you can output actions at the same time. And those actions are, in this case, either a keyboard and mouse or a game controller. So the output tokens of the model are, um, essentially equivalent to that of, uh, enhanced computer use model with more, more, more actions on the output space to internalize it. So you send frames, and you get actions. This works, again, because every single thing that you see here can control, um,

[5:26](https://www.youtube.com/watch?v=MpkARD5pq-M&t=326s) all of these different embodiments, right? So, um, as humans,

[5:35](https://www.youtube.com/watch?v=MpkARD5pq-M&t=335s) we do not know how to predict motor torque. We do not, we do not know what value to send to a machine. Um, there's this beautiful device that has been made for many years, uh, which actually does that work for us so we can control them. Most robotics companies today are trying to solve the problem at the end-to-end level, trying to predict exact poses or exact, um, values, uh, in the motor, uh, instead of just looking and picking up the thing that Every

[6:04](https://www.youtube.com/watch?v=MpkARD5pq-M&t=364s) robotics company has built to generalize our very l-limited, um, uh, abilities to have physics understanding and predict exact values and solve for it. And so our approach is, let's just use that. Um, and so Metal, at the moment, sees about a billion videos uploaded per year. This year on- we're on track for three billion, um, which lead to over a trillion and a

[6:34](https://www.youtube.com/watch?v=MpkARD5pq-M&t=394s) half, uh, action tokens per month. So just for comparison, to think about the bet, the total amount of data that Frontier Labs use today to train robotics models, we collect in, at the moment, half a day. Um, and so, uh, the, the, the type of bet-- and again, think of the fact that humans can use the exact same controllers and their eyes to play video games that they can't tell operate robots. It is the same thing. We just bet on scaling pixel space the same way that

[7:04](https://www.youtube.com/watch?v=MpkARD5pq-M&t=424s) Anthropic scaled coding space and betting that people will just, as we scale data compute into pixel space, that people will bring more problems into pixel space. And so what happens? Um, all of a sudden, you get an incredibly general agent that can go and solve problems on computers in the real world. Um, and in addition, similarly to how

[7:34](https://www.youtube.com/watch?v=MpkARD5pq-M&t=454s) Anthropic focuses on doing coding environments, we focus on world models, which most of you know us for, the Mirai release, uh, world models, to then generate additional environments for these models to learn in. One way of internalizing a big advantage of this approach against LLMs is if you're in a, if you're an LLM company, and you have to bet on the ability to generate environments for your models to learn, you have, you have two problems. One, you actually need to spend the compute to simulate environments, and the more realistic you try to simulate, the more expensive that gets.

[8:04](https://www.youtube.com/watch?v=MpkARD5pq-M&t=484s) Um, and so imagine you have-- you want to learn a new thing. You want to learn that new thing in the exact input space of the human, right? So you-- if, if we're learning from pixels, we're, we're learning by looking around. If I'm trying to communicate something, I want my robot partner to observe that exact same thing. And so, um, what, what I don't want is for data to have to go through an LLM, for a digital twin to have to be created, for the model to have to get deployed, for there to be a bunch of

[8:34](https://www.youtube.com/watch?v=MpkARD5pq-M&t=514s) verifier checks whether or not things are equivalent, and then for the robot to, to learn much later, right? And so one way of thinking about it is when you can learn in the native modality of the human, you can all of the sudden start solving a lot more problems a lot faster. What you'll see when these models come out is they're significantly faster than any LLM, um, uh, at doing these things. We-- Because games are so action-dense, so what you see, right, is, um, one thing that's interesting is,

[9:04](https://www.youtube.com/watch?v=MpkARD5pq-M&t=544s) uh, you saw, for instance, the model predicting multiple actions at the same time. Um, we're able to, uh, take that action density from gamers and convert them into really complex robotics workloads. So, for example, if you are operating a drone, and you need to make lots of little adjustments, you really, really, really wanna start with games. You don't want to start with, uh, tel operations because the human will just take a lot less actions. It has a lot less information. So you want a model that can predict really high APM. Um, and so

[9:34](https://www.youtube.com/watch?v=MpkARD5pq-M&t=574s) the APM of the dataset puts a lot of these problems all of a sudden in reach so that it can do a lot of these actions simultaneously. And as I mentioned, the model doesn't just understand the key it's predicting. It also understands the semantic meaning associated with that key it's predicting, which is really important for things like, um, interpretability. Uh, so, uh, we will be releasing the models this year. Um, uh, we are very happy. I think the correct word would be that we're, we're stunned by the

[10:04](https://www.youtube.com/watch?v=MpkARD5pq-M&t=604s) fact of what these models can do. Um, it is, uh... This is a live stream, so I unfortunately couldn't share the release videos without breaking the surprise. But if you're in New York, we are also in New York, and so, uh, we have started doing a few demos, uh, of the model. So if you can-- know somebody on the team, reach out, and we can start showing you some of the, uh, some of the capabilities live. Um, but we have essentially one model which can play video games at the highest level and control over ten types of robots, uh,

[10:34](https://www.youtube.com/watch?v=MpkARD5pq-M&t=634s) using the exact same primitives. So pixels in, actions out, completely real time. Um, and, uh, we'll be rolling out early access, uh, to these models over the next few months. Um, we already have some customers that are on them, but it's still, uh, private, so you're not hearing too much about it. But if you scan this QR code, you'll be able to get the models, um, because we go by who signs up on the wait list at which times. So, um, this is a great, this is a great moment to get in touch. As I

[11:04](https://www.youtube.com/watch?v=MpkARD5pq-M&t=664s) mentioned, we are in New York. I think what you'll find is similar to how, uh, Jev and a lot of these decision-making models have kind of opened people's eyes on what you can do when you have a new architectural primitive. People are ready to kind of do away with the unpr-unpredictability of an LLM and put something in that is more reliable and faster, and you just understand how it's going to react as opposed to having to work around it. If that resonates, these models are exactly what you're looking for. Um, and all of the sudden, um, you can start

[11:34](https://www.youtube.com/watch?v=MpkARD5pq-M&t=694s) exploring the environments that you generate with LLMs. All of the sudden, one of the use cases that I'm really, really excited about is, well, these models, they, um, can take actions as if a human was on the interface. And so if you're doing UX work, and you're trying to understand, is this an intuitive interface, well, this is a very interesting way of solving the problem. So all these problems that LLMs can't solve because they can't actually explore like a human, you can use these problems to start-- you can start using these models for. So,

[12:04](https://www.youtube.com/watch?v=MpkARD5pq-M&t=724s) um, the way that they're going to be released is we still expect you to use, uh, an orchestrator model. So you may use something like Gemini or Astra in order to do long-term planning and, and everything, and you'll be able to plug these into your workloads. Um, and we're actually-- The, the models are so general. It is, it is- Genuinely going to, I think, feel like the early LLM days when you have a new toy and you just wanna figure out what it does and all the things you can use it for. I th-- m-my reaction

[12:34](https://www.youtube.com/watch?v=MpkARD5pq-M&t=754s) was like that when, when I, when I first started using the checkpoints, and I think, I think you'll have the same reaction. Um, and I think a lot of you will, just like with LLMs, if you're a bit early and you start understanding these primitives, I think you'll understand, uh, that you can even build lots of businesses on top of this, right? You know, think about, for instance, um, uh, a sci- a science lab, right? If you, uh, all of a sudden, if you can reliably orchestrate robots in space and time, and they can take

[13:04](https://www.youtube.com/watch?v=MpkARD5pq-M&t=784s) text instructions and do things, all of a sudden, um, uh, it's a recursive problem, right? Robotics is interesting because one of the big problems of solving robotics is making more robots. And so there are all these businesses that, because it's so bottlenecked by intelligence, are going to emerge just from solving this one simple problem, which is pixels to actions. And I think if you're early on getting your hands on, on these, it will have a similar effect on getting your ear- your hands on early LLMs.

[13:35](https://www.youtube.com/watch?v=MpkARD5pq-M&t=815s) Um, I am most excited also, uh, one of the things that, that we're working on, which is interesting, is if you have any friends or family who, uh, are, um, in some way, um, uh, need accessibility help for using their computer, because these models can use computers like a human, all of a sudden, for instance, if you're, if you're, uh, if you only can use one arm, um, all of a sudden, the model can allow you to play video games because you can, you can use-- you can just control the mouse, and the model

[14:05](https://www.youtube.com/watch?v=MpkARD5pq-M&t=845s) is predicting the actions for your keyboard. And because the model also understands which actions you're predicting, because that's the input, right? It's, it's observing as it's acting. It will actually just not predict the things that you're doing, and it will just predict the keys. And so it has all these downstream effects of things that you can build on top that-- I'm an engineer. If you're interested in building in this space, uh, this is going to be the most interesting toy that comes out this year. I, uh, I, I'm fairly certain. Um, and I

[14:35](https://www.youtube.com/watch?v=MpkARD5pq-M&t=875s) also think a lot of the initial usage will sit in simulation. So if, if a lot of you are into generating environments with Astra or with Gemini, it's perfect. Now all of a sudden, you can have something that can explore it, and you don't have to do it yourself. Um, and, uh, it's perfect timing because of all the environment generation capabilities going, going into these models. Um, okay. So really quickly before I wrap up, um, I wanna talk about the team. Um, so we are based in New York City. Uh, the metal team and the general intuition team is here in New York City, so

[15:05](https://www.youtube.com/watch?v=MpkARD5pq-M&t=905s) if you're here, you should reach out to us and, uh, uh, we can both show you the models, uh, especially if you wanna join the team. I think we'll, we'll prioritize even higher.  But, um, uh, we really are looking for, um, people who, A, want to understand these models as much as we do. So people are-- if you're really, really good at generating novel environments, uh, typically if you have some sort of research, uh, background, but I don't have a research background, so that's okay. If you, you know, if you, um, if you come from an engineering background

[15:35](https://www.youtube.com/watch?v=MpkARD5pq-M&t=935s) and you're very, very happy to learn, this is, this is fine too. Um, and ideally, what we want is somebody who, um, uh, has experience shipping, uh, end-to-end, uh, AI, uh, products, but it doesn't have to be on the research side. So we just wanna kind of hire people that understand the entire thing. Um, let's see. Uh, we have three teams. So one is world modeling. The other is the policy, so the agent. And the other one is working on, um, essentially the, the data

[16:04](https://www.youtube.com/watch?v=MpkARD5pq-M&t=964s) generation and the environment generation. So these are the main teams. If you're an engineer, the best one to join is probably the environment generation team. Um, and then you s-sort of grow into, uh, getting access to the other ones. Um, yeah, we, uh-- I'm very, very excited for November. I really hope that, uh, you'll be on the lookout. Um, as I mentioned, it will be either late November or early December is the current planning. Um, one more thing. Uh, we are going to organize three hackathons. Um, so the first is going to be in New York City. It will be

[16:34](https://www.youtube.com/watch?v=MpkARD5pq-M&t=994s) mid-November. Um, and, uh, we will, we will send out, um, if you sign up for the partner wait list, we'll send out some emails so that everybody can get access. Uh, we hope to see everybody at the hackathons. I think it will be really the first time to get your hands on them the way that we're playing with them internally now. And, uh, there will be one in New York City and one in London. So if you're in either of the cities or want to travel to either of those cities, uh, those will be a good, uh, times to meet the team. Um, yeah, let's see. Um, I have time

[17:04](https://www.youtube.com/watch?v=MpkARD5pq-M&t=1024s) for one or two questions, and then, uh, I think we can, we can wrap. Uh, let's see.

[17:16](https://www.youtube.com/watch?v=MpkARD5pq-M&t=1036s) How is it different from GEPA? Uh, yeah. How is it different from GEPA? So GEPA is, uh, doesn't train alongside the action labels, so they bet on inferred action labels for the most part, which I think-- like, one of the ways of thinking about it is GEPA, I think, in the limit is the more correct approach, but it's just a lot less practical in the short term because you need to bet on a separate scaling law on the action inference itself. Um, and, uh, we really like GEPA as a concept. We think it's a, a, a, you know-- We, we use some of the things in the models. Um, and, uh,

[17:47](https://www.youtube.com/watch?v=MpkARD5pq-M&t=1067s) but at the end of the day, it's, it's, it's an approach, not a product. Um, and I think we'll, you know, borrow concepts both ways. Um, the one way of thinking about it is, um, because we have so much action data, let's say you're flying a plane and you're moving the rudder in the back. There's no way of being able to tell that from the video frames. It's, it's very, very difficult. So our mode on having the action data really transfers better to action sequences than it

[18:17](https://www.youtube.com/watch?v=MpkARD5pq-M&t=1097s) does to world models and to ge- to generating the pixels. And that's why we just went all in on training these general foundation models for action predictions, and we u- we use all sorts of techniques to do world models. You know, we-- our own, we, we really like other companies building them as well. Um, so great. One more, and then I gotta run, I think. Great. Um,

[18:41](https://www.youtube.com/watch?v=MpkARD5pq-M&t=1121s) good. All right. Thank you, everyone.  Have a good one.
