# Robotics: why now?

Quan Vuong · Jost Tobias Springenberg

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/cGLa8DsOYdk-robotics-why-now) · [Video](https://www.youtube.com/watch?v=cGLa8DsOYdk)

## Robotics: why now?

Vision-language-action models connect visual understanding to robot control, but dexterity and generalization depend just as much on how demonstrations are collected, annotated, and diversified.

Familiarity with transformers and model training is helpful; no robotics background is required.

### What changes when the environment stops being predictable?

[0:16](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=16s)

What would it take for one model to control any robot and perform any task? That is the research goal Quan Vuong and Jost Tobias Springenberg introduce at Physical Intelligence. It is not a capability they consider ready: multiple scientific breakthroughs remain necessary. Publishing research, releasing models, and discussing the work openly are part of how they intend to get there.

Industrial robotics shows how much robots can accomplish when the environment is constrained. A factory provides structured surroundings and repeatable motions. Move into an everyday environment, however, and even an apparently simple task can become difficult. The opening comparison pairs an industrial robot with an older clip of a humanoid struggling beside a shopping cart.

![Slide showing an industrial robot arm in a factory on the left and a humanoid robot beside a shopping cart on the right.](https://ai.engineer/api/talk-essay-media/49253b842ab94e7b640a0ac665d146a0d4ec521ec19acb294883169b14d13c42)

“Robotics Before”: an industrial robot arm and a humanoid beside a shopping cart.

The newer demonstrations change both sides of that comparison. A dancing humanoid displays complex physical motion. A laundry robot confronts a different difficulty: clothes have just come out of a running dryer, so nobody can precisely arrange each shirt before the task begins. The robot removes the shirts and puts them into a basket; Quan explains that the full video continues with transport to a table and folding. The challenge includes manipulating objects whose shapes and starting positions are difficult to control.

General progress in AI helps explain the change, but the specific bridge to physical behavior is the **vision-language-action model**, or VLA. It connects the ability to interpret a scene and an instruction with the ability to produce robot actions.

### From answering questions to controlling a robot

[2:24](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=144s)

A **vision-language model**, or VLM, takes text and images, embeds them, and passes those representations to a transformer that generates a textual answer autoregressively. A VLA adapts that arrangement for robotics. It also receives information about the robot itself, such as joint positions, and produces actions that directly control the robot.

| Model | Inputs | Output |
|---|---|---|
| VLM | Text and images | Textual answer |
| VLA | Text, images, and robot state | Robot-control actions |

The resemblance makes pretrained multimodal models useful starting points. It does not make training and deploying a robot policy equivalent to serving a chatbot.

For a downstream VLM application, the familiar pipeline starts with web data, perhaps supplemented by task-specific examples. An existing model can be fine-tuned on a cloud cluster and deployed using established inference libraries. Robotics introduces gaps at each of those stages.

- **Data:** There is no established equivalent of the web as a source of training examples for frontier robot dexterity. Springenberg treats finding that source as a major open research question.
- **Architecture:** A pretrained VLM backbone needs adaptation to generate actions at the high frequencies required for robot control.
- **Deployment:** At the time of the talk, the team sees no standard solution for deploying large robot policies across multiple locations, on premises, and on devices.

The presentation develops the data and model problems; it leaves detailed deployment engineering aside.

### A data engine built around human demonstrations

[5:09](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=309s)

Physical Intelligence builds its collection pipeline from the ground up, aiming for useful capabilities immediately while leaving room to scale over subsequent years. **Operating the data pipeline is a central part of the research effort.** Springenberg estimates that getting the pipeline, data selection, and data quality right accounts for more than 50% of the work.

The collection process follows a concrete sequence:

1. **Choose tasks.** An expanding set of tasks, including folding clothes and bagging groceries, tests what the robots can currently learn.
2. **Demonstrate with teleoperation.** Human operators use a custom runtime and move leader arms attached to their own arms. Software transfers those motions to the robot’s end effectors, allowing people to demonstrate intricate manipulation.
3. **Schedule and track episodes.** Collection runs around the clock. Each dash in the operations dashboard represents an operator performing an episode for a particular task.
4. **Annotate and filter.** Demonstrations are stored in cloud buckets, annotated, and selected for training using those annotations.

This makes task selection, robot operation, collection throughput, and training-data quality parts of one connected system.

![“From 0 to 1: Data Engine from the Ground Up” slide with task choices, a teleoperation image, an Operations Dashboard overlay, and dashed arrows leading to cloud storage and annotation.](https://ai.engineer/api/talk-essay-media/d0f42b3e8371384926922c017beed4b44a8ce4ca3fccaaebbda4dd85f7288410)

The data engine connects tasks, teleoperation, operations tracking, and an annotation pipeline.

Training on the demonstrations produces policies that perform the tasks autonomously. The example here is [**π0**](https://www.pi.website/blog/pi0), the model released late in the previous year, folding a shirt. The transition is from a human tracing dexterous movements through the teleoperation system to a learned policy controlling those movements itself.

### More hours, then more kinds of environments

[7:40](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=460s)

The presentation describes [Open X-Embodiment](https://robotics-transformer-x.github.io/) as the largest public robot dataset available when the team started: about 3,800 hours, largely from static scenes in laboratories around the world. That historical estimate provides the baseline for the team’s own collection effort.

After six months, the team reports about 10,000 hours of successful episodes across tens of environments and hundreds of tasks. That collection supported capabilities such as the shirt-folding policy. These are the talk’s operational collection figures, whose scope includes the successful-episode accounting and collection interval.

Another six months brought more static-scene data and significant collection using **mobile manipulation** setups. The data now covered hundreds of scenes and environments, with a much wider range of tasks. No new total in hours is supplied. Mobility expands where a robot can act and the situations it encounters, so the collection effort grows in diversity as well as volume. The accompanying autonomous policy illustrates the resulting capability gains, while the presentation emphasizes that the larger, more varied pipeline also creates additional engineering challenges.

### From early object selection to dexterous control

[9:04](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=544s)

The model trajectory follows a pattern familiar from multimodal language models, with a delay. Over the preceding three years, VLMs progressed from conversational agents toward reinforcement-learning-trained multimodal reasoning models and coding assistants. VLAs began developing after language models had already acquired vision encoders.

[**RT-2**](https://robotics-transformer2.github.io/) emerged in 2023 as an early example. The presentation credits colleagues now at Physical Intelligence, including Danny, with some of the early multimodal language-model work for robotics. These first VLAs demonstrated useful generalization: a robot could be asked to pick different objects in a scene resembling its training environments. Limited robot data still held them back, but the demonstrations helped generate wider interest in the field.

Dexterous models capable of controlling multiple robots began appearing in mid-to-late 2024. π0 was Physical Intelligence’s entry; the presentation describes it as open source and, in the team’s assessment, exceptionally dexterous. The broader category also includes [**Gemini Robotics**](https://deepmind.google/blog/gemini-robotics-brings-ai-into-the-physical-world/) and NVIDIA’s [**GR00T**](https://nvidianews.nvidia.com/news/nvidia-isaac-gr00t-n1-open-humanoid-robot-foundation-model-simulation-frameworks) family. Those examples extend beyond the initial 2024 wave: Gemini Robotics and GR00T N1 were announced in March 2025, and the talk does not specify a GR00T version.

The architectural change is consequential: these models use diffusion-based action generation to support the fast output needed for physical control. A robot needs more than an appropriate description of what should happen next; it needs a stream of actions that can execute dexterous behavior. That leads to the next question: what happens when this control architecture is trained on substantially more diverse environments?

### π0.5 couples task decomposition with continuous actions

[11:18](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=678s)

[**π0.5**](https://www.pi.website/blog/pi05) was developed to investigate how capabilities change as collection becomes more diverse, with the goal of generalizing to unfamiliar environments. Its training mixture combines static and mobile robot data with web data, object-detection data, and language annotations of robot demonstrations. A large annotation pipeline helps connect physical demonstrations to language and scene understanding.

The architecture starts with a pretrained transformer and adds an **action-expert transformer**. The large VLM backbone learns both to answer general questions about a scene and to break down a high-level request. For example, a request to clean a bedroom can become a subtask to pick up a pillow.

The action expert attends to the VLM’s internal representations, runs at a higher rate, and produces continuous robot actions using a flow-matching objective. The two components therefore connect different levels of the same task: the backbone selects a meaningful subtask, while the action expert generates the physical control needed to carry it out. The pillow subtask is a prediction to execute, not evidence that the pillow has already been moved.

![Architecture diagram with multimodal data on the left, a pretrained VLA and action expert in the center, robot action data on the right, and subtask prediction and continuous actions above.](https://ai.engineer/api/talk-essay-media/c84586f0e67eda07520818fea19f7181770b36ab06affdb180936d1c351b0115)

π0.5 architecture combines a pretrained VLA with an action expert and multiple data sources.

### Testing what transfers to an unfamiliar home

[12:47](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=767s)

The presentation reports difficult tasks lasting up to ten minutes per episode in entirely unseen homes. In the first demonstration, colleague Chelsea prompts the model to perform cleaning tasks, including cleaning a kitchen surface, in a home absent from training. The result is presented as an emerging sign of broad generalization. The duration describes the demonstrated capability, rather than a measured ten-minute reliability rate.

To separate the effect of environmental diversity from simply providing more data, the team trains π0.5 with a fixed amount of data while varying how many homes contribute to it, then evaluates in a held-out location. The presentation reports that held-out performance generally improves as training locations increase, with the yellow curve matching and slightly surpassing the comparison trained with that scene included.

The [π0.5 paper](https://arxiv.org/html/2504.16054v1) gives the more precise experimental conditions: the study varies training locations from 3 to 104 and uses 40,000 post-training steps to control the number of samples seen. Evaluation measures task progress on dishes, drawer packing, laundry, and bed-making in held-out mock homes. The paper describes the 104-location model as performing similarly to the control trained in the test home. The supported result is that **broader environment coverage can close the gap with location-specific training**; superiority over that control is not established.

The next demonstration makes the task hierarchy visible. In a new home, colleague Laura gives the high-level instruction to clean the bedroom. The model divides that request into activities such as putting trash in a bin and making the bed. The on-screen timer shows the robot operating autonomously for multiple minutes. Here, task decomposition supports a sustained sequence of physical actions from a single broad request.

### From unfamiliar homes to a partner’s robot

[15:03](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=903s)

Quan returns with a different transfer problem: a robot located far from the team’s office makes a cup of coffee end to end. He says the Physical Intelligence team has never physically accessed this robot. He also reports that producing the video required few iterations, without giving a success rate.

Hardware and deployment remain important, but Quan identifies software and model intelligence as major bottlenecks. His scaling hypothesis is that a model should work across hardware platforms without requiring substantial effort from the model team on each one. The possibility of deploying thousands or millions of robots is a conditional illustration of why that matters, not a deployment count or forecast. The coffee-making robot offers one piece of evidence: the team says its model controls the task despite its lack of physical access or detailed knowledge of the robot’s internals.

The partnership is technically direct. Physical Intelligence sends the partner the actual **model checkpoint** so it can run inference, and the teams engage in low-level technical discussions. That demonstrates a practical collaboration model; it does not establish that the robot required no adaptation data or integration work. Quan invites introductions to other companies, in person or by email, because the larger problem is still unsolved.

The closing bottleneck is people. A model that can operate any robot on any task remains an open scientific, engineering, and operational problem. Quan identifies recruiting outstanding people as the organization’s biggest current constraint and invites candidates across the roles a research organization needs—including people who can make a case for a role not yet listed. The invitation extends to conversations in person, online applications, and Twitter messages: advancing the models also requires building the team that can collect the data, operate the systems, and solve the remaining research problems.

## Resources

- [π0: Our First Generalist Policy](https://www.pi.website/blog/pi0) — Physical Intelligence's original model introduction, including laundry demonstrations and its approach to continuous robot control.
- [π0 Technical Paper](https://www.pi.website/download/pi0.pdf) — Architecture, training mixture, post-training procedure and evaluations for π0.
- [π0.5: Open-World Generalization](https://www.pi.website/blog/pi05) — An illustrated explanation of co-training and robot cleaning tasks in unfamiliar homes.
- [π0.5 Technical Paper](https://arxiv.org/html/2504.16054v1) — The original paper details the architecture, training recipe and controlled environment-diversity experiments.
- [openpi: Model Training and Inference](https://github.com/Physical-Intelligence/openpi) — Official code and instructions for working with Physical Intelligence models, including π0.5 support added in September 2025.
- [RT-2](https://robotics-transformer2.github.io/) — The original project page for transferring web-trained vision-language knowledge into robot actions.
- [Open X-Embodiment](https://robotics-transformer-x.github.io/) — A collection of robot demonstration datasets with standardized formats and accompanying RT-X research.
- [Introducing Gemini Robotics](https://deepmind.google/blog/gemini-robotics-brings-ai-into-the-physical-world/) — Google DeepMind's introduction to Gemini Robotics and Gemini Robotics-ER.
- [Introducing NVIDIA Isaac GR00T N1](https://nvidianews.nvidia.com/news/nvidia-isaac-gr00t-n1-open-humanoid-robot-foundation-model-simulation-frameworks) — NVIDIA's announcement of the N1 model within the GR00T family, explaining its reasoning and action components.

## Slides

### Physical Intelligence

Quan and Tobi aim to build a model that controls any robot to perform any task. Quan says the model is not ready today and will require scientific breakthroughs.

[0:36](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=36s) [Physical Intelligence title slide naming Quan and Tobi, with part of the mission statement obscured.](https://assets.ai.engineer/sites/aiecode2025/sha256/025abbb63445084c1b02b39c468a6bca2cc1632e72ab8cd0ebefe90f7fbfddd9.webp)

Physical Intelligence (π)

June 5, 2025

Quan & Tobi

[illegible] model that controls any robot to perform any task

Interface text:
AIE
aws

### Vision Language-Action Models

Physical Intelligence introduces vision language-action models and asks what they are.

[1:20](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=80s) [Physical Intelligence slide introducing vision language-action models and asking what they are.](https://assets.ai.engineer/sites/aiecode2025/sha256/5333085ab3e4f348362a887799993080b3b5406540ace5edf1706464f1b6fba6.webp)

TRAINING COMPONENTS
Physical Intelligence

Vision Language-Action Models (VLAs)

What are they?

4

Interface text:
AIE
aws

### Vision Language-Action Models

Tobi introduces vision-language-action models by starting with the familiar vision-language model.

[2:23](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=143s) [Physical Intelligence slide introducing vision language-action models and asking what they are.](https://assets.ai.engineer/sites/aiecode2025/sha256/c65fa452a2d33b71802c57bea75860ebe8cfe59599acc5b472a522a46067d3fb.webp)

TRAINING COMPONENTS
Physical Intelligence

Vision Language-Action Models (VLAs)

What are they?

4

AIE
Microsoft
smol ai

### From VLM to VLA

A VLM embeds an image and a text prompt, then generates a text answer. Here, it identifies the pictured object as a fruit bowl.

[2:28](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=148s) [Diagram of a vision language model receiving a fruit-bowl image and producing a text answer.](https://assets.ai.engineer/sites/aiecode2025/sha256/15f4d5a84ef54a7e8b9dbf8a1f2826053c54e02ec1da235a8032209051bb2487.webp)

TRAINING COMPONENTS
From VLM to VLA

“Answer: fruit bowl”

Vision Language Model (VLM)

ViT
“Given <img>. Describe what you see. Answer: fruit bowl”

5

Interface text:
AIE
Microsoft
smol ai

[2:47](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=167s) [A vision-language model uses an image of a fruit bowl to answer a description prompt.](https://assets.ai.engineer/sites/aiecode2025/sha256/9831db72a3ed1cf647c0720ae9a08df44df5b0ae1d185ad52e3ef78313952e1c.webp)

TRAINING COMPONENTS
From VLM to VLA

“Answer: fruit bowl”

Vision Language Model (VLM)

ViT
“Given <img>. Describe what you see. Answer: fruit bowl”

5

Interface text:
AIE
Microsoft
smol ai

### From VLM to VLA: Vision Language Action Model

A VLA adapts a vision-language model for robotics: it takes an image, an instruction and robot state, then produces actions to control the robot. The example asks it to fold a shirt.

[2:57](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=177s) [A vision-language-action model uses an image and robot state to produce actions for folding a shirt.](https://assets.ai.engineer/sites/aiecode2025/sha256/8019730192d00302c5036eb07ade4a1efbe8f533f55623ac8b86388f5147e522.webp)

TRAINING COMPONENTS
From VLM to VLA

“Action: [128, 9, 8, …]”

Vision Language Action Model (VLA)

ViT

“Given <img>. and <robot state> fold the shirt
Action: [128, 9, 8, …]”

6

Interface text: AIE; Microsoft; smol ai

### Physical Intelligence

The speaker introduces the engineering challenges of training VLAs and sets up a comparison with VLM training pipelines.

[3:19](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=199s) [A slide asks about engineering challenges in training vision-language-action models.](https://assets.ai.engineer/sites/aiecode2025/sha256/ba9ffb952260b7cf4bee13c8c9ecc4fd301a0de636316abd62d67717210f2a1b.webp)

TRAINING COMPONENTS
Physical Intelligence

Vision Language-Action Models (VLAs)

What are the engineering challenges in training?

7

Interface text:
AIE
Microsoft
smol ai

### Vision Language Model (VLM)

A VLM workflow uses web data, optionally supplemented with task-specific data. An off-the-shelf model is fine-tuned on a large cluster and deployed in the cloud.

[3:53](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=233s) [A VLM diagram shows web data and image question examples feeding a model, followed by training on a large cluster and deployment in the cloud. The speaker inset covers part of an example.](https://assets.ai.engineer/sites/aiecode2025/sha256/57fb4697fc2b48d08c407028de412a60f4d7cd880e7b9dcefc9f1abafe3709ca.webp)

Vision Language Model (VLM)

1. Any data from the web

“Given <img>.
Q: What’s in the image?
Answer in emojis.
A: 🍏🍌🍇🍐🍑🍈🍒

Who is wearing glasses?
man    woman

Is the umbrella upside down?
yes    no

OpenWebText2
arXiv
Wikipedia
The Free Encyclopedia

2. Train off-the-shelf model on large cluster
3. Deploy in the cloud

(π)
8

Interface text:
AIE
Microsoft
smol ai

### Vision Language–Action Model (VLA)

The VLA slide starts with a familiar web-data, cluster-training and cloud-deployment pipeline. The speaker asks what data source could play the web’s role in training dexterous robot behavior.

[4:06](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=246s) [A VLA heading and yellow model diagram appear above the same web-data, training and cloud-deployment steps shown on the preceding VLM slide.](https://assets.ai.engineer/sites/aiecode2025/sha256/697bbaa24d33c3bc36e9fbf8bfda42dcc12a7e158c0522a6cd099c36af65be09.webp)

Vision Language - Action Model (VLA)

1. Any data from the web

“Given <img>.
Q: What’s in the image?
Answer in emojis.
A: 🍏 🍌 🍇 🍐 🍑 🍈 🍒

Who is wearing glasses?
man    woman

Is the umbrella upside down?
yes    no

OpenWebText2
arXiv
Wikipedia
The Free Encyclopedia

2. Train off-the-shelf model on large cluster
3. Deploy in the cloud

(π)
9

AIE
Microsoft
smol ai

[4:15](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=255s) [Diagram of web data feeding a model, with training on a large cluster and cloud deployment.](https://assets.ai.engineer/sites/aiecode2025/sha256/d0aa48c55bd186649d936a00fb01346d2be7ba5ae013a4e834f2f40f5c7c06dc.webp)

Vision Language - Action Model (VLA)

1. Any data from the web

“Given <img>.
Q: What’s in the image?
Answer in emojis.
A: 🍏 🍌 🍇 🍐 🍑 🍈 🍒

Who is wearing glasses?
man    woman

Is the umbrella upside down?
yes    no

OpenWebText2
arXiv
Wikipedia
The Free Encyclopedia

2. Train off-the-shelf model on large cluster
3. Deploy in the cloud

(π)
9

AIE
Microsoft
smol ai

### The Robotics Data Question

Web sources and visual question-answer examples feed a model, but the equivalent data source for training dexterous robot behavior remains an open question.

[4:16](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=256s) [VLA slide asks what robotics data source is analogous to the web, alongside a model diagram.](https://assets.ai.engineer/sites/aiecode2025/sha256/70305344c19acbe684a5dd56ac7cb9891cbb8acac5e438490a8916e427be843c.webp)

Vision Language - Action Model (VLA)

1. What data source is analogous to web for robotics?
2. Train off-the-shelf model on large cluster
3. Deploy in the cloud

“Given <img>.
Q: What’s in the image?
Answer in emojis.
A: 🍏🍌🍇🍐🍑🍈🍒

Who is wearing glasses?
man    woman
Is the umbrella upside down?
yes    no

OpenWebText2
arXiv
Wikipedia
The Free Encyclopedia

(π)
9

Interface text:
AIE
aws

### VLA Model Adaptation and Deployment

Robotics still lacks a clear equivalent to web-scale data. The speaker says pretrained vision-language models need adaptation for high-frequency robot control, alongside custom real-time deployment.

[4:34](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=274s) [VLA slide identifies a robotics data-source question, model adaptation, and real-time capable custom deployment.](https://assets.ai.engineer/sites/aiecode2025/sha256/e8dda85075beef474319fb3587abcc3547fd0a30daaa088146a34c8c078dadb8.webp)

Vision Language - Action Model (VLA)

1. What data source is analogous to web for robotics?
2. Model needs adapting
3. Real-time capable custom deployment

“Given <img>.
Q: What’s in the image?
Answer in emojis.
A: 🍏 🍌 🍇 🍐 🍑 🍈 🍒

Who is wearing glasses?
man   woman

Is the umbrella upside down?
yes   no

OpenWebText2
arXiv
Wikipedia
The Free Encyclopedia

(π)
9

Interface text:
AIE
Microsoft
smol ai

### Physical Intelligence: Data Scaling

Physical Intelligence is building a data engine from zero, aiming to develop robot capabilities quickly and support scaling over the next few years.

[5:22](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=322s) [Section slide titled Physical Intelligence: Data Scaling, with the subtitle From 0 to 1 and 1 to Many.](https://assets.ai.engineer/sites/aiecode2025/sha256/4076d895383a165ad8459b97c21e698f5af3ef16e7d4faddd244d1e9ccded1ce.webp)

DATA COMPONENTS
Physical Intelligence

Data Scaling

From 0 to 1 and 1 to Many

12

Interface text:
AIE

### Open X-Embodiment Data Scale

When the team started, Open X-Embodiment was the largest public dataset, with about 3,800 hours of data, largely from static scenes in robot labs around the world.

[7:38](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=458s) [Data-scaling timeline marks Open X-Embodiment at approximately 3,800 hours, with an overview of its robot datasets.](https://assets.ai.engineer/sites/aiecode2025/sha256/df33252f1040c667e8af579bcb2ef14e3ea0b4336723c26c28533aba0846ba54.webp)

DATA SCALING

Open X-Embodiment*

1M Episodes from 311 Scenes
34 Research Labs across 21 Institutions
22 Embodiments
527 Skills
pour   stack   route
60 Datasets
1,798 Attributes · 5,228 Objects · 23,486 Spatial Relations

DATA
~3.8k hours
Timeline: ~6 months, ~12 months, now

* Open X-Embodiment Collaboration
17

Interface text: AIE

[7:43](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=463s) [Data-scaling timeline places Open X-Embodiment at approximately 3,800 hours, alongside a dataset overview.](https://assets.ai.engineer/sites/aiecode2025/sha256/9625201812eac9b403b3f1f73cd214f1f1451e3b34da6a02533751fd7f9adbcd.webp)

DATA SCALING

Open X-Embodiment*

1M Episodes from 311 Scenes
34 Research Labs across 21 Institutions
22 Embodiments
527 Skills
pour   stack   route
60 Datasets
1,798 Attributes · 5,228 Objects · 23,486 Spatial Relations

DATA
~3.8k hours
Timeline: ~6 months, ~12 months, now

* Open X-Embodiment Collaboration
17

Interface text: AIE, Microsoft, smol ai

### Pi Zero data scaling

The chart grows from ~3.8k hours to ~10k hours of internal data plus OXE at ~6 months, covering hundreds of tasks and tens of environments. The embedded video does not play.

[8:12](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=492s) [Data-scaling timeline shows approximately 10,000 hours of internal and Open X-Embodiment data, hundreds of tasks and tens of environments; an embedded video shows a play button.](https://assets.ai.engineer/sites/aiecode2025/sha256/8927b41cd3db70be0f4354a7fdf872516b50e05309f0e5d30de34f9abe26a7c2.webp)

DATA SCALING

Y-axis: DATA
~3.8k hours
~10k hours internal + OXE
π₀
Tasks: 100s
Envs: 10s
Timeline: ~6 months; ~12 months; now

18

Interface text:
AIE
Microsoft
smol ai

### Mobile manipulation data scaling

A bedroom demo shows an autonomous mobile robot following pillow and blanket instructions. The data spans >>10k static hours and >400 mobile hours, with hundreds of environments.

[8:36](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=516s) [Data-scaling slide reports over 10,000 hours of static data and over 400 hours of mobile data, with a bedroom robot demonstration labeled to pick up a pillow.](https://assets.ai.engineer/sites/aiecode2025/sha256/ee14c0918f12d7211260e154762dff4bbc15d3fed372fcdca6653f2ab6041785.webp)

DATA SCALING

>>10k hours static    OXE
>400 hours mobile    + Web-data

Y-axis: DATA
Timeline: ~6 months; ~12 months; now
Tasks: > 100s
Envs: 100s

Human: pick up pillow
autonomous, 10x speed
π

19

Interface text: AIE; Microsoft; smol ai

[8:53](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=533s) [Data-scaling chart with a robot adjusting a blanket in a bedroom.](https://assets.ai.engineer/sites/aiecode2025/sha256/99358c81e19e0c2f91816021ed385473e587ad9604d31d9f3c29bd2915af5da8.webp)

DATA SCALING

>>10k hours static    OXE
>400 hours mobile    + Web-data

Y-axis: DATA
Timeline: ~6 months; ~12 months; now
Tasks: > 100s
Envs: 100s

Human: adjust the blanket
autonomous, 10x speed
π

19

Interface text: AIE; Microsoft; smol ai

### Physical Intelligence: Model Scaling

The talk turns from data collection to model scaling, comparing industry trends in VLMs and VLAs to frame the capabilities those data can elicit.

[9:11](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=551s) [Section slide introducing model scaling and parallel industry trends for VLMs and VLAs.](https://assets.ai.engineer/sites/aiecode2025/sha256/7adf3c19ea9b3cd27cba9b20daafe3fcc2c3c97eb8ef4bff0e7e6ff3925cb1db.webp)

MODEL COMPONENTS
Physical Intelligence

Model Scaling

Parallel Industry Trends for VLMs / VLAs

20

Interface text:
AIE
Microsoft
smol ai

### Parallel Industry Trends

The timeline traces LLMs from conversational agents through multimodal and open models to reasoning models and coding assistants across 2023–2025.

[9:20](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=560s) [Timeline of language and vision-language model developments from 2023 to 2025.](https://assets.ai.engineer/sites/aiecode2025/sha256/4b943ae67f10bbaad63f74b9c435a6d6b6e5eac8d4c36ec7daf6cd072263fda3.webp)

VLMs & VLAs

Parallel industry trends …

LLMs (e.g ChatGPT):
conversational, interactive, emergent capabilities

(VLMs) GPT-4o, Gemini, Claude:
multimodal, generally useful

LLama, Gemma3, Molmo, Mistral....
Good open VLMs emerge

OpenAi-o1, DeepSeek-R1:
Reasoning models and coding assistants

Timeline: 2023, 2024, 2025

(π)
21

Interface text: AIE, Microsoft, smol ai

### Parallel industry trends

VLAs followed language and vision-language models on a lag: RT-2 appeared in 2023, followed by generation-zero models such as π0 that combine VLMs with diffusion for robot control.

[9:42](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=582s) [Expanded timeline adding embodied VLMs and an RT-2 robot example.](https://assets.ai.engineer/sites/aiecode2025/sha256/e3596070a24e7af939ef6e12ff5879337f91e68f30af3f63c04ea11a5ddd6499.webp)

VLMs & VLAs
Parallel industry trends ...

LLMs (e.g ChatGPT): conversational, interactive, emergent capabilities
(VLMs) GPT-4o, Gemini, Claude: multimodal, generally useful
Llama, Gemma3, Molmo, Mistral.... Good open VLMs emerge
OpenAi-o1, DeepSeek-R1: Reasoning models and coding assistants

Embodied VLMs: e.g. Palm-E
Initial VLAs: e.g. RT-2
“Given <img>... Bring me the rice chips from the drawer. Action: [128, 9, ...]”

Timeline: 2023, 2024, 2025
π
22

Interface text: AIE; Microsoft; smol ai

[10:39](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=639s) [Timeline of language and vision-language models alongside early and generation-zero vision-language-action models.](https://assets.ai.engineer/sites/aiecode2025/sha256/dc32d7005fa16d9a4fe4a0d51beeb64676dc2310eeaf456daa783b0441942de5.webp)

VLMs & VLAs
Parallel industry trends ...

LLMs (e.g ChatGPT): conversational, interactive, emergent capabilities
(VLMs) GPT-4o, Gemini, Claude: multimodal, generally useful
Llama, Gemma3, Molmo, Mistral.... Good open VLMs emerge
OpenAi-o1, DeepSeek-R1: Reasoning models and coding assistants

Embodied VLMs: e.g. Palm-E
Initial VLAs: e.g. RT-2
“Given <img>... Bring me the rice chips from the drawer. Action: [128, 9, ...]”

Gen 0 VLAs: e.g. π0, many others
VLM+Diffusion: Dexterous, multi-embodiment, object generalization

Timeline: 2023, 2024, 2025
π
23

Interface text: AIE; Microsoft; smol ai

### Physical Intelligence: next generation models

Physical Intelligence developed π0.5 while studying how more diverse data collection changes model capabilities. It describes π0.5 as a VLA with open-world generalization.

[11:18](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=678s) [Section slide introducing π0.5 and open-world generalization.](https://assets.ai.engineer/sites/aiecode2025/sha256/2783a4e41cfd3eef48a5b89ad504cb29fd71ac2e390a20943783f7d8c55f80ed.webp)

MODEL COMPONENTS
Physical Intelligence

Next generation models

π₀.₅ - A VLA with open world generalization

24

Interface text:
AIE

### π0.5—A VLA with open world generalization

π₀.₅ combines multimodal and robot action data. Its backbone breaks a request like “clean the bedroom” into subtasks, while an action expert produces continuous robot actions.

[11:58](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=718s) [Multimodal and robot-action data feed a pretrained VLA with subtask prediction and an action expert producing continuous actions.](https://assets.ai.engineer/sites/aiecode2025/sha256/78cd2cec278724987b2f43ce273d8d45c4256f9eb723f7c506f717a9125e50b8.webp)

WHERE WE ARE NOW
π₀.₅—A VLA with open world generalization
pi.website/blog/pi05

MULTIMODAL DATA
Verbal Instructions
“close the microwave”
Subtask Commands
“pick up the mitten”
Object Detection
Multimodal Web Data

PRE-TRAINED VLA
subtask prediction
“pick up the pillow[illegible]”
high-level prompt
“clean the bedroom”
low-level command
“pick up the pillow”
ACTION EXPERT (300M)
continuous actions
[illegible]

ROBOT ACTION DATA
In-the-wild Mobile Robot
In-the-wild Static Robot
In-office Static Robot
General Robot Data

26

Interface text:
AIE
aws

[12:07](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=727s) [Diagram connecting multimodal and robot action data to a pretrained VLA and action expert that predicts continuous actions.](https://assets.ai.engineer/sites/aiecode2025/sha256/4e77aa7e717e733396217fccb28ffad5487fdd53f97f73f42cbe8693e14b2fb3.webp)

WHERE WE ARE NOW
π₀.₅—A VLA with open world generalization
pi.website/blog/pi05

MULTIMODAL DATA
Verbal Instructions
“close the microwave”
Subtask Commands
“pick up the mitten”
Object Detection
Multimodal Web Data

PRE-TRAINED VLA
subtask prediction
“pick up the pillow[illegible]”
high-level prompt
“clean the bedroom”
low-level command
“pick up the pillow”
ACTION EXPERT (300M)
continuous actions
[illegible]

ROBOT ACTION DATA
In-the-wild Mobile Robot
In-the-wild Static Robot
In-office Static Robot
General Robot Data

26

Interface text:
AIE
Microsoft
smol ai

### π₀.₅ generalization across training locations

With a fixed amount of training data, π₀.₅ generally performs better in a held-out home as the number of training homes increases. At 100 locations, it slightly surpasses the oracle trained on the test scene.

[13:36](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=816s) [Chart showing test-scene performance generally increasing with the number of training locations, alongside an oracle reference.](https://assets.ai.engineer/sites/aiecode2025/sha256/4f262bc3fa62aabed89185999d564ba7e9169488dc78cd5315391ba420b76ea2.webp)

WHERE WE ARE NOW

π₀.₅—A VLA with open world generalization
pi.website/blog/pi05

Y-axis: Out-of-Distribution Follow Rate
Y-axis ticks: 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0
X-axis: Number of Training Locations
X-axis ticks: 0, 20, 40, 60, 80, 100
Legend: π₀.₅ tested on a new scene; Oracle, trained on the test scene

28

Interface text: AIE; Microsoft; smol ai

[13:54](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=834s) [Chart showing performance in a new scene generally increasing with the number of training locations, alongside an oracle baseline.](https://assets.ai.engineer/sites/aiecode2025/sha256/31ddc83769e879a90af5f958bcd067d45e756aca274a6d33c28117fb7085b57e.webp)

WHERE WE ARE NOW

π₀.₅—A VLA with open world generalization
pi.website/blog/pi05

Y-axis: Out-of-Distribution Follow Rate
Y-axis ticks: 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0
X-axis: Number of Training Locations
X-axis ticks: 0, 20, 40, 60, 80, 100
Legend: π₀.₅ tested on a new scene; Oracle, trained on the test scene

28

Interface text: AIE

### Partnerships: Coffee-making demonstration

A robot makes coffee end to end, dispensing it into a cup. The speaker says the team has never seen the robot in person and needed few iterations to produce the video.

[15:31](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=931s) [Coffee dispenses into a cup beside a robot arm and an orange coffee machine.](https://assets.ai.engineer/sites/aiecode2025/sha256/63f8a093bab350ef991eeff4ec5fea44233406356018c432d82d5437913ef426.webp)

WHERE WE ARE NOW
Partnerships

1x speed | Autonomous
Astribot 🤝 π

30

Interface text:
AIE
aws

### We’re hiring!

The team calls hiring its biggest bottleneck and invites candidates across research and engineering, including people who excel in unlisted roles. Apply at pi.website/join-us.

[17:10](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=1030s) [Hiring slide listing research, machine learning, software, electrical and mechanical engineering roles.](https://assets.ai.engineer/sites/aiecode2025/sha256/4cac628d5aa9d476bd877dbafd8cf35d8d32614ec29cb30ec243a5e23b1d09a1.webp)

OPEN ROLES
We’re hiring!

Research Scientist
Machine Learning Engineer
Research Engineer
Software Engineer
Electrical Engineer
Mechanical Design Engineer
<insert-job-you-excel-at/>

pi.website/join-us

31

Interface text:
AIE
Microsoft
smol ai

[17:14](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=1034s) [Hiring slide listing research, machine learning, software, electrical and mechanical engineering roles, with a careers link.](https://assets.ai.engineer/sites/aiecode2025/sha256/f506cda1c59d92b263aca9a84b584bf3346e79fe1fbbf620f66f20dade6c89af.webp)

OPEN ROLES
We’re hiring!

Research Scientist
Machine Learning Engineer
Research Engineer
Software Engineer
Electrical Engineer
Mechanical Design Engineer
<insert-job-you-excel-at/>

pi.website/join-us

31

Interface text:
AIE
Microsoft
smol ai

## Full transcript

[0:00](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=0s) [upbeat music] So good morning.

[0:16](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=16s) Uh, thanks for being here with us. My name is Quan. This is Tobi. Um, our mission is to make a model that can control any robot to do any task.

[0:26](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=26s) Now, this is not something that's ready today. Uh, we believe that there are multiple scientific, uh, breakthrough that needs to happen for us to get there. And so we're very open.

[0:35](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=35s) We publish our research, we open source our model, and we re- talk very publicly about what is it that we do.

[0:41](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=41s) And so if you think about robotics before, and not to say that it's not useful, you know, it's incredibly impactful on the world, um, but the scenario that you often see robotic in, it's either in a very constrained environment such as, you know, a factory, very repe- repetitive motion, very structure environment.

[0:58](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=58s) Um, and then when you try to bring them into the real world, just kind of like semi-structure. I think this is a pretty kind of well-known video of a full-body humanoid, you know, struggling to perform a somewhat simple task.

[1:12](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=72s) Um, it was from some time ago. Um, if you look at robotics today, what do you see? You see,

[1:25](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=85s) you see kind of humanoid dancing. I know-- I don't think I can do that dance move. Um, [laughing]

[1:32](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=92s) um, I'll try. And so, uh, you see, you see kind of very complex physical motion that robots are capable of. Um, and you also see, uh, this video on the right, which is what we released late last year of a robot operating with kind of somewhat semi-structure objects.

[1:48](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=108s) You know, this is clothing that just came out of a dryer that ran before, so it's very hard to control, you know, the exact initial scene of, you know, how the shirt should be.

[1:57](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=117s) You know, it takes out all of the shirts, manage to put them in a basket, and in the full video, you know, it goes on to, you know, bring it to a table and then complete to fold them.

[2:06](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=126s) So what really changed? You know, the obvious answer is that there is this AI wave that we're all riding on. Robotic benefits a lot from kind of general AI development.

[2:15](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=135s) Uh, but there is also this vision-language-action model that, uh, we're pioneering. And so what are they? I'll pass it off to Tobi.

[2:24](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=144s) Cool. Yeah. So vision-language-action models, what are they? As Quan asked. Um, well, you probably all know by now what a multimodal LLM, or we often refer to it as a vision-language model, or VLM, is by now.

[2:37](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=157s) Um, essentially, a VLM generally takes text and images as input, so you have some sort of prompt for the model. Um, and then it embeds them and pass them to a transformer model to autoregressively produce an answer in text, right?

[2:50](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=170s) So you interact with these models probably, uh, as I do every day now. Um, so and then a VLA is essentially an adaptation of a VLM for the purpose of robotics.

[3:00](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=180s) The model additionally gets inputs describing the robot state, so the-- such as its joints positions, and, um, instead of asking questions about what's going on in the scene, we ask the model to produce actions to control the robot directly, right?

[3:17](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=197s) So if VLMs and VLAs are so similar in kind of principle, then what are the additional engineering challenges that we face when we try to train such a VLA?

[3:27](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=207s) Um, to understand this, I wanna contrast a little bit pipelines for VLM training to what we have to do when we train a VLA for robotics. So when you as kind of like a downstream customer of these like big models that have been trained want to use a VLM, you generally kind of take, um, a model, and

[3:44](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=224s) you can source data from the web, and maybe you have a little bit of extra data that, that you have for your specific task that you're interested in that you supplement.

[3:52](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=232s) Um, and then you probably train and use an off-the-shelf model, and you fine-tune it on a large cluster somewhere in the cloud. And then finally, you can use well-established libraries for inference and deployment in the cloud, right?

[4:03](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=243s) So all of this is like probably bread and butter for, for a bunch of you.

[4:08](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=248s) Now, in contrast to that, um, if we wanna do VLA training,

[4:12](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=252s) um, you want to train a model to exhibit kind of dexterous frontier-level behavior, then it's an open question what the analog of data source for the web actually is, right?

[4:24](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=264s) And we believe that this is kind of a trillion-dollar question for the industry in some sense, and it's an entirely open research question, as Quan already, uh, said.

[4:33](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=273s) And then secondly, while we can use, um, VLM backbones, so we can reuse some pretrained models, we typically have to adapt them and adapt the model architectures in order to allow for models that can control robots at so- somehow high frequency controls that we need to actually make progress.

[4:50](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=290s) And then finally, we-- there's also no standard solution for deploying large robot policies in multiple locations on premise, on device for robots, right? So this just doesn't exist. I won't really have time to talk about the third thing in detail here, but I wanna dive into a little bit about data and model training for robotics that we

[5:09](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=309s) do at PAI. So first, how can we design a data engine that enables robust, highly dexterous policies for difficult tasks with robots? At PAI, we kind of believe that there currently is no standard solution for this at all that would enable us to do this, and so we're building essentially a data engine from the ground up from

[5:27](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=327s) zero. We are designing this pipeline to get us very quickly to some sort of impressive capability, and I hope at the end of the talk you agree that it looks somewhat impressive, but also to enable significant scaling in the next few years.

[5:39](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=339s) Um, so we've seen firsthand that operationalizing this pipeline is actually kind of one of the main ingredients. It's probably more than fifty percent of the work is getting the data pipeline right, getting the right data, getting it to be high quality.

[5:51](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=351s) So how does it work? Well, we typically start from a set of ever-expanding tasks that we pick to test what's possible in the moment, right? So these are tasks such as folding clothes, bagging groceries, many other tasks that we're interested in.

[6:04](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=364s) And we then have human operators control our robots using a custom runtime and teleoperation system.

[6:10](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=370s) You can see the system in this video here. Uh, so what happens there is there's human operators, and they are controlling what we call leader arms, where they basically trace out motions with their arms, with, uh, robot arms kind of strapped to their arms, and the motion gets transferred via software to the actual robot end effectors, right?

[6:28](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=388s) And this way you can demonstrate fairly intricate, highly dexterous tasks and collect that data for training afterwards. Um, and then afterwards, we have a bunch of, uh, facilities to do kind of, uh, tracking of metrics of what's going on at the moment.

[6:47](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=407s) I think there's another slide here. Yep. Uh, so we basically schedule these data collection, uh, sessions all the time, and each dash in this, uh, dashboard here show us from this week, this is I think from Th- from Tuesday, an operator doing a specific episode for a specific task, right?

[7:03](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=423s) So we collect a lot of this data all around the clock, basically.

[7:07](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=427s) And then we, uh, annotate that data in the cloud. We, we, we serve it, uh, in, in big buckets there and can filter it back down based on annotations and use it for model training.

[7:19](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=439s) After we've trained, we then get policies that can actually solve the demonstrated tasks autonomously. So here in this video, you see a model that is Pi Zero. So this is the model that Quan referred to earlier already that we released, uh, late last year.

[7:32](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=452s) And as you can see, it can kind of do these like f- fairly impressive, highly dexterous tasks such as, as shirt folding.

[7:40](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=460s) Okay, so how far have we gotten by following this approach? Well, when we started, the biggest publicly available dataset was the Open X-Embodiment dataset, which contained about three thousand eight hundred hours of data, largely from static scenes in different robot labs around the world.

[7:56](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=476s) After kind of running this data pipeline that I've described for six months, we had collected about ten thousand hours of successful episodes as a successful data, uh, in tens of environments covering hundreds of different tasks, enabling, for example, the kind of shirt folding policies you, you saw before.

[8:11](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=491s) Oops. Um, cool. Is that video actually playing? Why is that video not playing?

[8:20](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=500s) Can I get it to play? It can't.

[8:26](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=506s) Let's see what we do with the next video. Um, and then after another six months, oh, this plays, uh, we have collected many more hours of data in static scenes, but crucially have also started to collect significant amounts of data using mobile manipulation setups such as the ones you, you see here.

[8:42](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=522s) So the data now spans many, many more tasks, and importantly has massively grown in diversity, covering hundreds of different scenes and, and environments. And as you can imagine, this scale and diversity enables new leaps, as you can kind of see here by this policy that is r- already running autonomously, and I'll go into detail a little bit

[8:59](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=539s) how this works, but also brings lots of additional engineering challenges.

[9:04](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=544s) So now that we have kind of described how we get the data, then the question is what kind of capabilities can we elicit with this data in VLAs, right?

[9:13](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=553s) And to understand where we are today, I think it's kind of useful to draw an analogy between industry trends for VLMs and VLAs, which have been kind of ongoing in the last three years.

[9:23](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=563s) So first, for multimodal LLMs or VLMs, we have seen a constant stream of improvements over the last three years, kind of starting from initial conversational agents that you've all interacted with, all the way to the RL-trained multimodal reasoning models and coding assistants that we all have today, right?

[9:40](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=580s) And we all use. Um, for VLAs, they follow a similar but time-lagged trajectory, basically. So initial VLAs such as, uh, RT-2, for example, done by some of my colleagues that are now at PAI, emerged in twenty twenty-three after LLMs had already been enhanced with, with vision encoders.

[9:58](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=598s) And in fact, actually some of the earliest multimodal LLMs were trained for robotics purposes by some of my colleagues, Danny and others, that are now working with us at PAI.

[10:09](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=609s) These were kind of impressive as first proofs of concept and showed some generalization capabilities, so you could kind of like ask them to pick different objects in the same kind of scene that, that kind of is in the training data.

[10:20](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=620s) But they are generally held back by a lack of, of available robot data. But, you know, nonetheless, they sparked this big explosion of interest in the field, which is probably why you're here today.

[10:30](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=630s) Um, then in the mid-twenty, so mid-twenty twenty-four towards end twenty twenty-four, the first kind of really dexterous multi-robot VLAs, uh, appeared, right? And the industry at large has now produced several of those.

[10:42](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=642s) Uh, there are models, for example, such as Gemini for robotics or NVIDIA's GR00T models that I think you'll hear a little bit about later as well. And our entry in this category was Pi Zero, which we believe is kind of perhaps the most dexterous, uh, multi-robot model that you can, you can use, and in fact is open

[11:00](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=660s) source as well. So these models generally adjust architectures to produce actions via diffusion to enable kind of very fast generation at, uh, high frequencies you need for, for robot control.

[11:12](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=672s) So if that's where we were then, what's next? Where are we now, right? Um,

[11:18](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=678s) so for us, the next leap was to study just how exactly model capabilities change when we increase data collection diversity, and this led us to develop π0.5, which is basically a VLA with open world generalization.

[11:32](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=692s) And I wanna talk a little bit more about this. Um, so what does this look like? In general, we have expanded massively, as I kinda said, the data we take in during training.

[11:40](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=700s) It now consists on both static and mobile robot data on the right here, as well as an extended set of multimodal VLM data, such as data from the web, object detection data, and general language annotations for the robot data that we've collected.

[11:54](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=714s) So we have a huge annotation pipeline as well, and this is what's on the left here. We then feed this data into a specially designed VLM, which starts from a pretrained transformer model and is expanded with an action expert transformer to the right.

[12:08](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=728s) This VLM part, so the big backbone of it, is trained to give predictions for both general questions about the scene now, but also to subdivide high-level requests that a human might have, uh, for the model, such as, for example, clean my bedroom, right?

[12:22](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=742s) So it is trained to subdivide these tasks into, um, subtasks such as pick up a pillow in the case of cleaning the bedroom. And at the same time, the action expert transformer can attend to the internals of the large VLM and can run at much higher rate and produces the actual continuous output actions via diffusion flow matching

[12:40](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=760s) objective, basically. Cool. I hope this video plays.

[12:47](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=767s) Can we get it to play? Yes. So training this architecture on all our data then leads us to a VLA that can perform difficult long-horizon tasks of up to ten minutes in, in each episode.

[13:00](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=780s) And this is, I think, much, much longer than what we've seen before, right? And it can do this in entirely unseen homes, showcasing perhaps the first sign, like the first true sign that broad generalization can emerge from VLA training.

[13:13](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=793s) So in this video here, you see my, uh, colleague Chelsea prompting the model in, uh, in an, yeah, you know, uh, an entirely h- new home that is not in the training data to essentially do p-perform multiple cleaning tasks such as cleaning a surface here, uh, in a kitchen.

[13:33](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=813s) Cool. And to understand this a-ability to generalize a little bit further, we tested how it emerges by training π0.5 on a fixed number of data, but varying the number of, uh, homes from which data was introduced during training.

[13:49](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=829s) Basically, then, uh, testing it in a held-out location, right? And as you can see, with increasing amounts of locations added, so this is the yellow curve here, performance generally increases in the test scene, as you would expect, until it surprisingly perhaps matches and even slightly surpasses training with the held-out scene specifically, right?

[14:07](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=847s) So this was a very cool result for us because we saw that we could kind of expand with more and more training data collections in different homes the capabilities of the model in new environments, which I think is, is pretty cool.

[14:23](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=863s) And then here we see the same model performing a bedroom cleanup task in a new home. And in this specific case, Laura, a colleague of mine, uh, is prompting a model only to basically clean the bedroom, right?

[14:35](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=875s) And you see here the power of this, these capabilities of subdividing into several tasks such as, you know, throwing trash in the bin and then, uh, cleaning, uh, making a bed basically.

[14:45](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=885s) And the-- as you can see at the bottom with the timer, you know, this policy is autonomously collecting, uh, uh, controlling this robot for multiple minutes at a time, basically.

[14:57](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=897s) Cool. And with that, uh, I give it back to Quan to talk to you a little bit about partnerships.

[15:03](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=903s) Thank you, Tobi. So what you're seeing is a robot that we have never seen in person. Um, we mean the team at PAI, uh, will never have access to it.

[15:16](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=916s) Um, this robot is running very, very far away from where our office is. Um, it's performing a somewhat interesting task of making a cup of coffee end to end.

[15:27](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=927s) Um, and it works pretty well. We didn't need to kind of iterate many times to, to, to be able to produce this video. Now, why is this important? Um, when you think about robotic, you know, oftentimes you would think about hardware, you would think about kind of real deployment challenges.

[15:45](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=945s) Now, those are important, uh, but it's our belief that actually one of the main bottleneck is software and just model intelligence. And if you think about what, if successful, would scale with maximum velocity in the sense that, you know, suddenly next year you have thousand and millions of robot deploy, uh, it-it's really about demonstrating the hypothesis that

[16:06](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=966s) our model can run across many different hardware platform out there, um, without us having to invest significant time and effort into trying to work with that hardware platform. So I think the demonstration that we've never touched this robot before, we don't know how it works internally, and yet our model can control that robot to perform a fairly

[16:26](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=986s) interesting task, um, is a piece of evidence in that direction. Uh, we have many more evidence here. Um, and so that's why it's important. Um, we are also very open when we work with other company because we believe that, you know, the problem is far from being solved.

[16:42](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=1002s) So when we work with this company, for example, that you can ask how do we run inference, we literally sent them the model checkpoint for them to run like inference with.

[16:51](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=1011s) Um, and you know, we, we have very low level technical discussion with them. This is to say that, you know, if you think there is a company that we should be talking to, um, please let us know.

[17:01](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=1021s) You can let me and Tobi know in person, and also, you know, just shoot us an email. Um, yep.

[17:09](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=1029s) And if you ask, you know, what is our biggest bottleneck right now? Uh, because we're after this mission of building a model that can work on any robot to perform any task, um, it's a scientific problem, engineering problem, operation problem that are far from being solved.

[17:26](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=1046s) And so our biggest bottleneck really is we need the best people in the world in this area to help us accelerate progress. Um, and so for a research organization, you know, any role that you might think we might need, we're hiring for it.

[17:38](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=1058s) Even if there is a role that you think you're exceptional at that, you know, we don't have on our website, please feel free to also let us know that, you know, you should really be hiring for this role and happy to have a conversation with you about it.

[17:49](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=1069s) Um, again, you can talk to me and Tobi in person about, you know, what is our hiring needs right now, but you can also apply online and shoot us a DM on Twitter.

[17:58](https://www.youtube.com/watch?v=cGLa8DsOYdk&t=1078s) Uh, thank you for listening. [clapping] [upbeat music]
