# From VLM/VLA's to Embodied Agents — Armen Aghajanyan, Perceptron AI

Armen Aghajanyan, Perceptron AI

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/ZZcE0HeO-Hc-from-vlm-vlas-embodied-agents) · [Video](https://www.youtube.com/watch?v=ZZcE0HeO-Hc)

## From VLM/VLAs to Embodied Agents

Armen Aghajanyan explains Perceptron AI’s approach to combining perception, reasoning, and control: learn useful visual targets, spend compute on relevant tokens, and use video pretraining to reduce the need for expensive robot demonstrations.

### Perception, reasoning, and action in one model

[0:13](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=13s)

A camera keeps producing observations; a robot has to turn those observations into decisions and movements. Perceptron AI’s co-founder and CEO, [Armen Aghajanyan](https://www.linkedin.com/in/armenag), starts with that combined problem. The goal is physical intelligence that can perceive, understand, and interact with the world in real time, whether attached to a robot, an instrument, or a sensor.

![Source frame: Perception, reasoning, and action in one model](https://ai.engineer/api/talk-essay-media/711999b83f3bca95302202b04f8c0f312c46cc90a70cd90928a8e91918b8eb66)

Source frame: Perception, reasoning, and action in one model

The architectural starting point is **early fusion**: bring modalities together as early as possible. Aghajanyan connects this to his work scaling multimodal recipes at FAIR. The difficult part is choosing representations that work together on both sides of the model—what it receives and what it produces. Joining inputs early does not make text, images, video, and actions interchangeable; their representations still have to preserve what matters about each.

The familiar model categories describe different input and output capabilities:

- **Vision-language models (VLMs):** receive images or video alongside text and produce text. Embodied reasoning variants can also produce grounding points and handle spatial questions.
- **Vision-language-action models (VLAs):** extend the output space to actions, commonly using a VLM backbone.
- **World models:** produce video conditioned on inputs that may include images, video, and actions.
- **Semantic world models:** learn representations intended to be useful later, rather than directly producing a visible output.

An **embodied foundation model** is Perceptron’s framing for putting standard perception, embodied reasoning, and control within one model. It should reason across sensory inputs and support the relevant outputs together. That ambition immediately raises two training problems: how to learn enough from visual data, and how to process a stream that never stops.

### A million visual tokens, very little supervision

[3:45](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=225s)

Depending on the representation, one hour of video can bring roughly 1 million visual tokens into a model. The training target may be only a transcript or answers attached to a few synthetically labeled frames. In Aghajanyan’s example, the loss is calculated on something like 2% of the incoming token count. The model processes a large visual sequence while receiving a much smaller amount of explicit target information.

![Source frame: A million visual tokens, very little supervision](https://ai.engineer/api/talk-essay-media/433ec4c5c9c4ab3dceb8e5bb9cfb9e8953343e7a7d4cd5d6d270252841afeb43)

Source frame: A million visual tokens, very little supervision

Predicting every pixel supplies a dense target, but density alone does not allocate learning effort well. A background pixel receives the same importance as a gripper tip, a contact point, or evidence of a physical failure. For manipulation, those details have different consequences: the gripper and contact geometry can determine whether an action succeeds, while much of the background contributes little to that decision.

Perceptron’s proposed perceptive objective predicts percepts expected to matter in the future. A hand-chosen version might predict a robotic gripper’s future tip position: a useful target, but one whose importance the designer has already specified. The research question is whether the model can learn which semantic percepts deserve prediction automatically. Aghajanyan says Perceptron has a method, but does not disclose its construction here; the talk explains the target-selection problem rather than a reproducible objective.

### Let the task decide where compute goes

[5:46](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=346s)

Always-on cameras create the second problem: context bloat. Robots do not pause their observations while a model catches up. Video supplies many tokens, but useful information can be sparse relative to text. The first practical response is compression. Perceptron began by averaging patch representations across images or video, which Aghajanyan says can achieve up to 10× compression. It works, but he regards that fixed compression strategy as a hack rather than a sufficient architectural answer.

![Source frame: Let the task decide where compute goes](https://ai.engineer/api/talk-essay-media/a74d41e0dd20f2a4ba50fdcb51e46e814b4a09efdbb8fa93313c23d1cc905954)

Source frame: Let the task decide where compute goes

**Data sparse mixture of experts** gives the model a learned choice about which tokens to process. A router predicts which tokens to admit and which to skip across the model’s layers. This addresses a different question from the perceptive objective: the objective concerns what the model should learn to predict; routing concerns where the model spends computation while processing its input.

The described compute visualizations show two useful behaviors:

- **Information-driven allocation:** within a figure, the model concentrates on the graph, a region containing dense information.
- **Task-driven allocation:** a general question spreads computation across an image; asking to segment all the fruit shifts more tokens toward regions the model considers fruit.

What changes when the question becomes specific? The diagram follows the fruit example: the visual input stays the same, while the requested task changes the allocation. A general question gives the model little reason to favor a particular region. Segmentation supplies that reason, and computation concentrates on likely fruit. The architectural choice enables selective processing without hardcoding fruit locations or a fixed region of interest.

Same image, different compute allocation
The router selects or skips tokens across layers. Task-specific prompting changes which image regions receive more computation.
- General question — No specific object class is requested.
- Compute across the image — Allocation remains spread across regions.
- Segment all the fruit — The task identifies what deserves attention.
- More tokens for likely fruit — The learned allocation concentrates on task-relevant regions.
- General question → Compute across the image (Broad allocation)
- Segment all the fruit → More tokens for likely fruit (Task-dependent allocation)

### Finding a bird by changing how the image is examined

[8:54](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=534s)

Perceptron combines these ideas in a model trained on a reported 1 petabyte of text, images, videos, and trajectories. The trajectories include desktop use and video games; the collection combines internet crawls, custom training recipes, and synthetic pipelines. Aghajanyan reports performance exceeding Gemini’s embodied reasoning model at roughly 15× lower cost. These are his reported comparisons; the talk does not specify the benchmark conditions or cost basis needed to generalize them.

![Source frame: Finding a bird by changing how the image is examined](https://ai.engineer/api/talk-essay-media/d0b8cc6d8dcfbc7a2cd89557a40b2d8c2ded4787399159f7e98f7094e9d7319c)

Source frame: Finding a bird by changing how the image is examined

One resulting capability turns object detection into an agentic task. The model can write code, request a closer view of part of an image, and change contrast. A difficult image therefore becomes something it can investigate through successive operations, rather than something it must resolve with one bounding-box prediction.

The bird-finding example makes that investigation concrete. The model decides to tile the image, adjusts contrast, and proposes a candidate box. Aghajanyan then describes a contrast increase that lets it find the bird. The observable change is in the evidence available for detection: subdividing the image makes regions easier to examine, and changing contrast makes the target easier to distinguish. The final box follows those inspection steps.

The important connection is between perception and action even before a robot moves. Here, actions alter the model’s view of the input. Detection becomes a small investigation in which the model chooses how to look, then uses the resulting view to locate the object.

### Long tasks need decomposition—and ways to check the evidence

[10:51](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=651s)

Physical tasks add another design choice: how much work belongs to a single action policy, and how much belongs to an orchestrator. Making coffee might take three minutes. One approach asks a VLA to handle the entire task. Another uses an embodied reasoning model to break it into subtasks, with a tactile control policy handling execution. Aghajanyan presents a spectrum between these arrangements and treats embodied reasoning itself as an unsolved problem.

![Source frame: Long tasks need decomposition—and ways to check the evidence](https://ai.engineer/api/talk-essay-media/6cae3db67db53df7a1eab466b70603dbb32a2a55b8c60ace9b80edabcac0687a)

Source frame: Long tasks need decomposition—and ways to check the evidence

Robotic video annotation shows a related use of reasoning without physical execution. The model moves among different portions of a video, clips relevant intervals, checks whether captions match them, and verifies its own annotations. This extends the bird example’s inspection behavior into time: the model chooses which evidence to revisit instead of accepting an initial description unchecked. Aghajanyan connects the practicality of repeated inspection to the model’s speed and lower cost.

### Trading video pretraining for expensive teleoperation

[12:35](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=755s)

The largest research result Aghajanyan reports concerns how training sources substitute for one another. The unified model mixes control-based training, trajectories, perceptive training, and embodied reasoning. With suitable objectives and mixing, Perceptron finds a lever that a pure policy-training setup does not expose as strongly: general video pretraining can reduce the amount of teleoperation data needed.

![Source frame: Trading video pretraining for expensive teleoperation](https://ai.engineer/api/talk-essay-media/e9edeb1f6d7a0c1a23da6f689f37a9e711cf5d5594dfa3e7c81bdfd73283ba21)

Source frame: Trading video pretraining for expensive teleoperation

Teleoperation data costs on the order of $100 per hour in his account. The same budget can collect substantially more video pretraining data. Pure policies still benefit from additional teleoperation, but joint training opens a different spending decision: acquire more general visual experience instead of buying every increment of capability through robot demonstrations.

The reported trade is 10× more video pretraining data for 10× less teleoperation data. That relationship has held within the compute range Perceptron tested; the talk does not establish the task metric, experimental setup, or behavior beyond that range. Its practical significance is the possibility of reducing dependence on costly demonstrations while retaining control training, rather than eliminating demonstrations altogether.

### Reading a book title becomes a control decision

[14:22](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=862s)

The control demonstrations return to the original ambition: one model performs embodied reasoning and emits control tokens. The book-sorting task requires more than moving an object. It reads a book’s title, uses knowledge of what kind of book it is, and assigns it to an appropriate bin. Perception supplies the title; reasoning determines the category; control carries out the placement.

![Source frame: Reading a book title becomes a control decision](https://ai.engineer/api/talk-essay-media/3a01b919c7fb7cb0cba309359aa0b8221957d3f948cefde36b51adf56d6c32ba)

Source frame: Reading a book title becomes a control decision

Where does understanding enter the movement? The diagram separates the dependencies within the task, while the demonstration uses a single model. Choosing a bin depends on understanding the book, so a control model unaware of the necessary perceptual work has a harder problem than a policy that only needs to move an already classified object.

The motion is visibly jittery in Aghajanyan’s description; improving it through scale remains a hope. He also reports relatively good zero-shot behavior and anticipates opening a smaller model in July. That is an announcement made during the talk, not confirmation of a subsequent release. Larger model weights are offered to a limited set of partners.

The dependencies inside book sorting
These are explanatory stages of the task, not separate deployed models. A title must inform a category before control places the book in a bin.
- Read the title — Perception extracts information from the book.
- Identify the book type — Knowledge connects the title to a category.
- Assign the appropriate bin — The category determines the intended placement.
- Emit control tokens — The same model produces control output for the task.
- Read the title → Identify the book type (Interpret the title)
- Identify the book type → Assign the appropriate bin (Choose placement)
- Assign the appropriate bin → Emit control tokens (Carry out the task)

### Temporal understanding still needs context management

[16:17](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=977s)

The first Q&A answer narrows the deployment claim. Temporal understanding is not yet solved to a reliably deployable level. Even a context of roughly 1 million tokens can fill readily with high-frame-rate video. Aghajanyan describes keyframes versus delta frames as an earlier context-management approach Perceptron used and moved beyond: the decision is how much complete visual state to retain versus how much change to represent.

Pretraining also needs data that teaches relationships useful to robots. Left, right, and below are simple examples, yet internet crawls do not necessarily label them explicitly. Aghajanyan connects early attention to the data distribution with learning these capabilities quickly. Large quantities of general data do not automatically supply every spatial distinction a robot needs; the training mix must make those distinctions learnable.

### Background changes, missing cameras, and structured extraction

[18:12](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1092s)

Background robustness provides a concrete test of what the policy has learned. Aghajanyan describes fine-tuned action policies that fail when the table’s background changes. Joint perceptive and control modeling has made Perceptron’s models more tolerant of background changes and modest lighting differences in his observations. That tolerance has limits: he expects a flashlight directed into an arm camera could still break the behavior.

Training adds explicit variation alongside joint modeling:

- **Camera loss:** online augmentations simulate an arm camera being off.
- **Lighting changes:** augmentations simulate sunlight arriving from a particular direction.

These examples expose the model to altered observations during training. Aghajanyan credits the larger gains to bringing early fusion into robotics, with augmentation contributing additional robustness.

The final question asks about knowledge bases. The useful capability identified here is captioning and deep structured extraction from images and video, including complex egocentric annotation for robotics partners. Building an ontology is a separate undertaking. The model’s role is to turn difficult visual evidence into detailed structured information—the same kind of inspection and annotation work demonstrated earlier. Aghajanyan closes by saying the demonstrated capabilities are available through public APIs and that benchmarks are public.

## Key takeaways

- Visual supervision needs useful targets as well as density. Predicting every pixel can spend learning effort on background details; Perceptron proposes automatically learning future percepts, without disclosing the objective here.
- Learned token routing makes compute allocation task-dependent: a general question spreads attention across an image, while fruit segmentation concentrates more tokens on likely fruit.
- Perception can itself involve actions. Tiling, zooming, changing contrast, and revisiting video intervals let a model gather better evidence before producing a box or annotation.
- Joint training reportedly lets 10× more video pretraining substitute for 10× less teleoperation data within Perceptron’s tested compute range, offering a way to reduce dependence on demonstrations costing around $100 per hour.
- Combining reasoning and control enables tasks such as sorting books by their titles, but temporal reliability and resistance to severe visual disruption remain open problems.

## Resources

- [Armen Aghajanyan on X](https://x.com/ArmenAgha) — A contact route for the limited partner access to larger model weights offered near the end of the talk.

## Slides

### From VLM/VLAs to Embodied Foundation Models

Armen Aghajanyan proposes moving beyond distinctions among VLMs, VLAs, and world models toward embodied foundation models.

[0:32](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=32s) [Perceptron title slide introducing embodied foundation models, presented by Armen Aghajanyan.](https://assets.ai.engineer/sites/aiecode2025/sha256/f22e142c9c54b2beb497c449e8438b8f4e2c9b92def2f4e9b792f147300eb291.webp)

Perceptron

From VLM/VLA’s to Embodied Foundation Models

Armen Aghajanyan, Co-founder & CEO

Interface text:
AIE Conf
Audience - AIE Conf - Figma Slid[illegible]

### Perceptron's Physical AI Mission

Perceptron aims to build real-time physical AI that can provide intelligence to devices, from cameras and sensors to robots, bridging the physical and digital worlds.

[0:42](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=42s) [Perceptron's mission is to build physical AI that perceives, understands, and interacts with the physical world in real time.](https://assets.ai.engineer/sites/aiecode2025/sha256/807d85d0236815bfc846452a1b6dead1bb24df35ca9747bdc1865ac000ea9083.webp)

We build Physical AI to perceive, understand, and interact with the physical world in real-time

Our mission is to bridge the physical and digital worlds

Interface text:
AIE Conf
Audience - AIE Conf - Figma Slid

### Embodied Foundation Models

Aghajanyan frames embodied foundation models as a single perception–reasoning–action loop. The diagram brings together models that produce text, grounding, actions, video, or learned representations.

[1:12](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=72s) [Model types feed into an embodied foundation model that cycles through perception, reasoning, and action.](https://assets.ai.engineer/sites/aiecode2025/sha256/9416031cb723e53c96cc80c1b302bb7bc722f040b8296359d9cf5e7dc821b19e.webp)

Embodied foundation models unify perception, reasoning, and action

VLM — Vision-language models
Understand scenes, objects, and video
In: image / video, text
Out: text

VLA — Vision-language-action models
Map observations and goals to actions
In: image / video, text
Out: actions

ER — Embodied reasoning models
Reason about tasks, space, and affordances
In: image / video, text
Out: text, grounding

PWM — Pixel world (action) model
Predict future video conditioned on actions
In: image / video, actions
Out: video

SWM — Semantic world model
Predict in a learned representation space
In: image / video, text · optional
Out: learned representation
No output token — read out as grounding or text when probed.

EMBODIED FOUNDATION MODEL
Perceive
Reason
Act

INPUTS
image / video
text
actions

OUTPUTS
text
grounding
actions
Representations

Interface text:
AIE Conf
Audience - AIE Conf - Figma Slid…

### Two Fundamental Challenges

Armen Aghajanyan introduces two research challenges and plans to discuss how his company and others approach them.

[3:32](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=212s) [Section divider titled Two Fundamental Challenges.](https://assets.ai.engineer/sites/aiecode2025/sha256/3ed8c1b3024efaa0f638fcd05c8c602f464982b9e08bf9f110c4cc5ecd0b69c9.webp)

Two Fundamental Challenges

Interface text:
AIE Conf
Audience - AIE Conf - Figma Slid

### Traditional Objectives Are Weak

An hour of video yields roughly 1M visual tokens, while its transcript has about 2K. Aghajanyan argues that pixel prediction spreads attention across less relevant details; Perceptron instead targets future-relevant percepts.

[3:41](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=221s) [A diagram contrasts dense video tokens with sparse supervision and presents a perceptive objective alongside synthetic enrichment and pixel world models.](https://assets.ai.engineer/sites/aiecode2025/sha256/67d40d24a092ad75ed9e8fb9f1166208d0a7876a859b0d226a51fda78621ed11.webp)

1. Traditional Objectives are Weak
Severe disparities in modalities

1 hour of video
≈ 1M visual tokens
dense physical experience

≈ 500× fewer tokens

AVAILABLE SUPERVISION
transcript ≈ 2K tokens
captions / QA / labels · future pixels

Wrong training signal
too sparse, too synthetic, or too indiscriminate

Perceptive objective
predict future-relevant percepts

Synthetic enrichment
Adds captions, QA, labels, and traces.
Still lossy. Still bounded by the teacher.

Pixel world models
Predict future pixels — a dense signal.
But poorly allocated: background pixels compete with gripper tips, contact points, and failures.

Perceptive objective
THE RECIPE
Predict the percepts that will matter in the future.
Exact construction is proprietary.

Interface text:
AIE Conf
Audience - AIE Conf - Figma Slid[illegible]

### Context Bloat

Always-on cameras and robots can require minutes to hours of history. The slide illustrates the token burden with separate text, image and video examples: approximately 20, 1,024 and 8,192 tokens, respectively.

[5:44](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=344s) [Text, image, and video examples compare approximate token counts for continuous input.](https://assets.ai.engineer/sites/aiecode2025/sha256/77c60faa88cf9e86aa4962d72e08f2c85c9e4a73c16f850c801a8f19c1c4d633.webp)

2. Context Bloat

Always on input - Cameras and robots don't ask a question and wait — they don't stop.

Long context - Reasoning over physical events takes minutes to hours of history.

Text
The forklift entered the aisle at 14:02, made a three-point turn, and stopped beside pallet B-12.
TOKENS
~20

Image
TOKENS
~1,024

Video
Timeline: t=0s, t=2s, t=4s, t=6s
TOKENS
~8,192

Interface text:
AIE Conf
Audience - AIE Conf - Figma Slid

### Context Bloat: Token Compression

The diagram keeps text dense while compressing 16 image patches into 4 tokens and 64 video patches into 8 tokens, using spatial and temporal compression.

[6:18](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=378s) [Diagrams compare text, image, and video compression at illustrative rates of 1×, 4×, and 8×.](https://assets.ai.engineer/sites/aiecode2025/sha256/bd20b52cfc4665b31272960ef4995d6db7f8a84cb0db4a22be2eed0a82fd1b7a.webp)

Context Bloat: Token Compression

The core issue with context bloat is each modality is inherently different - our models are designed to compress modalities separately.

TEXT
Stays dense.
forklift → forklift
aisle → aisle
14:02 → 14:02
stopped → stopped
Compression rate: 1×

IMAGE
Spatial Compression.
16 PATCHES
4 TOKENS
Compression rate: 4×

VIDEO
Spatial × Temporal Compression.
64 PATCHES
8 TOKENS
Compression rate: 8×

Interface text:
AIE Conf
Audience - AIE Conf - Figma Slid[illegible]

### Context Bloat: Sparse Architectures

A router predicts which tokens to process or skip. The diagram sends one image token to three experts, illustrating dynamic compute per token.

[7:01](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=421s) [An image token is routed to three experts in a mixture-of-experts diagram.](https://assets.ai.engineer/sites/aiecode2025/sha256/f6c634147c3f1766e5c1e89795040e54ba0c772a730d140a4d766ca8b9a5ae64.webp)

Context Bloat: Sparse Architectures

Data sparse architectures - Mixture of Experts (Killian et al. 2026), allow our models to naturally model the sparsity of data leading to dynamic compute per token

text₁
text₂
img₁  3×
img₂
img₃

E1
E2
E3

Interface text:
AIE Conf
Audience - AIE Conf - Figma Slid[illegible]

### Emergent agentic capabilities

The detection demo scans and zooms into a ground image, then marks a snake with 0.94 confidence. The speaker frames detection as an agentic task in which the model can request actions such as zooming or changing contrast.

[9:50](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=590s) [Detection interface marks a snake in a ground image and displays a completed reasoning trace.](https://assets.ai.engineer/sites/aiecode2025/sha256/d049f09c81de779be140b98d56900f43dda3fd9d24041cbce120cda322b55b12.webp)

Emergent agentic capabilities

snake · 0.94

Perceptron detect
mk1-image

QUERY
find: snake

mk1 · reasoning
Coil geometry, body taper and the banded scale pattern are all consistent with a snake.
Confidence 0.94, above the acceptance threshold — locking the detection box.

proposes next step
→ lock detection · 0.94

TELEMETRY
tiles  6 · 640px / ovlp 110
focus  T5 [2,0]
zoom  2.7x
conf  0.94
box  330,1249 · 418,1305

DECISION TRACE
[illegible] · scan
[illegible] · tile
[illegible] · focus T5
[illegible] · zoom 2.7x
[illegible] · enhance
[illegible] · lock snake

mk1 — lock snake

Interface text:
AIE Conf
Audience - AIE Conf - Figma Slid[illegible]

### Emergent agentic capabilities

In a search for a ptarmigan, the demo tiles the landscape and zooms into a region. The speaker says the model increases contrast to find the bird.

[10:20](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=620s) [Detection interface searches a gridded landscape image for a ptarmigan.](https://assets.ai.engineer/sites/aiecode2025/sha256/ea8a594c4e57f6aa5895c4fa863fdef86e778d20c8352db464863ddfa99202cd.webp)

Emergent agentic capabilities

Image grid: A B C D E F; 1 2 3 4 5 6

Perceptron detect
mk1-image

QUERY
find: ptarmigan

mk1 · reasoning
The

TELEMETRY
tiles  6 · 640px / ovlp 110
focus  T6 [2,1]
zoom  2.7x
conf  0.25
box  –

DECISION TRACE
[illegible] · scan
[illegible] · tile
[illegible] · focus T6
[illegible] · zoom 2.7x

mk1 – zoom 2.7x

Interface text:
AIE Conf
Audience - AIE Conf - Figma [illegible]

### Embodied Reasoning as Orchestration

The bin-sorting example pairs a reasoning orchestrator with a VLA model. The speaker describes the orchestrator breaking tasks into subtasks, while the diagram connects it to thinking, tool use and planning.

[10:49](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=649s) [A robotics example connects an orchestrator and a vision-language-action model with thinking, tool use, and planning.](https://assets.ai.engineer/sites/aiecode2025/sha256/9d65da129eaa7319d0d6471c2b18dc9f384105d2899d6627f3c09de7d700ff56.webp)

Embodied Reasoning as Orchestration

Hey ALOHA, can you use my location and sort the items into the correct compost recycling and trash bins?

Based on San Francisco guidelines, I'll use the green bin for compost, the blue for recycling and the black for trash.

Agentic system
Gemini Robotics-ER 1.5 (Orchestrator)
Gemini Robotics 1.5 (VLA) model
Thinking
Tool use
Hinted search text
Planning

Interface text:
AIE Conf
Audience - AIE Conf - Figma Slid[illegible]

### Frontier Embodied Reasoning

Perceptron’s demo segments a robot motor-assembly video into seven actions. The model moves through the video, clips it and checks whether its captions are correct.

[12:10](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=730s) [A robot assembly video is accompanied by an analysis panel and a timeline divided into seven clips.](https://assets.ai.engineer/sites/aiecode2025/sha256/8f470672c27e85d78bb0cdffdd6d8b45984296b5774192273191a80f82ef1559.webp)

Frontier Embodied Reasoning

Perceptron understand
mk1-image

TASK
break episode → goals + <clips>

mk1 · reasoning
Episode segmented into seven atomic actions. Running a final pass and emitting the clip annotations with timestamps and component-level detail.

proposes next step
→ emit <clip> annotations

GOAL
Assemble the motor: grasp the rotor, position it over the coil-wound stator base, seat, align and secure it, then retract.

CURRENT ACTION

Video timestamp: t = 15.78s
mk1 · done · 7 clips
EPISODE TIMELINE
Timeline: 0s, 2s, 4s, 6s, 8s, 10s, 12s, 14s
Clips: 1, 2, 3, 4, 5, 6, 7
15.78s · 7/7
15.8s

Interface text:
AIE Conf
Audience - AIE Conf - Figma Slid[illegible]
0:17 / 0:17
1x

### Robot manipulation demonstrations

Two tabletop robot setups appear side by side; one arm reaches toward a bowl. The speaker says some fine-tuned policies fail when the table background changes.

[18:24](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1104s) [Two video views show robotic arms operating on tabletop setups.](https://assets.ai.engineer/sites/aiecode2025/sha256/21882f2400fcd6650ef6ae333270af5c21b6e34e3a1824bf925893d16ff62d17.webp)

2x speed

Interface text:
AIE Conf
Audience - AIE Conf - Figma Slid[illegible]
1:54 / 1:54
1x

### AI Engineer World’s Fair

[20:35](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1235s) [AI Engineer World’s Fair title card in white on black.](https://assets.ai.engineer/sites/aiecode2025/sha256/e44eae26b7060b247bdfc58fc6705f418ab33d1d71b1a5a544bbd4d10cf58069.webp)

AI Engineer
World’s Fair

HTTPS://AI.ENGINEER

## Full transcript

[0:13](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=13s) Yeah, I guess uh let's get started. I'm

[0:15](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=15s) I'm Armen. I'm I'm the co-founder and

[0:18](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=18s) CEO of Perceptron. Um, we'll get a

[0:20](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=20s) little bit into into what we do, but uh,

[0:23](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=23s) primarily what I want to talk about

[0:24](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=24s) today is kind of our research stance

[0:27](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=27s) that we want to move away from

[0:29](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=29s) distinctions between VLMs, VAS, world

[0:32](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=32s) models, whatever you want to call it, to

[0:34](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=34s) something that we call embodied

[0:35](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=35s) foundation models.

[0:42](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=42s) So specifically what we what we do at

[0:45](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=45s) Perceptron kind of our northstar really

[0:46](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=46s) is we want to be able to build physical

[0:49](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=49s) AI foundations that give us the ability

[0:51](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=51s) to perceive, understand and interact

[0:53](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=53s) with the physical world in real time.

[0:55](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=55s) And so kind of the the northstar mission

[0:57](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=57s) is really to bridge the the physical and

[0:59](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=59s) digital world uh worlds fundamentally

[1:02](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=62s) meaning that our our goal is kind of

[1:04](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=64s) wherever there is a um a device, an

[1:07](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=67s) instrument, a robot, a camera, a sensor,

[1:10](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=70s) uh we're essentially there providing

[1:11](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=71s) intelligence to it.

[1:14](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=74s) And specifically when we talk about this

[1:16](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=76s) paradigm of being able to perceive,

[1:18](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=78s) being able to reason, being able to act,

[1:20](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=80s) we view this as kind of a a unification

[1:23](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=83s) of uh traditional multimodal modeling.

[1:25](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=85s) So I came from fair, I was there for six

[1:28](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=88s) years and and and my target there was

[1:30](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=90s) really to try to figure out how to scale

[1:32](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=92s) up recipes for multimodal models. And so

[1:35](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=95s) one of the early things that we we

[1:36](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=96s) started working on and we've published a

[1:38](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=98s) lot in this domain was around early

[1:39](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=99s) fusion. So this concept that you want to

[1:41](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=101s) bring in all these modalities as early

[1:43](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=103s) as you can and and the real complexity

[1:46](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=106s) there is trying to figure out what is

[1:47](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=107s) the correct way to actually properly

[1:48](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=108s) represent all the different modalities

[1:51](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=111s) uh both on the input and on on the

[1:53](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=113s) output that you want to be able to

[1:55](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=115s) represent holistically and so VLMs have

[1:57](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=117s) kind of became the when I talk about

[1:59](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=119s) multimodal models um you probably think

[2:01](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=121s) of VLM. So this is the ability to take

[2:03](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=123s) in some image video and some text and be

[2:05](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=125s) able to output some some text

[2:07](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=127s) essentially. And there's there's

[2:09](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=129s) variations of this. There's there's

[2:10](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=130s) models like the ER models, the embodied

[2:12](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=132s) reasoning models that are able to output

[2:14](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=134s) maybe some grounding points um that are

[2:17](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=137s) able to do spatial understanding or

[2:18](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=138s) reasoning a little bit better. Then we

[2:20](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=140s) have things like VALAS that extend kind

[2:22](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=142s) of the output domain apart from just

[2:24](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=144s) text to now actions. And these are

[2:26](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=146s) traditionally of course continue to be

[2:28](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=148s) built on by standard VLM backbones.

[2:30](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=150s) Although there's been some efforts to

[2:32](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=152s) try to migrate away from uh from VLMs to

[2:36](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=156s) things like world models or world action

[2:38](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=158s) models, although nothing that's been

[2:39](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=159s) super fruitful just yet. And then we get

[2:42](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=162s) into kind of more interesting and

[2:44](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=164s) complex uh variants of multimodal models

[2:46](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=166s) like world models where you essentially

[2:48](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=168s) are outputting video uh from some inputs

[2:51](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=171s) and the inputs can be either image or

[2:52](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=172s) video or honestly image video and and

[2:55](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=175s) actions. Um and the last point that I'll

[2:57](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=177s) talk about is something recent which we

[2:59](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=179s) call semantic world models which is you

[3:01](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=181s) don't really output anything but you

[3:03](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=183s) learn some type of representation that

[3:04](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=184s) you think is useful um in the future.

[3:07](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=187s) What we kind of view as an embodied

[3:08](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=188s) foundation model is actually uh a

[3:11](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=191s) framing that allows you to both to do

[3:13](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=193s) the standard perception the embodied

[3:15](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=195s) reasoning and then the northstar target

[3:17](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=197s) of control all within one model. Um, so

[3:20](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=200s) being able to reason across different

[3:22](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=202s) sense of modalities on the input and

[3:24](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=204s) being able to do kind of most of what I

[3:26](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=206s) mentioned on the output all within one

[3:28](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=208s) single unified model.

[3:31](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=211s) And so very quickly I'm going to talk

[3:33](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=213s) about two challenges and these are kind

[3:34](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=214s) of fundamental research challenges that

[3:36](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=216s) we face and I'll talk about kind of how

[3:38](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=218s) our company has approached this um and

[3:40](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=220s) and and what other folks are doing in

[3:42](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=222s) here as well. So the very first thing is

[3:45](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=225s) think about purely if you're going to

[3:46](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=226s) try to model something like video like

[3:48](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=228s) one hour of video depending on the

[3:50](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=230s) representation you might have something

[3:52](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=232s) like 1 million visual tokens that are

[3:53](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=233s) coming in. The truth is is that there's

[3:55](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=235s) not actually any ground truth that you

[3:58](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=238s) can use uh effectively, right? So you

[4:00](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=240s) can do things like and people have done

[4:02](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=242s) this of course like pull out the

[4:03](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=243s) transcripts uh predict the transcripts

[4:06](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=246s) from the video or or maybe synthetically

[4:08](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=248s) label some frames, ask it some questions

[4:10](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=250s) and it turns out that this is kind of a

[4:12](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=252s) humongous waste, right? So if you think

[4:14](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=254s) about what's going into your model, you

[4:15](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=255s) have 1 million tokens going in and

[4:17](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=257s) you're essentially calculating the loss

[4:18](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=258s) on something like you know 2% of all the

[4:21](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=261s) tokens that are going in. And so this is

[4:24](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=264s) very fundamentally problematic and and

[4:25](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=265s) and the truth is is any way you try to

[4:27](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=267s) figure out how to fix this, you're

[4:29](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=269s) essentially injecting a wrong training

[4:30](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=270s) signal. Either the signal is too sparse,

[4:33](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=273s) it's too synthetic, or it's too

[4:35](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=275s) indiscriminate. And so approaches beyond

[4:37](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=277s) just synthetic enrichment have been,

[4:39](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=279s) well, let's predict every single pixel.

[4:41](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=281s) I mean, true, this is a very dense

[4:43](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=283s) signal, but it actually very poorly

[4:45](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=285s) allocates attention. there's, you know,

[4:47](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=287s) you're comparing a, you're treating a

[4:48](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=288s) background pixel with the same degree of

[4:50](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=290s) importance as you're treating a gripper

[4:52](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=292s) tip or the contact points or the

[4:54](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=294s) specific failures or the physics. And so

[4:56](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=296s) what we do kind of at Perceptron is, and

[4:59](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=299s) this is kind of some of our core IP is

[5:01](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=301s) we think about what does a natural

[5:02](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=302s) perceptive objective look like? So

[5:04](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=304s) specifically, how can I predict the

[5:06](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=306s) percepts that we think will matter in in

[5:08](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=308s) the future in a very very automatic way.

[5:11](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=311s) So, as an example, you might hardcode

[5:13](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=313s) something like, well, if you have a

[5:14](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=314s) robotic arm, well, the the the tip of

[5:16](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=316s) the grippers turns out to be a very

[5:17](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=317s) useful percept that you can predict into

[5:19](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=319s) the future. And folks have started doing

[5:21](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=321s) this. Um, like the Momo app folks from

[5:23](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=323s) AI2 have done this. There are other VAS

[5:25](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=325s) that have done this. But this is still a

[5:27](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=327s) hard-coded percept. So, the question is,

[5:29](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=329s) can you figure out an automatic way that

[5:31](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=331s) the model semantically is able to learn

[5:33](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=333s) this very unique objective? And the

[5:36](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=336s) truth is, we have figured out a way. uh

[5:38](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=338s) we're not going to share how we do it

[5:39](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=339s) here, but this is just kind of hinting

[5:41](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=341s) at how we approach the the problem of of

[5:43](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=343s) of sparity.

[5:46](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=346s) The the second core problem that we've

[5:48](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=348s) spent a lot of time focusing on is is

[5:50](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=350s) context bloat. So, if you have always on

[5:52](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=352s) cameras, if you have robots that don't

[5:54](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=354s) necessarily wait, they don't stop,

[5:55](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=355s) you're essentially having to reason over

[5:57](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=357s) a very very long um very long amount of

[6:00](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=360s) tokens. And and so text is relatively

[6:02](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=362s) dense and video is relatively sparse.

[6:05](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=365s) And so the question is, are there

[6:07](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=367s) architectural breakthroughs that are

[6:09](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=369s) there in order for you to be able to

[6:10](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=370s) deal with this problem problem natively

[6:13](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=373s) rather than just trying to on some

[6:15](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=375s) synthetic level figure out how to fix

[6:17](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=377s) this imbalance?

[6:19](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=379s) Um, and so there's a couple of things

[6:21](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=381s) that you can do. One kind of core first

[6:24](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=384s) principle is you need to start treating

[6:26](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=386s) different uh modalities as completely

[6:28](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=388s) different. So you can't treat text

[6:30](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=390s) tokens as the same as image tokens as

[6:32](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=392s) the same as audio or video tokens. So,

[6:35](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=395s) one thing that you can start thinking

[6:36](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=396s) about doing is focusing on spatial

[6:38](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=398s) compression or token compression. And

[6:40](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=400s) people do really dumb things and we

[6:41](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=401s) started off doing the dumb things and it

[6:43](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=403s) does work. You can start thinking about

[6:44](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=404s) averaging, you know, patch-wise

[6:47](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=407s) representations across a a video or an

[6:49](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=409s) image. Um, and you can start getting

[6:51](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=411s) some interesting compression rates, you

[6:53](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=413s) know, up up up to 10x.

[6:55](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=415s) But still, this is relatively of a hack

[6:57](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=417s) and there's there's not, you know,

[6:59](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=419s) wellused architectural methods to

[7:01](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=421s) actually solve this. Um, so what we've

[7:04](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=424s) done in in in the last couple of months,

[7:05](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=425s) we've released what we think is our

[7:07](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=427s) approach to dealing with um varying

[7:10](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=430s) degrees of sparity, which is just let

[7:12](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=432s) the model figure out what tokens it

[7:13](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=433s) should look at and what tokens it

[7:15](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=435s) shouldn't look like. And so we released

[7:17](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=437s) our data sparse mixture of experts

[7:18](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=438s) paper, which essentially allows you to

[7:20](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=440s) do this. Um, it allows the model, well,

[7:22](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=442s) there's a router in the model allows it

[7:24](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=444s) to actually predict what token I should

[7:27](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=447s) input, what token I should skip. Um, and

[7:29](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=449s) it allows us to do it kind of for free

[7:31](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=451s) through all the different layers.

[7:34](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=454s) And it turns out that if you just let

[7:36](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=456s) the model learn, if you're not actually

[7:37](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=457s) hard coding any significant um

[7:40](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=460s) architectural priors, um, the model

[7:42](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=462s) actually does learn. So if you end up

[7:44](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=464s) visualizing the data sparse compute uh

[7:47](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=467s) that our models use, you'll actually see

[7:48](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=468s) that our models innately learn to start

[7:51](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=471s) focusing on very um high density

[7:53](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=473s) information or task relevant

[7:55](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=475s) information. Um so in in in the upper

[7:57](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=477s) right you can kind of see that the model

[7:58](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=478s) decides to focus in on the graph which

[8:00](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=480s) actually you as a human would also kind

[8:02](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=482s) of zoom into because this is likely uh

[8:05](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=485s) what's interesting within the figure. Um

[8:07](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=487s) and it turns out that even task

[8:09](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=489s) dependent um uh task dependent

[8:12](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=492s) allocation ends up happening as well. So

[8:14](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=494s) if you look at the bottom left if you

[8:15](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=495s) just ask you know a very very general

[8:18](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=498s) question you're kind of going to see an

[8:20](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=500s) attention graph that is throughout the

[8:21](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=501s) whole image. So the model just doesn't

[8:23](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=503s) know what the proper way to allocate

[8:24](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=504s) compute is. At the same time, if you ask

[8:26](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=506s) it to do something like, you know,

[8:27](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=507s) segment out all the fruit, you can see

[8:29](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=509s) that it's going to allocate more tokens

[8:31](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=511s) to what it thinks are fruit tokens. And

[8:33](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=513s) so this is a very nice and clever trick

[8:35](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=515s) that that we use and we've published and

[8:37](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=517s) and other folks are starting to use

[8:38](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=518s) around embedding priors into the

[8:41](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=521s) architecture that are useful to deal

[8:44](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=524s) with the sparsity imbalances of your

[8:46](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=526s) modalities but not too harsh to the

[8:48](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=528s) point that the models aren't actually

[8:50](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=530s) learning uh natively.

[8:54](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=534s) And so kind of we we put all this

[8:55](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=535s) together and you guys might have seen

[8:56](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=536s) the release, but we essentially released

[8:58](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=538s) our our our model which was uh what we

[9:01](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=541s) considered to be the first embodied

[9:03](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=543s) foundation model a couple weeks ago. And

[9:05](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=545s) this model is essentially frontier um

[9:08](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=548s) with respect to Gemini 3.1 Pro. It's

[9:10](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=550s) actually better than Gemini embodied

[9:12](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=552s) reasoning. Um and it's something like 15

[9:14](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=554s) times cheaper. And it's essentially

[9:15](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=555s) trained on this one pedibyte data set

[9:18](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=558s) that we've collected across uh literally

[9:20](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=560s) everything. It's uh from from internet

[9:23](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=563s) crawls to to our own custommade training

[9:27](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=567s) recipes or synthetic data pipelines. We

[9:29](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=569s) have this one one pabyte of data across

[9:31](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=571s) text, images, videos, trajectories. And

[9:33](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=573s) these trajectories can be very very

[9:35](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=575s) general. They can be desktop use

[9:37](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=577s) trajectories. It can be playing a video

[9:39](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=579s) game trajectory.

[9:41](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=581s) And it turns out that once you start

[9:43](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=583s) doing these things, very interesting

[9:45](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=585s) properties end up emerging. And so the

[9:46](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=586s) the biggest property that we kind of saw

[9:48](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=588s) which is kind of obvious in retrospect

[9:50](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=590s) is that you can actually start thinking

[9:52](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=592s) of doing classical CV tasks as being an

[9:56](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=596s) agentic task. And so in this case we

[9:58](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=598s) essentially reframe detection as an

[10:00](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=600s) agentic task. So our model can you know

[10:02](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=602s) write code it can it can ask to zoom in

[10:04](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=604s) into specific portions. Uh it can change

[10:07](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=607s) the contrast. Um and you can actually

[10:10](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=610s) see here this is a very hard problem. I

[10:12](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=612s) think there's a whole Reddit subreddit

[10:14](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=614s) of these problems of trying to find very

[10:16](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=616s) hard objects and images. And our models

[10:18](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=618s) essentially do this very well, but they

[10:19](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=619s) do this in an agentic sense. So this

[10:21](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=621s) isn't the this isn't a classical, you

[10:23](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=623s) know, detect this one box. This is the

[10:25](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=625s) model actually deciding that it needs to

[10:27](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=627s) tile things up. It needs to change the

[10:29](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=629s) contrast. It proposes a box here. I

[10:31](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=631s) think here it yeah uh increases contrast

[10:35](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=635s) and it can find the bird. So again, very

[10:37](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=637s) hard to do if if like even for a human

[10:40](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=640s) and humans are very good at perceptive

[10:41](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=641s) tasks. This is a relatively tough thing

[10:43](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=643s) to do. And this all comes from just

[10:45](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=645s) having natively embodied models that

[10:48](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=648s) actually understand how to look at

[10:49](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=649s) different modalities.

[10:51](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=651s) And so following up kind of how does

[10:54](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=654s) this relate to the general physical AI

[10:56](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=656s) stance around robotics? So I kind of

[10:58](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=658s) stole this slide from from GDM folks.

[11:00](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=660s) And so one thing we've we're starting to

[11:02](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=662s) see from kind of robotics um u agentic

[11:06](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=666s) systems is this separation between what

[11:09](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=669s) we call kind of embodied reasoning

[11:11](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=671s) models or orchestrators and tactile

[11:13](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=673s) policy models. You can think of problems

[11:15](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=675s) as like you know if I have a if if I'm

[11:19](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=679s) making coffee and that takes me three

[11:20](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=680s) minutes to do I mean one thing I can do

[11:21](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=681s) is try to force my whole you know VLA to

[11:24](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=684s) try to figure out how to do this

[11:25](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=685s) individual task or what I can do is I

[11:27](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=687s) can have an orchestrator model that

[11:28](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=688s) breaks up this tasks into subtasks and

[11:31](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=691s) there's a tactile control policy that's

[11:32](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=692s) running on top and there's kind of a

[11:34](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=694s) full spectrum between you know full VA

[11:36](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=696s) only all the way to this kind of agentic

[11:38](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=698s) system. Uh but the main thing I'm trying

[11:40](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=700s) to highlight is that embodied reasoning

[11:42](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=702s) is actually very interesting and complex

[11:44](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=704s) problem that is yet to be solved. Um

[11:46](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=706s) that being said, our models continue to

[11:48](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=708s) be frontier on embodied reasoning. And

[11:51](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=711s) because they're frontier, we start

[11:52](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=712s) seeing really cool things that we

[11:53](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=713s) haven't seen before. Here's a concrete

[11:56](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=716s) example of doing very complex egocentric

[11:59](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=719s) or not egocentric, but this is robotic

[12:01](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=721s) data annotation. And you can kind of see

[12:03](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=723s) here the model is jumping around looking

[12:05](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=725s) at different portions of the video, uh,

[12:07](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=727s) you know, clipping it, figuring out

[12:08](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=728s) whether or not the captions are correct,

[12:10](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=730s) self-ver uh, self-verifying. And we can

[12:13](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=733s) all do this because AR models are fast.

[12:15](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=735s) They're significantly cheaper than

[12:17](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=737s) anything else that's out there. So, if

[12:19](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=739s) you try to do this with Gemini, this

[12:20](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=740s) video would probably cost you a couple

[12:21](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=741s) of dollars where for us, it's probably

[12:23](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=743s) in the sense. Um, and these all kind of

[12:25](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=745s) emerged from being able to have these

[12:27](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=747s) frontier embodied reasoning capabilities

[12:29](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=749s) that we just previously have not seen

[12:31](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=751s) from other models.

[12:35](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=755s) Um,

[12:37](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=757s) okay. Probably going to share the the

[12:39](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=759s) biggest research breakthrough that we've

[12:40](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=760s) had and I think we'll we'll share more

[12:42](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=762s) of this uh in the upcoming weeks

[12:45](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=765s) probably on Twitter. Uh, but one thing

[12:46](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=766s) that we found is we've we've we've

[12:48](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=768s) discovered new scaling laws for embodied

[12:50](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=770s) foundation models. So these are models

[12:52](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=772s) again that you can jointly do

[12:54](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=774s) control-based training, you can do

[12:56](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=776s) trajectory training, you can do

[12:58](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=778s) perceptive training, you can do embodied

[13:00](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=780s) reasoning training. If you just figure

[13:02](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=782s) out what the right way to mix this all

[13:03](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=783s) together is and the right objectives to

[13:05](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=785s) use, you actually start seeing very

[13:07](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=787s) interesting levers that you maybe

[13:08](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=788s) previously haven't been able to see

[13:10](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=790s) before. So the concrete lever that I'll

[13:13](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=793s) talk about is this ability to trade um

[13:16](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=796s) this ability to trade general video uh

[13:20](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=800s) pre-training data for teleop data. So

[13:22](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=802s) kind of as we know teleop data is very

[13:24](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=804s) expensive. It's on the orders of you

[13:26](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=806s) know $100 per hour of data for a hundred

[13:29](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=809s) you know dollars I can collect

[13:30](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=810s) significantly more video pre-training

[13:32](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=812s) data. And so what this graph is showing

[13:34](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=814s) is that um if you're just training pure

[13:37](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=817s) VA's pure policies there's this band

[13:39](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=819s) that you have. So you do actually still

[13:41](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=821s) have scaling loss. So you do get

[13:43](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=823s) benefits from more and more teop data.

[13:45](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=825s) That being said, the benefits are not as

[13:47](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=827s) substantial as if you are really

[13:48](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=828s) training these unified embodied

[13:50](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=830s) foundation models. And so this is what

[13:52](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=832s) kind of the bottom bottom half of the

[13:54](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=834s) graph is. Uh and the really cool kind of

[13:58](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=838s) uh lever that we get is you can

[14:00](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=840s) essentially trade 10x uh less teleop

[14:03](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=843s) data if you have 10x more video

[14:05](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=845s) pre-training data. And so far this is

[14:07](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=847s) kind of held for the amount of compute

[14:09](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=849s) that our company has. Um and it will be

[14:11](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=851s) continuing to to kind of push the fold

[14:14](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=854s) on uh how far you can push these

[14:16](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=856s) embodied foundation models.

[14:22](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=862s) So here's a couple of videos of of a

[14:25](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=865s) policy that hopefully will will um we'll

[14:28](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=868s) open source one of the smaller models in

[14:29](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=869s) a couple of weeks. Uh but this is all

[14:31](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=871s) running natively within a single model

[14:33](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=873s) that is capable of doing the embodied

[14:36](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=876s) reasoning in order to figure out the

[14:37](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=877s) task actually is outputting control

[14:40](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=880s) tokens. Um you can see it's a little bit

[14:42](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=882s) jittery but that's okay. Hopefully it'll

[14:44](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=884s) be figured out at scale and you can

[14:46](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=886s) actually see very complex tasks that

[14:48](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=888s) previously I think would be really tough

[14:49](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=889s) for pure VA to do. So I think if you

[14:52](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=892s) look at the right hand video, this is

[14:54](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=894s) requiring them all to actually read the

[14:56](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=896s) title of the book, have the knowledge

[14:57](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=897s) about what type of book this is and then

[15:00](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=900s) properly allocate it within one of the

[15:02](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=902s) bins. Um so this is actually a

[15:03](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=903s) multi-step task between perception and

[15:06](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=906s) control that is really really tough to

[15:07](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=907s) do if you have a kind of a pure

[15:09](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=909s) endto-end control model that is not

[15:11](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=911s) aware of the different perceptive tasks

[15:14](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=914s) that it needs to accomplish in order to

[15:15](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=915s) do this task.

[15:23](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=923s) Yeah,

[15:25](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=925s) it's pretty cool. It also works zero

[15:27](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=927s) shot relatively well out of the box. So,

[15:29](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=929s) we're excited to get this in the hands

[15:31](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=931s) of folks in in a couple of weeks

[15:33](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=933s) sometime in July.

[15:38](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=938s) Um, yeah, going to leave a couple of

[15:40](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=940s) minutes for for for general questions,

[15:42](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=942s) but if you guys are interested, let's uh

[15:45](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=945s) let's connect. So, one cool thing that

[15:47](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=947s) we do with our company is we actually

[15:49](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=949s) for a limited set of partners give

[15:50](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=950s) access to our Mark1 weights. We give

[15:52](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=952s) access to our larger embodied foundation

[15:54](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=954s) models weights. So, email me, DM me on

[15:57](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=957s) Twitter, uh, whatever is easier. Um, and

[15:59](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=959s) then yeah, I'll open up. There's a

[16:01](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=961s) couple minutes left for for questions.

[16:17](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=977s) these models

[16:22](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=982s) and like when I saw

[16:31](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=991s) a lot of contss

[16:39](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=999s) So can you talk a little bit on that?

[16:41](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1001s) >> Yeah, I think it's it's so yeah. So the

[16:43](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1003s) question is how are we able to nail

[16:44](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1004s) temporal and uh understanding to this

[16:47](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1007s) degree? Um it's a good question. I mean

[16:49](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1009s) to be honest it's not nailed. So there's

[16:51](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1011s) still a lot of work to actually get it

[16:52](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1012s) to a place where you can reliably deploy

[16:54](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1014s) it. The the core thing is how do you

[16:56](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1016s) think about context management? So you

[16:58](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1018s) have relatively limited context. So I

[17:01](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1021s) think the the the models here have 1

[17:03](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1023s) million context, but that's relatively

[17:04](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1024s) easy to fit in with the, you know, high

[17:07](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1027s) FPS video. And so you have to start

[17:09](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1029s) thinking about, are there interesting

[17:10](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1030s) things that you can do? I'll throw

[17:12](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1032s) something out there. We we used to do

[17:14](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1034s) this, but we got past this, but like how

[17:15](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1035s) do I think about like uh uh key frames

[17:18](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1038s) versus delta frames? How can I manage my

[17:20](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1040s) context by training these two off? Um,

[17:23](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1043s) and then you start thinking about during

[17:25](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1045s) your pre-training objective, how can I

[17:26](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1046s) start kind of natively ingesting things

[17:30](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1050s) that I think will be useful for the

[17:31](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1051s) robotics tasks. Um, so for example, a

[17:34](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1054s) very basic thing that even kind of the

[17:35](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1055s) the Gemini models used to struggle at, I

[17:38](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1058s) think the new ones are pretty good, but

[17:39](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1059s) being able to tell cardalities. So like

[17:41](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1061s) left and right is very hard to tell if

[17:43](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1063s) you do internet scale crawls because no

[17:45](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1065s) one on the internet is necessarily

[17:46](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1066s) labeling things as, you know, this

[17:48](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1068s) object is to the left of this object,

[17:50](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1070s) it's below this object. So really

[17:52](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1072s) thinking about data distributions early

[17:53](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1073s) on gives you this ability relatively

[17:55](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1075s) quickly. And it's also like er this type

[17:58](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1078s) of embodied reasoning was a very

[17:59](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1079s) concrete focus with this which is why I

[18:02](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1082s) think we were able to kind of surpass

[18:03](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1083s) Gemini ER with relatively less compute.

[18:12](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1092s) >> One issue with

[18:16](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1096s) sometimes I was wondering whether

[18:20](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1100s) you looked

[18:22](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1102s) your models are more robust than

[18:28](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1108s) >> Yeah. So, probably the coolest

[18:29](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1109s) robustness that we've seen is that uh

[18:32](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1112s) for VA models specifically, like if you

[18:34](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1114s) go and you take one of the Chinese ones

[18:37](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1117s) um and you try to fine-tune it for a

[18:38](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1118s) specific policy, if you just change the

[18:40](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1120s) background of the I don't know, you can

[18:43](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1123s) like in the table uh if you change the

[18:45](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1125s) the background of the table, the policy

[18:47](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1127s) will actually fail. What's really

[18:49](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1129s) interesting, if you do this type of

[18:50](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1130s) joint perceptive and control modeling,

[18:52](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1132s) you're much more robust to these types

[18:53](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1133s) of errors or if the light is hitting it

[18:56](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1136s) a slightly different way. And I think we

[18:58](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1138s) primarily view this as robustness to

[18:59](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1139s) background uh in a way that I think

[19:02](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1142s) traditional models don't necessarily

[19:03](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1143s) have. That being said, I'm not going to

[19:05](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1145s) overclaim like uh I mean like it's still

[19:07](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1147s) relatively hard. I think if I was going

[19:08](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1148s) to go and shine a flashlight into one of

[19:10](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1150s) the one of the arms, it's probably not

[19:12](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1152s) going to work. Uh but being able to

[19:14](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1154s) jointly model these things helps a

[19:17](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1157s) significant amount. We also do a lot of

[19:18](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1158s) online augmentations. So, so we do

[19:20](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1160s) actually, you know, fake um I don't know

[19:23](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1163s) um one of the one of the arm cameras

[19:26](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1166s) being off, right? We fake uh you know,

[19:29](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1169s) sunlight coming in from a certain

[19:31](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1171s) direction, right? So, we do these things

[19:33](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1173s) during training to improve robustness.

[19:35](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1175s) Uh but the big gains come from taking

[19:37](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1177s) this early fusion paradigm and then

[19:39](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1179s) moving it into the robotics domain.

[19:44](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1184s) >> Cool.

[19:47](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1187s) uh create knowledge base using the K1

[19:50](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1190s) model.

[19:51](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1191s) >> Uh oh, knowledge bases. Um it's useful

[19:53](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1193s) for like I I mean if you want to caption

[19:55](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1195s) images, videos, it's relatively well. We

[19:58](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1198s) work with robotics partners for like

[19:59](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1199s) very complex egocentric annotation like

[20:01](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1201s) this. Um so it's not necessarily

[20:03](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1203s) building out an ontology, but being able

[20:05](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1205s) to do kind of very deep structured

[20:07](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1207s) extraction, I think our models are are

[20:09](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1209s) very good at. Yeah. By the way,

[20:11](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1211s) everything that I kind of showed here is

[20:13](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1213s) kind of public APIs, so you can go play

[20:15](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1215s) around with it. The benchmarks are

[20:16](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1216s) public. Um, I think I'm out of time.

[20:19](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1219s) They're cutting me off. So, uh, I can

[20:22](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1222s) talk with folks outside, but thank you

[20:23](https://www.youtube.com/watch?v=ZZcE0HeO-Hc&t=1223s) guys.