From VLM/VLA's to Embodied Agents — Armen Aghajanyan, Perceptron AI

A camera keeps producing observations; a robot has to turn those observations into decisions and movements. Perceptron AI’s co-founder and CEO, Armen Aghajanyan, starts with that combined problem. The goal is physical intelligence that can perceive, understand, and interact with the world in real time, whether attached to a robot, an instrument, or a sensor.

The architectural starting point is early fusion: bring modalities together as early as possible. Aghajanyan connects this to his work scaling multimodal recipes at FAIR. The difficult part is choosing representations that work together on both sides of the model—what it receives and what it produces. Joining inputs early does not make text, images, video, and actions interchangeable; their representations still have to preserve what matters about each.

The familiar model categories describe different input and output capabilities:

• Vision-language models (VLMs): receive images or video alongside text and produce text. Embodied reasoning variants can also produce grounding points and handle spatial questions.
• Vision-language-action models (VLAs): extend the output space to actions, commonly using a VLM backbone.
• World models: produce video conditioned on inputs that may include images, video, and actions.
• Semantic world models: learn representations intended to be useful later, rather than directly producing a visible output.

An embodied foundation model is Perceptron’s framing for putting standard perception, embodied reasoning, and control within one model. It should reason across sensory inputs and support the relevant outputs together. That ambition immediately raises two training problems: how to learn enough from visual data, and how to process a stream that never stops.

Depending on the representation, one hour of video can bring roughly 1 million visual tokens into a model. The training target may be only a transcript or answers attached to a few synthetically labeled frames. In Aghajanyan’s example, the loss is calculated on something like 2% of the incoming token count. The model processes a large visual sequence while receiving a much smaller amount of explicit target information.

Predicting every pixel supplies a dense target, but density alone does not allocate learning effort well. A background pixel receives the same importance as a gripper tip, a contact point, or evidence of a physical failure. For manipulation, those details have different consequences: the gripper and contact geometry can determine whether an action succeeds, while much of the background contributes little to that decision.

Perceptron’s proposed perceptive objective predicts percepts expected to matter in the future. A hand-chosen version might predict a robotic gripper’s future tip position: a useful target, but one whose importance the designer has already specified. The research question is whether the model can learn which semantic percepts deserve prediction automatically. Aghajanyan says Perceptron has a method, but does not disclose its construction here; the talk explains the target-selection problem rather than a reproducible objective.

Always-on cameras create the second problem: context bloat. Robots do not pause their observations while a model catches up. Video supplies many tokens, but useful information can be sparse relative to text. The first practical response is compression. Perceptron began by averaging patch representations across images or video, which Aghajanyan says can achieve up to 10× compression. It works, but he regards that fixed compression strategy as a hack rather than a sufficient architectural answer.

Data sparse mixture of experts gives the model a learned choice about which tokens to process. A router predicts which tokens to admit and which to skip across the model’s layers. This addresses a different question from the perceptive objective: the objective concerns what the model should learn to predict; routing concerns where the model spends computation while processing its input.

The described compute visualizations show two useful behaviors:

• Information-driven allocation: within a figure, the model concentrates on the graph, a region containing dense information.
• Task-driven allocation: a general question spreads computation across an image; asking to segment all the fruit shifts more tokens toward regions the model considers fruit.

What changes when the question becomes specific? The diagram follows the fruit example: the visual input stays the same, while the requested task changes the allocation. A general question gives the model little reason to favor a particular region. Segmentation supplies that reason, and computation concentrates on likely fruit. The architectural choice enables selective processing without hardcoding fruit locations or a fixed region of interest.

Perceptron combines these ideas in a model trained on a reported 1 petabyte of text, images, videos, and trajectories. The trajectories include desktop use and video games; the collection combines internet crawls, custom training recipes, and synthetic pipelines. Aghajanyan reports performance exceeding Gemini’s embodied reasoning model at roughly 15× lower cost. These are his reported comparisons; the talk does not specify the benchmark conditions or cost basis needed to generalize them.

One resulting capability turns object detection into an agentic task. The model can write code, request a closer view of part of an image, and change contrast. A difficult image therefore becomes something it can investigate through successive operations, rather than something it must resolve with one bounding-box prediction.

The bird-finding example makes that investigation concrete. The model decides to tile the image, adjusts contrast, and proposes a candidate box. Aghajanyan then describes a contrast increase that lets it find the bird. The observable change is in the evidence available for detection: subdividing the image makes regions easier to examine, and changing contrast makes the target easier to distinguish. The final box follows those inspection steps.

The important connection is between perception and action even before a robot moves. Here, actions alter the model’s view of the input. Detection becomes a small investigation in which the model chooses how to look, then uses the resulting view to locate the object.

Physical tasks add another design choice: how much work belongs to a single action policy, and how much belongs to an orchestrator. Making coffee might take three minutes. One approach asks a VLA to handle the entire task. Another uses an embodied reasoning model to break it into subtasks, with a tactile control policy handling execution. Aghajanyan presents a spectrum between these arrangements and treats embodied reasoning itself as an unsolved problem.

Robotic video annotation shows a related use of reasoning without physical execution. The model moves among different portions of a video, clips relevant intervals, checks whether captions match them, and verifies its own annotations. This extends the bird example’s inspection behavior into time: the model chooses which evidence to revisit instead of accepting an initial description unchecked. Aghajanyan connects the practicality of repeated inspection to the model’s speed and lower cost.

The largest research result Aghajanyan reports concerns how training sources substitute for one another. The unified model mixes control-based training, trajectories, perceptive training, and embodied reasoning. With suitable objectives and mixing, Perceptron finds a lever that a pure policy-training setup does not expose as strongly: general video pretraining can reduce the amount of teleoperation data needed.

Teleoperation data costs on the order of $100 per hour in his account. The same budget can collect substantially more video pretraining data. Pure policies still benefit from additional teleoperation, but joint training opens a different spending decision: acquire more general visual experience instead of buying every increment of capability through robot demonstrations.

The reported trade is 10× more video pretraining data for 10× less teleoperation data. That relationship has held within the compute range Perceptron tested; the talk does not establish the task metric, experimental setup, or behavior beyond that range. Its practical significance is the possibility of reducing dependence on costly demonstrations while retaining control training, rather than eliminating demonstrations altogether.

The control demonstrations return to the original ambition: one model performs embodied reasoning and emits control tokens. The book-sorting task requires more than moving an object. It reads a book’s title, uses knowledge of what kind of book it is, and assigns it to an appropriate bin. Perception supplies the title; reasoning determines the category; control carries out the placement.

Where does understanding enter the movement? The diagram separates the dependencies within the task, while the demonstration uses a single model. Choosing a bin depends on understanding the book, so a control model unaware of the necessary perceptual work has a harder problem than a policy that only needs to move an already classified object.

The motion is visibly jittery in Aghajanyan’s description; improving it through scale remains a hope. He also reports relatively good zero-shot behavior and anticipates opening a smaller model in July. That is an announcement made during the talk, not confirmation of a subsequent release. Larger model weights are offered to a limited set of partners.

The first Q&A answer narrows the deployment claim. Temporal understanding is not yet solved to a reliably deployable level. Even a context of roughly 1 million tokens can fill readily with high-frame-rate video. Aghajanyan describes keyframes versus delta frames as an earlier context-management approach Perceptron used and moved beyond: the decision is how much complete visual state to retain versus how much change to represent.

Pretraining also needs data that teaches relationships useful to robots. Left, right, and below are simple examples, yet internet crawls do not necessarily label them explicitly. Aghajanyan connects early attention to the data distribution with learning these capabilities quickly. Large quantities of general data do not automatically supply every spatial distinction a robot needs; the training mix must make those distinctions learnable.

Background robustness provides a concrete test of what the policy has learned. Aghajanyan describes fine-tuned action policies that fail when the table’s background changes. Joint perceptive and control modeling has made Perceptron’s models more tolerant of background changes and modest lighting differences in his observations. That tolerance has limits: he expects a flashlight directed into an arm camera could still break the behavior.

Training adds explicit variation alongside joint modeling:

• Camera loss: online augmentations simulate an arm camera being off.
• Lighting changes: augmentations simulate sunlight arriving from a particular direction.

These examples expose the model to altered observations during training. Aghajanyan credits the larger gains to bringing early fusion into robotics, with augmentation contributing additional robustness.

The final question asks about knowledge bases. The useful capability identified here is captioning and deep structured extraction from images and video, including complex egocentric annotation for robotics partners. Building an ontology is a separate undertaking. The model’s role is to turn difficult visual evidence into detailed structured information—the same kind of inspection and annotation work demonstrated earlier. Aghajanyan closes by saying the demonstrated capabilities are available through public APIs and that benchmarks are public.