Perceptron AI

Aghajanyan frames an embodied foundation model as one model supporting perception, spatial reasoning, and control. Early fusion brings modalities together, while leaving the choice of input and output representations as a central design problem. He also discusses the alternative of an orchestrator decomposing a longer task, such as making coffee, into subtasks executed by a control policy. This gives readers a useful architectural comparison between a unified action model and a system that separates planning from execution; he treats embodied reasoning as unresolved.

In his illustrative accounting, an hour of video can contain roughly one million visual tokens, while transcript or sparse frame-label targets yield loss on something like 2% of the incoming token count. Predicting every pixel makes supervision dense but weights background pixels alongside gripper tips and contact points. Perceptron’s proposed perceptive objective instead learns to predict future percepts that matter, rather than hardcoding a gripper target. Aghajanyan explicitly withholds the method, so this part explains the training problem and intended objective, not a reproducible recipe.

Always-on cameras create context bloat. Aghajanyan reports up to 10× compression from averaging patch representations, then describes data sparse mixture of experts: a router selects or skips tokens across layers. His compute visualizations concentrate processing on a graph within a figure and shift more tokens toward likely fruit when asked to segment fruit. These examples distinguish learned, task-dependent allocation from fixed compression. He reports training on one petabyte spanning text, images, video, and trajectories, including desktop use and games. His performance and roughly 15× lower-cost comparisons lack benchmark conditions and a cost basis in the supplied material.

The bird-finding demonstration turns detection into successive inspection: the model can write code, tile or zoom into an image, adjust contrast, and propose boxes. Robotic video annotation extends that pattern through time, revisiting intervals, clipping footage, and checking captions against evidence. In the book-sorting example, Aghajanyan describes a single model reading titles, identifying book types, choosing bins, and emitting control tokens. He acknowledges jittery motion. Joint training reportedly permits 10× more video pretraining to compensate for 10× less teleoperation data, which he estimates at about $100 per hour. That relationship is limited to the compute range tested; task metrics and experimental methodology are not supplied.

The Q&A qualifies the demonstrations: temporal understanding is not yet reliably deployable, and even roughly one million tokens of context can fill quickly with high-frame-rate video. Aghajanyan reports improved tolerance to background and modest lighting changes through joint modeling, supplemented by simulated camera loss and directional sunlight during training, but expects severe glare could still defeat the policy. The final answer identifies captioning and deep structured extraction as useful capabilities without establishing ontology construction. He closes by saying the demonstrations use public APIs and benchmarks are public. His earlier anticipated smaller-model release sometime in July remains a statement made during the talk; September publication metadata establishes neither the recording date nor subsequent release status. The official website separately documents API access and commercial weight licensing.