Robotics Has Been Stuck for 70 Years — Deepak Pathak, Skild AI

A robot looks at a picture of blocks and arranges physical blocks to match it. The task requires connecting a two-dimensional image to three-dimensional objects, then moving those objects precisely. In Deepak Pathak’s opening example, that apparently modern demonstration comes from the 1960s: the MIT copy demo. Pathak, Skild AI’s co-founder and CEO and a Carnegie Mellon professor, uses it to challenge the excitement surrounding robotics as AI’s next frontier. 1:27

The next clip reaches further back. A human controls robot manipulators through a leader–follower system in 1957. Its basic principle survives in modern teleoperation: a person supplies the motion, and a robot follows. Later examples include juggling, foosball, and a Berkeley robot that clears a table and packs objects into a box, which Pathak describes as the work of one graduate student with a single GPU machine. His dinner wager is deliberately provocative: distinguish old robotics videos from today’s.

The historical comparison supports Pathak’s diagnosis: robotics has concentrated on building particular machines without developing a general brain. That is a claim about the field’s direction, rather than proof that nothing improved. The useful distinction is between making one impressive behavior work and building intelligence that transfers to different tasks and situations.

Moravec’s paradox gives the problem its memorable phrasing: “Hard is easy, easy is hard.” Mathematical competition and chess look intellectually demanding to humans; climbing stairs and picking up chess pieces feel ordinary. Yet success at the intellectual task does not settle the physical one. Winning a board game leaves the problem of moving its pieces on an unfamiliar board.

Large datasets and large models have been a productive recipe elsewhere in AI. Robotics lacks an equivalent internet of action data. Teleoperation supplies useful examples, but each example requires a physical robot and a human’s time. Pathak estimates roughly one minute per example. More operators can increase collection, but they do not remove its labor and hardware costs.

Skild AI’s proposed answer is an omni-bodied intelligence: “any robot, any task, one brain.” A humanoid, quadruped, conveyor-belt arm, and dexterous hand should all contribute to the same learning system. The motivation is data scarcity. Restricting training to one machine would discard experience from other hardware, tasks, and environments that the model needs. 7:22

The teaser presents robots from different companies running the Skild brain, including locomotion in new scenarios and responses to disturbances. Its intended relationship is reciprocal: shared intelligence controls many bodies, and experience from those bodies improves shared intelligence. The demonstrations and deployment results throughout the talk are Pathak’s reports; they do not establish general success rates across arbitrary robots or environments.

Pathak’s retrospective moves through several attempts to collect robot experience. Each solves part of the problem and leaves another part exposed. 9:31

• Curiosity-driven exploration: Robots explore and collect their own “play data,” including forces and joint angles. This removes the need for a human to guide every action, but collection still proceeds through physical machines in the real world. Hardware cost and physical time constrain scale.
• Teleoperation: Human-guided trajectories supply data closely tied to robot action. Collection requires both robots and operators, and thousands of examples in one setup can still offer little environmental diversity. Moving the robot to a different home every day would improve coverage while making collection harder.
• Simulation: Deep reinforcement learning can train in simulation and transfer to a real robot. Simulated experience is easier to multiply, but someone must build the scenes. More trials within engineered scenes do not automatically create more kinds of environments.
• Human video: Videos offer scalable, diverse demonstrations of people doing things. Their weakness is distance from the robot’s action space: observing a human movement does not directly supply the joint commands a different body needs.

The evaluation has three axes: scalability, diversity, and closeness to the robot’s own joint-angle ground truth. Repeating a task in one room can produce a large dataset without teaching the robot much about a different room.

There is no single winning source. The useful discovery is that their weaknesses differ. Human video contributes variety that carefully engineered simulation struggles to supply. Teleoperation contributes direct robot experience that video lacks. Combining them gives each source a job instead of asking one collection method to satisfy every requirement.

The training recipe separates breadth from precision. Pre-training uses scalable sources such as simulation and human videos, accepting their imperfect relationship to real robot action. Post-training uses smaller amounts of high-quality teleoperation data. The analogy to language models is functional: broad initial learning gives later, narrower training something useful to adapt.

Deployment adds a third source. Once robots operate at scale, their experience can feed subsequent training. Pathak expects this data eventually to outweigh the initial sources, but that expectation depends on achieving deployment scale. A model restricted to one robot shape or hardware version would fragment the experience as machines change. An omni-bodied model is intended to keep those different deployments useful to one shared brain.

How does experience become reusable across the training stages? The diagram follows the proposed loop. Simulation and video supply the starting breadth; teleoperation helps turn that base into deployable behavior; operational experience returns to training. Supporting many bodies matters at the return path, where otherwise incompatible deployments must contribute to the same model.

The demonstrations begin by reconsidering what makes a manipulation task difficult. Laundry folding has a reputation for difficulty because fabric is awkward to model with hand-built physics. But a learned policy faces a different challenge: how much execution error can the task tolerate? A hand that lands a centimeter away from its intended position may still produce an acceptable fold. Pathak uses that tolerance to explain why laundry can be forgiving for learning-based control, despite being difficult for classical modeling.

Putting an AirPod-shaped earbud into its case exposes a tighter constraint. The robot has a parallel-jaw gripper, whose jaws close around the object without the in-hand manipulation available to fingers. The arm must approach and grasp the earbud in an orientation that permits the later insertion. A convenient pickup can leave the object inconveniently positioned for the next step.

The observable change is from a loose earbud to one seated in its case. Its causal sequence starts earlier: orient the arm, close the jaws around a usable grasp, carry the held object to the case, then align and insert it. The props are imitation earbuds costing roughly $5–$10 each, without magnets to help pull them into place. That detail removes a source of mechanical assistance: the robot must achieve the fit itself.

The next learning example uses egocentric human video followed by less than one hour of additional robot data to transfer behavior to humanoids. Third-person video remains a direction under investigation in this account; successful transfer would open access to a wider range of ordinary videos. The important bridge is from watching a human do something to making a different body do it, with limited robot-specific experience.

Those humanoids have hands, but Skild is not deploying hands in factories at the time described. Pathak’s objection is mechanical reliability. A more capable brain can make a simple gripper useful, while adding fingers does not solve the problem of fragile hardware. His challenge to anyone with a better hand is practical: he would buy ten units to test.

To explain learning from little robot data, Pathak introduces the robot’s “dream”: generated scenarios inside its own model. The displayed video represents imagined experience rather than a recording of physical trials. The proposed mechanism multiplies learning across scenarios without physically collecting every one. The talk does not specify the generation objective or how imagined outcomes are checked against real physics, so this remains a high-level account of the data expansion.

Omelet making then tests the approach on a roughly $4,000 setup. Its only sensor is a camera; it has no force sensing. Contact tasks therefore depend on visual information to guide actions that apply force. Pathak does not recommend avoiding better sensors. His point is that existing hardware can support more capable behavior than its current software extracts.

The cooking behavior is described as fully end to end, without a hand-written state machine deciding when the sequence starts or ends. An empty plate in front of the robot provides visual context for making an egg. Objects can move or be replaced, and the robot continues; the pan and gas stove remain the same while other objects differ. Pathak reports less than ten hours of training data for this example and attributes the resulting tolerance to the base model. The fixed pan and stove make the demonstrated scope concrete.

End to end has a specific meaning here: camera observations enter the brain, and its output drives motor power. Pathak contrasts this with a traditional pipeline containing explicit mapping and planning stages. He also mentions a final control component bridging 100 Hz to 500 Hz; the captions render its name as “P,” leaving its technical identity unclear. The supported distinction is direct learned control without an explicit map-and-plan pipeline, rather than a claim that no lower-level control exists. 20:41

Locomotion returns to Moravec’s paradox. Backflips and dancing look spectacular, but Pathak describes them as comparatively easy because the robot primarily needs to control its own known body. Stair climbing adds unfamiliar geometry. The controller must see the steps, respond to their height and width, and adjust to disturbances. Knowing the body is insufficient when the next foothold depends on the surroundings.

What changes in the control problem when a robot encounters stairs? The comparison below makes the extra information visible. A familiar body supports rehearsal of a body-centered maneuver. An unfamiliar staircase requires visual information about the environment to affect action. That connection explains why an ordinary-looking ascent can test more general capability than a dramatic flip.

The stair examples use a torso camera without constructing a three-dimensional map of the surroundings. The robot intentionally steps over obstacles, traverses different stairs including fire escapes, and responds to being pulled while climbing. Pathak identifies the model as the same across these scenarios. He then shows parkour and reports that the displayed behavior takes half an hour to train, while the earlier visually guided locomotion takes much longer. The training-time comparison reinforces his warning: spectacle is a poor shortcut for judging difficulty.

The final stretch moves from demonstrations to deployment. After an example of LAN-cable insertion, Pathak identifies deployment itself as the hardest part of starting the data flywheel. Physical operations introduce problems beyond intelligence and hardware, so waiting for a perfect robot brain would also delay collecting the experience needed to improve it.

• GPU assembly: Pathak describes work with NVIDIA for its Houston factory and reports that the assembly system is already deployed at the time of the talk. The example combines precise manipulation with a randomized, noisy setup. Its practical aim is to operate on existing factory lines amid the mess that accompanies human work. Tolerance of disturbances alone, however, does not establish that operation alongside people is safe without guarding.
• Package delivery: The robot travels from a truck to a house’s front door. Reaching the destination requires identifying which part of the scene is the front door, so the task combines movement with an understanding of the destination. Pathak reports deliveries with partners and says the same setup also works in warehouses.

The ending gives omni-bodied learning another purpose: adaptation when the body changes unexpectedly. A robot with a broken leg has effectively become a different robot. A brain trained to support different bodies may therefore handle damage as a change in embodiment rather than assuming that its original hardware remains intact. Pathak presents transfers to previously untrained robots and recovery after a leg failure as examples of that capability.

In one example, disabling legs leaves the robot learning to walk on two legs in three trials. In another, jamming a wheel causes the robot to start walking. Pathak describes adaptation ranging from milliseconds to roughly 30 seconds across these examples, and reports a further experiment in which two halves of a robot can operate separately. These are distinct forms of recovery, rather than one universal recovery time. Pathak presents continued operation after damage as a potential safety benefit, but these adaptation examples do not establish that continued motion improves safety or provides a complete safety guarantee.