Robotics Has Been Stuck for 70 Years — Deepak Pathak, Skild AI

[music]

I think uh all of you know here that how

much progress has AI made in the last

five years. Uh seems like it's more than

uh last 100 years of progress. And what

is happening especially in AI is AI came

for language then speech audio video and

everybody's excitement is next is

robotics okay and this excitement has is

going on peak uh this year uh for some

reason which is still beyond my uh

understanding and you can also see uh

leaders like Jensen talking about

physical AI as the next frontier of

robotics. So it seems like if you open

Twitter or LinkedIn, it seems like

robots robotics is already here and

we'll have home robots in our homes

sometime by December uh as uh as it's

being said by multiple people. But I'm

here to highlight that robotics has been

almost here for the last 70 years

[sighs]

now. Those of you who got into robotics

in the last four or five years, go and

take any any any course in robotics and

if you can distinguish the videos in

robotics from 30 years ago versus today,

uh I owe you a dinner. So to give you an

example, let's look at this robot. So

this robot here is an image and the goal

of this robot is to look at this image

of blocks and arrange these blocks in

front of it so that it looks the same

pattern as in the image. Okay, this is

an extremely simple task. But if you

think about this, it's a 3D block. You

have to look at understand 3D from each

side. And the robot can do this pretty

precisely. Any guesses how old this

result is?

Anyone? It's a guess. You can make any

guess. Huh?

>> 40 years. Okay, that's the highest.

[panting]

This is from 1960s. This is way before

there were computers. Uh, okay. So, this

is this is called MIT copy demo. Uh,

this was the start of uh this is this

predates AI. Uh, as you know, today look

at this one.

>> One exhibit at the Nuclear Congress in

Philadelphia has the perfect formula for

>> what's happening here. The guy behind

the scene is controlling these robots, a

leader follower system. And if you look

at any big lab, any big company, any big

academic lab, how they get data, they

use teley operation system, which

follows the exact same principles.

Any guesses for the year for this one?

>> Now, everybody would correct

aggressively on the year, but this is

from 1957, 68 years ago. Okay? Ask

anybody in your family who is older than

actually they may not be around like so

because this is for 60 you have to ask

somebody who is 80 year old what was the

state of technology at that time like

RAM for few KB of RAM you would have a

size of a room uh like this big of a

setup and this is from that time when

people could make it work on not just

this you keep coming back every decade

the videos look as cool like here is

robot a humanoid juggling playing

foosball uh with human and this is also

from you can see this is all they were

all predate deep learning this from 8

years ago so this robot is a result from

Berkeley can clean up the whole table

can arrange items in a in a in a box

this was done by one grad student with a

single GPU machine nothing more than

that okay now this picture of and if I

colorize all these videos from past and

apply genai filter for the modern voice

you cannot tell that this [laughter]

video is from 19657 or this is from

today And these are not even the oldest

results the oldest ones go to 1940s.

Okay. So what is this that if you look

at last 70 years tech every other

technology every other technology has

come way far language understanding

computer vision mobile phone chips why

is robotics is stuck in this primitive

age uh for for this long and the reason

is uh a general brain. Robotics has

always been approached as a hardware

problem from ground up and this the

reason that everything around robotics

has progressed but robotics is still

stuck in the same land from the last 70

years. So there's a very famous paradox

called Morave paradox. I don't know if

you know about Moravec. He was also a

CMU professor and I'm also CMU professor

so have to quote him for sure. He was

one of the founding figures in in AI.

After 30 years of working in robotics he

arrived at a very simple conclusion.

Hard is easy, easy is hard. Okay.

Whatever human believe to be hard is

extremely easy for computers and vice

versa. And you can see it happening

right in front of your eyes. You would

say doing math is hard. Math Olympiad

gold medal is really really hard.

Climbing a stair oh super easy. Now look

around you in technology where we are in

terms of what has been solved and what's

not being solved. So this is robotics

for you. Okay. It's very it is not yet

another application of AI. It is what AI

was founded for in the very beginning

and has made very little progress

towards. This is not yet another

application where deep networks can come

in and attack the field. This is has to

be thought of with fundamentally first

principles from the ground up. This is a

very famous example of where you can

have a you know a computer beat uh Gary

Casper on chess in '90s but you can you

still don't have a a a computer or robot

that can pick up the chess pieces and

arrange them on any chess board. Now,

so how do we go about solving this?

Okay,

so on a more positive note, uh we have a

magic sauce for AI success, right? You

get big data set, you train big models

and magic happens. Okay, now we know

this template is working well very well

in many topics. Okay, now can we apply

the same recipe to robotics? Well, it's

not uh directly applicable because we

have no data. There is no internet of

robotics data and as I said earlier you

can go and collect data uh manually on

the robot called telly operation and as

you notice teley operation is not 5 year

old thing this is a 68 or 70 year old

thing so the the funny part is if you go

back and you look at the whole evolution

of GPD3 GPD4 models go back to GPD3 uh

three years ago seems like a like a

lifetime ago GPD3 only began working and

caught people's attention when you could

train those models on trillions of

tokens. So GPD3 was already trained 30

trillion plus token and today hundreds

of trillions of tokens and if you

collect data by manually by tell

operation it takes you about 1 minute to

get one example. You can do the math if

you hire all of US population it will

take you more than a century to reach

the same scale as of GPD3 uh which is

like uh and nobody uses GP3 today. Okay.

So this is very slow and expensive. So

in robotics I would argue nobody has

really scaled robotics yet and we are

very far from talking about scale in

robotics with the way other uh other

areas have seen scale. Okay so this is

where uh we have been focusing on uh our

scaled is about uh 3 years old but uh I

have been working on the problem for

more than a decade uh that's all I've

done in my career uh nothing else. Uh so

at scale what our thesis is is to build

what we call an omniodied intelligence

any robot any task one brain okay any

robot it can be a humanoid it can be a

quadriped it can be a robotic arm on a

conveyor belt or a dextrous hand doesn't

really matter this even this hypothesis

is way more general than uh one would

argue humans are because we control our

own body and why do we have to go so

general well the argument is in robotics

there there is no data anyway. So I

can't pick and choose which hardware do

I use data from. And we should be able

to use data from any kind of hardware,

any kind of task, any kind of scenario.

And that's the only way to truly achieve

the scale of uh of what language models

achieved uh three years ago. Okay. So

the goal here is this uh this whole idea

of any robot any task one brain. And

through this talk I'll hopefully

convince you why this is the this is the

way to go towards robotics. Okay. So

this is the uh rough intro but before I

go into the more details let me show you

just a teaser result.

So in this result, every single robot uh

it's from a different company, different

hardware and they're all controlled by

uh skilled brain whether it's humanoids

going up and down stairs, any kind of

scenarios.

And these are not new results. They're

like a couple of year old uh results in

here. Robust to dust disturbances.

You can put them zero shot in new

scenarios.

So the idea here any every single robot

in this video takes any action anywhere

in the world the brain behind the scene

[music] improves because it's an

omni-body brain.

>> Yeah, this is really hard right because

like eggs are okay.

So that's a teaser of what we what we

work on. So I want to make this talk

more scientific and more uh more

informative than a company ad. So I'll

talk about uh how do we scale data in

robotics. Okay, let's take a tour back

as to how have people addressed this. uh

I'll take a look back at my own career

and my hypothesis for data has been

changing over the years. Okay. When I

began uh uh working scaling things the

idea was you can have robots you can

collect data for robots manually but it

is too slow and robots should be allowed

to collect data by themselves. So we had

this whole idea of curiositydriven

exploration. Uh allow robots to explore

in a curious way in the environment and

it has a lot of good parts like no human

required. Robots can go and keep playing

collect more data. We called it play

data at the time. It's very rich because

it has force sensors and joint angles.

But the difficulty is it is only in the

physical world. So it's very difficult

to scale. E Google was at it at the

time. Uh h having hundreds of robots.

Even for Google it's too expensive and

too slow to scale. The other idea is

again telly operation. As I said it's an

old idea but again the same issues

impossible to scale because now you

don't need robot you also need humans.

Uh so it's even more expensive and

diversity is very limited because even

if you put a robot in one setup with a

human you can get thousand examples but

they'll all be in the same setup. So

what you ideally want carry the robot to

new home every day in new scenario and

that's just impossibly hard to scale.

Then we had major breakthrough in

learning from simulation. This was one

of the first result where uh one could

show deep reinforcement learning uh

trained in simulation transfer to real

robot. This is one of one of the

award-winning paper uh at the time and

now it's used in every humanoid uh every

company out there. uh this was from our

lab at Berkeley and CMU. But again they

there are pros and cons. It's very easy

to scale but diversity is hard to get

because you have to engineer every scene

in simulation manually. So if you're

listening to this like there is no

single answer I'm coming to. Uh there is

no uh no no uh no single solution. This

is uh another work we did earlier. This

is all before skilled learning from

human videos. You watch the human do

things and then robot copies this. Again

high diversity. You can use videos from

YouTube etc. Highly scalable but it's a

very poor form of data because it's very

far from robot. Okay. So what is the

solution here? The answer is there is no

there is no golden path. You have to

think about data in the context of

different features. And in my opinion

there are only three features that

matter in in robotics.

scalability, diversity and how close you

are to your robot uh uh robot joint

angles. Okay, scalability means can I

quickly scale it across scenarios.

Diversity means can I get diverse data

because just having 100 trillion tokens

is completely useless if they're all in

the same environment and same scenarios.

Okay. And closeness to robot mean how

far are you from the ground truth of

robot own joint angles. So if you look

at simulation very scalable low diverse

uh uh but moderately close to robot

human videos are very scalable and

diverse but very far from robot data you

have to learn to map the human to robot

this other two teleop and the and the

manipulation interfaces they are kind of

in between and you can see here uh right

now around the world if you look at

different companies they're all focusing

on one of these approaches most of them

are on teleop or or yumi setups very few

on everything else and but there There

is no golden answer like they every each

one of them have a downside but the

golden light here is that they are all

complemented to each other like they are

not their cons are not exactly matching

from each other. So this is where uh the

recipe that we have converged to over

the years is to realize

separate the training into two parts

pre-training and post- training but

that's not surprise right but how do we

pre-train you want to pre-train using

data which is highly scalable and

diverse but maybe low quality so

simulation human videos etc. So that's

how you pre-train and then for post

training you use data from telly

operation. Now what is teleoperation

data? It's very high quality but low in

amount. Okay, high quality low in

amount. It exactly reminds us of the

recipe in language models. You pre-train

data on the internet then you post train

for coding for uh uh for your own

company etc.

But in robotics there is one more bucket

which is deployment data and deployment

data is highly scalable once deployment

scaled. So over time this data will take

over everything else. And what we are

trying to build is what we call this

data flywheel which goes which takes

this data from post training time and

puts back in pre-training. And now you

can see why this idea of omniodied brain

is extremely important because this is

very unlikely that you have only one

robot only one version deployed forever

in every task around the world. This

never happens in any area of hardware

except chips because they are very hard

to manufacture. Uh and you can still see

even in the chips uh inference chips are

coming left and right for many companies

these days. So in hardware this is never

the case. you have only one version of

the robot, one shape, which is why

omniodied intelligence is the enabler of

what we call a deployment data flywheel.

Okay, so let's now look at a few results

uh very quick. Now this brain is

extremely general. So we can do variety

of tasks very quickly. You may have you

have se you may have seen many tasks

like laundry folding etc. It's very

popular task in Silicon Valley for some

reason. uh and uh and the and the

argument here is uh when have you ever

thought while folding a t-shirt that if

I miss my hand by 1 cm my t-shirt fold

will be a blunder you don't think like

this people don't even think while

folding they just hold anywhere you just

do something so that the tolerance for

error is extremely extremely high then

why is this task hard

anybody why why do people get why do

people get fooled into believing this

task is

Let me put it that way.

Fabric, right? Why is fabric hard?

Simulation. But nobody's using

simulation anyway. This is all from

telly operation. Why is this hard? You

know why is it hard? Because it is hard

for classical version of robotics.

Classically in robotics, people would

model the whole physics, create models

by hand and then do this. It's very hard

for that. But for deep learning based

robotics, this is the easiest task

possible because it has high tolerance

for error. So it is completely uh now so

many companies focusing on this task.

Now what is hard is I would say

something about like let's say this

task.

If I ask you before seeing this video is

this task doable without having hands

very likely half the people will say no

because it requires putting like how

many of you have lost airports? Uh like

and in our company there's a channel

called right airpod because people keep

losing their right. Now in this case the

robot does not have hand it has gripper.

So here the task is very hard for the

gripper. So it requires a higher level

of intelligence because the arm has to

go and orient itself to pick up the

airpod in the right manner such that it

can be inserted because the grippers can

only close up and like like this.

They're parallejo grippers. So you

cannot turn the uh the the the airpod at

the very end. Now what you are seeing

here these are not real airpods. They

are fake ones from teu uh like $5 $10

each. So they don't have a magnet

inside. So the robot has to really work

hard to put this inside properly because

there is no magnet to pull it uh easily.

So these are all fake uh fake ones. This

is even harder than it appears uh in the

video. You can also uh like once you

build the general brain behind the

scene, you can also go and learn it from

variety of just human videos without

actually having any finetuning data at

tell operation time. So in this scenario

what we did, we trained on human videos

like this like egocentric videos. we are

now trying to transfer it to more third

person uh uh videos and if third person

works you can learn from YouTube any any

kind of open source data. So in this

case we see the video and then we add

less than 1 hour of robot data. So very

quick uh transformation and then the it

can transfer to humanoids. Now these

ones have hands. Hands or no hands is a

big debate. People often use hands as an

excuse as to why robotics is not here

but that's not the case. It's always the

intelligence uh behind the scene.

We don't deploy hands right now because

there are none available which can be

deployed in factories. They all break

within 100 yards. Uh pick anyone.

If you have better hand uh I would love

to buy 10 units ASAP to test. So it's

robust to scenarios. Now the idea is you

can do many tasks by watching humans.

And the reason we can work with very

less data which is less than one hour of

robot data is because robot imagines

things in its own head and tries to

multiply the learning from for many

scenarios. So what you see here is a

completely fake video uh you can call it

robot's dream. So it's inside the

robot's own model where it can imagine

scenarios uh in so this is not real

data. This is all fake uh from own

robot's own model. You can transfer it

to you know more complex task and even

lower cost hardware. So this is a egg uh

uh egg making like omelette making task.

Uh now here the these this whole setup

costs about $4,000. So extremely cheap

arms uh compared to the uh what you see

out there. And if you notice here this

setup is too cheap to even have any

sensors. So the only sensor here is just

a camera. No force nothing else. And the

robot can do task which require forces

uh from vision. Now I'm not saying

that's a future like you should not be

doing this. I'm sure the sensors will

improve but

>> from the existing sensors we are way

behind than where we can be from just

intelligence perspective.

>> Okay. So this can keep on going. This is

my adviser from Berkeley. He did not

believe the robots can do it. He just

kept standing for like uh 1 hour and the

robot kept making omelette. Uh and and

the interesting part here is this is

fully end to end system. There is no

state machine nothing. So the reason

robot is looking egg because there is an

empty plate in the front. I don't know

why it's fluctuating. It's not in the

there's no cut in the video. I I assure

you of that. It's HDMI problem. So as

soon as you remove the plate uh sorry I

don't know why this is happening.

Yeah. So there's no state machine when

the robot starts when the robot ends.

It's all automated and it's very robust

to even disturbances. You can change

objects around. Uh you can add these are

all unseen and unseen objects. Uh only

the pan and the gas stove is same.

Everything else is different and the

robot can keep on going. So you can get

basic robustness. So this was this is

all trained with less than 10 hours of

data. Uh so it's very low data to be

learning this robustness and it's coming

from the base model behind the scene.

I'm skipping uh this in the interest of

time. So unlike traditional robotics

pipelines where you have planning,

mapping etc. This is an end toend brain.

Now end to end is heavily used term in

in many areas of AI. But when I mean end

to end, I really mean end to end. It

reads directly from the cameras and it

applies power directly to motors. So we

use nothing uh in between except just a

P at the very end to go from 100 to 500

Hz. But P is not robotics contribution.

This is before it predates to like World

War II or something. Okay. Now the the

the interesting part is vision for us is

yet another input. So nothing that crazy

about it. So if you have seen, you may

have seen a lot of lot of results of

robots dancing, doing karate, kung fu,

backflip.

Go back and think how many results have

you seen of robots going up and down

stairs. Very few. Like even the top

companies out there in humanoid, they

have not shown beyond one sample stair

just to check the tick tick box. You do.

And people talk about humanoids are

coming, China versus US, all of this is

sort of BS. If humanoids cannot climb

stairs, what's the point of having legs?

All right, that's the only reason why

you have legs. Now, this is actually a

Moex paradox add action. Left one

actually is much easier for a robot. A

backflip, dancing, etc. is much easier

for the robot. And you may think, okay,

why is this paradox exist? Think one

level deeper. When a robot is doing a

back flip, it has to only know about its

own body, nothing else. Right? So it's

and when everything is known or fully

observed that's what computers are good

at because they can do they can simulate

every possible uh uh setup but the right

one even simply climbing on stairs

requires seeing the stair at the first

at the first place or what's the height

what's the width you don't measure

height and width exactly but you adjust

to all the disturbances and that

requires vision so right one requires

understanding and that's why it is hard

and you don't see any of this but for us

it's just get another result it doesn't

really matter so we And uh we put this

result out one and a half year ago. We

shot this like two and a half years ago.

Uh but uh like this robot can go any

scenario. It is not stumbling on things.

It intentionally steps over things and

there is no mapping or planning. It

doesn't make any 3D map of the

surroundings. It's all operating from

camera on the torso. And you can put

this in any kind of setup, any kind of

stairs. Uh I'm going faster here. These

are, you know, uh fire escape stairs.

It's a bit weird. Fire escape stairs are

to be used when there is fire in urgency

and they are the hardest in every

building. Uh so we are testing in all in

all those setups but you can put this in

anywhere you can disturb it and stay

doesn't really matter. This is

superhuman capability because robot

cannot see it's being pulled. So it's a

surprise factor for the robot that

you're pulling the violet while

climbing. And again it's the same model

everywhere in all these scenarios. So

very easy although easy parkour does

look fun uh uh does look fun and people

often find like uh this is uh you know

the robots can do all this but this is

so much easier than what I showed

earlier and even if you see these videos

right now many of you will find this

more impressive even though I'm telling

you this is easy it takes half an hour

to train this while the previous one it

takes much longer uh and much more

difficult to train

so you can take this Based model and you

can transfer to variety of tasks very

quickly. So here this is a very old

result from one and a half year ago uh

for inserting LAN cables etc. We have

very advanced systems now but we are

deploying these models already across

variety of application. So for instance

because to to build the data flywhe the

hardest part is the deployment itself.

There are so many other issues beside

intelligence and hardware that come up

when you deploy robotics. It is not the

same. This is not same same as deploying

chat GP over an app because you have to

work around many of the and I think

Skyio gave a talk before this and they

can talk to you about all day about the

hardness of deployment. So we have to

start deployment now so that we can have

this data flywheel in a foreseeable

future and I'll give you a few examples

of deployment we are doing. So one of

the examples with Nvidia like Nvidia is

opening their first factory in Houston.

uh and their GPUs what you use right now

they're all built outside US in in

Taiwan mainly and it's all done manually

over there and uh humos are really

efficient and really uh really good at

making these things but sustaining cost

and scalability and throughput here in

US it's very hard to maintain without

this labor force so here we are

automating uh this GPU assembly for them

this was we did a live demo in Nvidia

GTC this is deployed in factory already

uh last week so it's already live but

the interesting part I want to highlight

this is a very traditional factory task

but if you look at the the whole setup

this is extremely randomized extremely

noisy and you go to any factory

traditional setup they're extremely

clean so here the robot the same brain

can not only do very precise task it can

be very robust to any kind of

disturbance which means now these robots

do not need to be in a cage uh and they

can be uh just deployed as is on on

these factory lines without any change

to any factory line alongside humans and

wherever there is humans there is mess.

Okay, you can put the same brain to

delivery applications. So here the robot

delivers from the from the truck to the

to the door. It has to find where the

front door is. So it's a common sense

problem. It's not mobility problem. And

we already have been delivering uh

packages to with working behind the

scene with many partners to delivering

packages to front door of the houses uh

in these areas. same setup works across

warehouses and all this. Um now just to

close the talk uh close the topic I

mentioned this is an omnibody brain and

I hope it's very clear why it's

extremely essential to have an omnibody

brain to build data flywheel but there's

one extra benefit and the benefit is

your robots may break over time and uh

or the safety uh scenarios and safety is

a byproduct of this omnibody brain

because if you have a we can put the I'm

just skipping very fast here so you can

see the online this is all online uh but

we can put the same brain across each of

these systems each of these robots

and in this case we did not even train

on these robots. This is completely zero

shot transfer to all these humanoids and

all these things and even if the robot

breaks like here the leg breaks it can

recover within milliseconds because when

your leg breaks your three-legged robot

is a new robot. So it doesn't really

matter what's the shape of the robot is

anything changing here we disabled the

legs of the robot it learns to learn it

learns to walk on two legs in just three

trials. So what you are seeing on screen

is all the training for the robot that

they are that that is happening in these

systems. So it's it adapts in few

milliseconds to uh uh 30 seconds or so.

And like one example here it's it's

going on wheels. You jam the wheel it

starts walking. So this is a another

byproduct of omnibody brain. Safety

takes a very different meaning uh with

these kind of models. when the robot can

fly like or sorry when the robots can

walk or or operate with only half the

body. I removed some of the gory scenes

from this but we also cut the robot in

half and it can still work uh with both

the two halves separately but for that

you can go to YouTube uh and that's all

I have. Thank you.