One Operator, Many Drones: Inside Skydio's Autonomy Stack — Suchet Bargoti, Skydio

Thanks everyone for coming. So we'll be

talking about how um to use Agentic sort

of orchestration to command many many

different drones to achieve the

different tasks that we have ahead of

us. and just going to rethink this

presentation a bit and not just jump

straight into slides, but instead we're

going to fly. Um, so what we have here

on the left hand side is a real time

view. What we're going to be doing right

now, am I showing anything?

>> Well, I'm flying. There you go. Sweet.

Okay. Um, we are somewhere here in the

city down south, which where where our

headquarters is. Uh we have a few drones

up and I'm going to hit launch. Uh and

these are drones that are in docked

stations. So we calling this drones as

infrastructure uh where we actually have

thousands of these drones now planted

around the country with power utilities

with uh public safety uh with

construction companies. And what I'm

doing right now is using my keyboard to

simply fly here. Uh for those of you

that know the area, this is San Mateo.

And perhaps if I kind of zoom in here,

we may see a very faint render of the uh

SF skyline. Although the uh the fog city

is always there. So we might just say

hello to um the uh SFO airport here.

Some planes are launching. If anyone has

a plane tracker app, they can sort of

see this is uh realtime stuff there. So

what we have been building is the full

autonomy stack behind like how does a

vehicle operate autonomously? How does

the cloud system operate here? How do

the cloud servers work? uh and where

does the uh different levels of

intelligence and automation needs to

happen for us to make this happen

robustly such that when I'm here and

saying hey oh there's an incident

happening here I need to go respond I

could at the same time go back here and

say hey what about that other thing

that's happening on the other side of

the country what if we launch that

instead so while that's happening I'm

now going to launch something in

Colorado

so this is a imagine some sort of fire

incident has happen on your on the power

lines and we wanted to go out and

inspect those things. So, a dock is now

opening up in Colorado while the first

drone is still flying safely. Um, and

I'm just going to uh let it do all the

safety checks that it needs to do before

the launch. And this is all happening in

sort of conference Wi-Fi traffic. So,

you can imagine I can shut down my

laptop right now and everything needs to

safely happen behind the scenes.

Um, so I'm going to do this while the

first part is still happening.

Let me let's go back to the first drone.

Might give it another set of

instruction. Let's have a look around

here on the first drone. The second

drone has sort of kick started off and

maybe there's some cars sort of going

around there um that we can perhaps

track. So, let's track and what's this

car doing here? All right. So, we're now

tracking this car. Maybe this is a

runaway car that we needed to follow.

And I'm hands off right now. This is uh

the autonomous system kind of taking

over um through the various interactions

that I've given uh for it to do. And at

the same time while this is happening we

can do the final thing which is yet

another sort of drone in the system uh

back at our HQ

and we can say hey let's why don't we

run a third drone.

So what we're building towards is how do

we enable autonomy at scale where

traditionally how it started off

historically 15 years ago that you might

have a drone at home. It's a hobbyist

drone and you might play around with it.

You will tinker with it. You will work

with the controlling software. And then

about 10 years ago drones started to

become a lot more available and they

started to become a tool. A lot of

industries out there started to use

them. They will carry them with them in

the truck, go out there, deploy it. And

the next era that we're working towards

is drones as infrastructure. Imagine

these systems. And for the sake of this

conference, I'm going to call them

agents. These are physical emboded

agents that are kind of available at all

times at anywhere for the different sort

of use cases that we're interested in

that can automatically uh uh launch,

execute, and do their tasks. And what is

the minimum amount of autonomy that

needs to be begged in? And what does the

future interface look like today?

Everyone needs to be a dedicated pilot.

I go went through a certification

exercise. I need to think about the

safety standards here. But you can

imagine in a few years time the safety

is going to be determined by the

autonomous system and the interface

becomes really high level. It could be a

little slack bot that says, "Hey,

something's happened here. Why don't we

go send a drone?" And you might not even

know that the drone launched.

So, while this is happening, I'm going

to tell all the drones to pause and

return to doc.

So, we'll write this. They're all going

to start returning to doc. And while

that's happening, I'm going to start on

the presentation.

So, what I've shown you here is not just

concept. These are literally systems

that are being used in production. Uh as

was mentioned we are the largest

manufacturer of drones in the US and we

want to give people superpowers uh

through this technology.

Let's take a quick look at what some of

the people are doing here. This is um in

the northeast coast of the US uh where

our client has set up uh the system next

to a power station and they kind of used

this to do normal patrols and as they

flew around they found that some of the

there was a the pole was burning from

the inside and it was really starting to

show up here and this could have fallen

at any time and created a fire risk that

they would have otherwise not caught

without having to send people there

which itself is quite uh expensive.

Moving on to the sort of next use case

is on the public safety side. This is

San Francisco. Uh we work very closely

with SFPD. Um normally when a car gets

stolen, you'll see a high-speed chase

happening in the city. Quite dangerous.

But what if you could deploy a drone

instead where the uh people in the car

don't even know that there's a drone

following them. Here's a person who has

stolen the car on the right hand side

and they're about to change their

license plates uh on that. So they go

here, get out their tools. They come

back, luckily they point the license

plate up so the drone can see it and we

know exactly what they're doing. But

they're now replacing the plate uh in

the car uh with a new one. And all this

time they don't know that there's

they're being chased. The the the how

they behave in the uh public, how they

behave out there is very different and

it's a lot safer uh in how the

operations are done. They're now going

to go ahead and tint the windows. uh but

this allows the police to like

strategically position themselves in the

most safest way form to intervene at the

right time rather than doing a

high-speed chase outside.

So these are being used across many

different industries

and we are really starting to treat this

as infrastructure that can operate day

in day out nighttime rain sunshine.

We've got a few of these docs deployed

in Alaska, so very cold weathers. Few of

these docks deployed in uh Texas, so

extreme heat. And these need to be

reliable down to 99.9999%

uh where we do our sort of simulation

and testing to be able to prove that.

And today, uh we have about 16 million

people living within 2 mi radius of this

infrastructure that the uh um public

safety, the power companies can kind of

use this technology to be able to

respond to such incidents without having

to travel there.

So, what really happens when this

happens at scale? Can I get a hands up

of people that have flown a drone

before? A couple of hands up. When I

started to fly it, it was like kind It

took me a few hours to like really

figure it out. Then I put on the FPV,

that was even more tricky, but it felt

good to get get that expertise up, but

it's kind of like a skill that you

develop and you develop that skill over

time and you say, "Okay, for each

skilled person, we're going to put them

next to a drone and they're going to

start working." But now you want more of

these. So, uh, police companies,

infrastructure companies need to start

hiring these people more. And at some

stage, this really starts to break down.

The more 911 calls come that come in,

the more alerts that can come in, it

doesn't really scale. So, we kind of

rethinking as to what this means in

terms of this like initial firsterson

view engagement with these systems to

how do we convert this to a more

strategic uh, multi- aent view that you

can kind of command the entire fleet

with an objective in mind without having

to worry about the flight. Let me just

go back and see if that was all working

well. Great. They all landed. I am still

pleased when that happens successfully.

Although it's meant to happen all the

time.

All right, let's go back to this.

Oh,

all right. We're just going to carry on.

So, what does that mean when we start to

launch different things? You saw me

launch them. I was still kind of

thinking about it. Okay, I need to

launch this one, that one, that one. And

even myself in this like uh who's used

to this, I'm going to have some

cognitive challenges where our vision is

to be able to launch uh many many of

them uh in a potentially unsupervised

way. So what sort of commands these

things? How do we get it to like just

hey get out there, launch, search in

this area, find a missing person or look

out for this type of car and hold your

position there.

And in order to do that, we really need

to think about how do we get like really

like the many nines of reliability that

we need in autonomous flight. And that's

where we get an edge in the industry

because at Sky we're sort of controlling

the hardware, the software, the cloud,

the user interface to be able to manage

all that and specifically the autonomy.

uh allowing us to think about how our

underlying vision system should work to

see the environment to behave in the

environment correctly whether it's at

high altitudes whether it's cloudy

whether it's at high speeds how do we

deal in the rain on the bottom left

we're showing how do we navigate in

cities how do we plan large scale and be

able to do that and for anyone that has

worked with any sort of GPS device in

the city even our phones they kind of

suck uh so how do we robustly do that

and how do we also like do tracking when

there's a lot of occlusion

These are the all the different places

where we're thinking about how to train

AI systems quote unquote models. The

word model itself is uh has different

meanings in different places and I'll

discuss a little bit on what that means

for us.

But in order for us to you like really

harness this is we are learning on the

go. This is a learning flywheel that

we're getting out there, we're

collecting data, we're operating, and

we're coming back and doing that so that

each flight we can log the data. Uh kind

of like Google Street View where we need

to think about sanitizing that data,

make sure there's no private information

left there. Uh and make sure we don't

sort of like uh uh the customers know

exactly what they're sharing with us.

But if we can once we do that, we have

access to a huge amounts of data that we

can kind of learn from every single time

we instructed this but the drone did

this. every single time. We thought this

was going to happen, but this happened.

That can come back to our uh learning

agents, our reinforcement learning

ecosystems to be able to retrain,

evaluate, and send it back out there and

kind of continue that flywheel that

allows us to uh have that robust

framework.

And the other added advantage that we

have is it's not just about having

autonomy on the drone. As I mentioned

earlier, we have the luxury now to have

autonomy on the edge device, but also

have autonomy on the cloud. what I was

showing you earlier, all that video

feed, all the telemetry that's going

through a cloud server. We could set up

GPUs and we could set up inference

engines there to be able to have that

heavier lifting, maybe that longerterm

planning there, whereas the immediate

autonomous actions happen on the drone

and kind of always thinking about the

trade-off that we need to make uh to

make that successful.

One thing is true, however, that uh once

you do start thinking about having your

agents from the cloud, this the amount

of data coming to the cloud really

matters. Uh here's us sort of investing

in how we think about enabling the best

um uh video quality coming up through to

the servers uh in uh low lower low sort

of bandwidth areas. Uh being able to

sort of optimize that being able to un

uh um encode that information into

smaller sizes and be able to decode into

something clear that allows us to have

much higher quality exact same sort of

network conditions here. uh and it's the

some of the areas of investments that we

make so that we can have this more

cloud-based infrastructure uh to be able

to manage this at scale.

So once we do that I want to sort of

explore at a high level some of the uh

models that we um uh have in our system

uh that allows us to orchestrate all of

this.

Firstly a model this is a section about

world models. I want to sort of talk

about world models from a context of

maps. Not too dissimilar to how uh Whimo

works. They have a map of the world and

they kind of navigate in that world.

They do local perception, but also think

about global planning. If I want to go

from part A of the city to part B, I

can't just like keep hitting every

building and kind of navigating around

them. I need to think about what's the

optimal path uh along the way. So, we

start off with a lot of prior

information. uh um and we merge that

with uh not just sort of building data,

but maybe there's uh vector data such as

where the power lines are, where the

roads are, if you want different

behavior in these areas. And we think

about how to combine these resources to

ultimately build a map that we can plan

and navigate around. And the drone has

knowledge of this map uh at all times to

be able to go around that.

But like any map, maps can go out of

date. Uh luckily we have so many eyes in

the sky to think about how to maintain

and update these maps along the way. On

the left hand side uh we are rendering

our knowledge of the world in the points

onto our video feed. However, that

doesn't line up perfectly everywhere. Uh

there's some sections here that um if I

sort of zoom out here, there's some

sections here that a new construction

site had set up uh that we did not know

about. Uh however as a drone now starts

to fly and not this is not just one

drone but your fleet of drone they're

now observing these things that we can

feed back into our uh map syncing

process that can come back land give the

data and now once uh in the next

iteration all the drones in the fleet

have this most updated map of the world

that they can uh do all the planning in.

A second type of model is uh perhaps

today a more conventional sort of uh

machine learning uh inference model uh

which is the ability to uh be able to

track understand objects in the scene

but be able to track them be able to

track them behind occlusions. So the uh

this implicit representation behind the

scene is some sort of world

representation of the object that hey

it's gone behind this building and it

might come out on the other side. So I

should navigate myself so I can kind of

follow it there or I should move myself

in a different direction to be able to

do that. Maybe 5 years ago this would be

more done uh in a more conventional way.

You have to like manually think about

how to move. But now you can think about

uh more reinforcement learning style

techniques or more learned end to-end

approaches that can really help out here

without you having to engineer all the

edge cases uh that can go in. And then

obviously like how do we do this

robustly rain, snow, day, night um uh

using um vision only.

One other thing that we're sort of doing

here on the tracking side is we have

some minimal set of tracking that's

happened on device on the edge but we

can do some higher level tracking that

happens on the cloud. So perhaps it can

reason more about your entire map.

Perhaps you can reason more about use

heavier models. Use VLMs uh uh with

lower which have u which don't respond

as quickly uh at a rate of like let's

say 7 to 10 hertz but can give you

feedback at a 1 to two second uh latency

but that's good enough for us to make uh

broad decisions about um where to move.

For a lot of our um infrastructure

customers uh we're doing a lot of

semantic reasoning. So what is there in

the scene? Uh here is an illustration of

us thinking about um uh utility poles

and we want to uh when a instruction

comes in hey go look at this line

there's something gone wrong the drone

needs to go there it needs to understand

the scene then it needs to take actions

within that scene and we're constantly

looking at how to build these primitives

these tools ultimately uh that uh we

today code but uh at any time an agent

can access these tools to better

understand uh what the drone is seeing

and what it could do about

So an example of uh the agentic sort of

system in action is a visual language

model on the top left side. The user

here is typing in look for a white jeep

uh and it's doing a sort of find and

follow. It in it uh accesses a drone API

to command a certain sort of trajectory

that it should take. While that is

happening the detection head is kind of

uh the V uh the VLM is kind of running

here to say hey what's what in the scene

am I looking for? it finds something,

then it has access to the tools that

allow the drone to track and follow. And

that's without any specific coding of

that law rules, but instead uh having a

more sort of agentic uh giving the

agents the all the tools that it needs

to be able to um understand the drone

state and make decisions given the uh

information and the context that's

available there.

And then there's uh always the sort of

long-term vision uh that's often there

in the self-driving community here right

now or any sort of robotic system. What

if uh we could just give it raw sensor

data and out comes uh the perfect uh

results. Uh perhaps that's the actuation

that happens. Perhaps it's where the

drone is pointing. Perhaps it's where

the drone goes. And we're definitely

sort of doing a lot of testing and

triing with reinforcement learning on

what that looks like if we have multiple

uh instantiations of uh this um

behavior. Does it get to the right um

end spectrum? The main consideration

whenever we work with a physical system

is that you're often looking at really

high volumes of reliability as I was

saying many nines of reliability. uh and

the um doing a completely end to-end

system uh does have its challenges in

the sense that the observability of

what's going wrong and the guarantees on

reliability is very difficult today. So

while it's a direction that we're

continuously taking and exploring, it's

kind of figuring out which segments of

your end to-end chunk need to move to a

more uh um like sort of world model

representation of it. Uh, and it's

specifically the things where we always

find ourselves, okay, I need to

handgineer this. I need to code this in

specifically. I need to look at these

rules. Like search and rescue is one of

those things like, oh, go look for these

trees, but if you don't find the trees,

look under here or then turn on thermal

or but then if you didn't find it here,

look there. We're trying to get away

from having to have all these if

statements and the code and these

branching strategies and kind of let the

agent have some highle tools uh to be

able to instruct these very high level

commands uh for the drone.

So um I talked about a quadcopter today,

but we're kind of doing this similar

with a much smaller from form factor

quadcopter and we're now starting to

look at this what this looks like from a

fixed wing as well. all in the world of

infrastructure. Uh so they can be

launched from anywhere, recovered from

anywhere and ultimately the sweet spot

where we can kind of start to very

quickly build on the cloud uh to

orchestrate these things. And the way we

thinking about it is allowing these

systems to have basic APIs of

interactivity that the cloud agents can

come in and uh tap into and make

decisions on and ultimately allow for

very high level thinking when we uh uh

work with these drones.

So like many talks here we are hiring uh

as as I said we have full stack sort of

uh we do the end toend thing hardware

software autonomy full stack front end

back end uh wireless networking

everything all the technologies on our

mobile phones we're now making them fly

um so uh come say hi or look at our

website uh would love to talk more thank