What does it take to build a personal, local, private AI Agent that augments you deeply?

[upbeat music] So, uh, why do I-- Well, first of all, who am I?

Uh. [laughs] Do you know this thing called PyTorch? Um, a lot of people in AI used to know it, but now a lot of people just use high-level APIs and don't know what's, uh, powering things underneath.

Uh, but, like, PyTorch is, uh, the software probably powering your AI APIs. Um, so I work on it. I co-founded the project. And it's, uh, it's a big project that is, uh, majority funded by Meta, where I work at.

Um, and so I'm talking about, uh... So I'm not talking about Llama at all. [chuckles] I work on Llama a little bit, but, uh, unfortunately, uh, I am not in charge of Llama to try to sneak in some secrets for you guys.

Um, I'm not gonna tell you when the next Llama is gonna come or anything like that. So, uh, why am I thinking about personal local agents? Well, the-- A-as AI kinda started becoming more and more useful, one of the things that saved my time the most every single day was Swix's AI News.

And it's basically, like, I have to keep up to date with all of what's going on in AI. That's my job. And now, instead of basically spending three, four, five hours a day, um, looking at a bunch of sources, like, the AI was aggregating a bunch of news for you, and I thought that was, like, one of

the s- like, one of the first applications that I thought was, like, mind-blowingly, uh, personally, like, effective for my own productivity. And that's when I started going into like, "Hey, I'm gonna, like, augment, uh, AI within my day-to-day in a deeper way."

Um, that's not an agent, though. AI News is more like a, an aggregator. Uh, but that's how it kinda started. The other thing is I also work on robotics.

Um, and robots are essentially agents. They act in the world. So, um, I, I... My goal is to build home robots so that I don't need to do any errands.

Um, and so, um, as part of that journey as well, I've been, like, kind of getting into like, okay, how do I get into understanding AI agents more deeply?

Um, the key takeaway I'm gonna really, like, drill down, uh, to you today is agents, like, especially personal agents, have so much possible agency in taking actions on your behalf and stuff, and they have so much of your life context to actually be useful to you that you're better off keeping them local and private.

And I'm gonna try to, like, sketch out a plan on how to do it, but I don't think I have a complete solution either. Um, so first, like, agent.

What is an agent? Like, and why did I say Swix's AI News is not an agent? Well, an agent is something that can act in the world. Like, an agent is something that has agency.

It can actually, like, take an action in the world. Uh, anything that can only get context and do things, but then eventually can't act in the world is not an agent.

That's how I think about it. And what I think is, like, a highly intelligent agent without the right context is as good as a bag of rocks. It's, like, really useless.

I'll give you a couple of examples very quickly. Uh, let's just say I build a personal agent. It has, uh, access to my Gmail, my WhatsApp, my calendar. And I was like, "Did, did I get my prescription renewed?"

And it's like, "No, not yet." And, like, it's totally lying, uh, except it didn't know because, like, I got the text from CVS on my iMessage, and it didn't have access to, uh, that source.

And it was doing the best it can with the information it has, but if it didn't have the context, like, it's not gonna, like, know how to do better.

Uh, similarly, I mean, you can make up like a hundred examples like this where, like, you have access to one bank account, but, like, it, your money came into a Venmo, and you're like, uh, the agent lied to you.

What happens is, like, a personal agent that doesn't have the right context, it's largely gonna be irritating to use. It's like you don't know when it is useful and when it is not useful, so it's essentially not useful.

Um, like, even when it gives you some answer, you're like, "Hmm, is this actually right? I'm gonna have to go dig in." Right? So unless it hits a certain level of, like, reliability and predictability that you know it is right, uh, it's not gonna be actually useful to you.

Um, so now, like, why am I talking about personal agents specifically? Um, and how do you, like, how do you get all this context to the agent? So let's just say you have, like, your OpenAI API or some other API or some local LLM.

What is all the context in the world that is personal to you, and how do you give it to the agent? Well, like, the number one thing that you probably want to do is, like, just have variables, right?

You're just like, you can, uh, the... Your AI should see everything you see and listen to everything you hear. Um, and that is, like, obviously the best case of providing context to your AI agent, except, like, there's no battery life for any of these variable things, so that's not really practical.

Maybe one day when you have, like, crazy batteries, but that's not really gonna work. The other thing it could be, like, okay, like, most of my life is on my phone, uh, in ways that I care about from, like, an agent perspective.

Uh, what about just, like, uh, running an agent on my phone? It's running in the background, and it's just, like, always, like, watching my screen or something. Well-

You know, that's where, like, Apple kicks you because, you know, they don't let you run a bunch of stuff and, and, like, on your phone asynchronously. If you-- even if you do, they have a lot of restrictions.

So, like, the ecosystem's kind of, like, kill you and not allowing you to do that. And unfortunately, I use, like, Apple. Um, so that's, that's out. [chuckles] So the next one is like...

Okay, actually, like, the thing that I found, like, relatively useful is, like, if you use, like, Apple in your daily life, uh, you can actually get a Mac Mini and, like, just put it somewhere in your home, connect it to the internet, and you can run your agents asynchronously.

There's no battery life issues. You can just log into all your services on your, like, Mac Mini. And, um, it also connects with all the Android ecosystems because Android is actually open.

Um, [chuckles] so I work at neither of these companies, so I can say whatever I want. [audience laughing]

Um, so I, I think that's, like, what I think is a feasible, um, um, device to use to, like, run your AI agent right now. Um, the next thing I wanna talk about is, like, okay, why are you talking about local and private?

Why can't you just, like, run this in the cloud, like, just subscribe to one of the large tech companies', uh, agent services and run your life out of it?

Well, I wanna give you, like, a few points here. Uh, first is, um, I wanna talk about how this is different from you using other digital services. And I think it is different meaningfully, and I think it's also easy to understand.

So let's just think about, like, a lot of you in this room probably use, like, a cloud email service that is free for all of your life. All your taxes are going in there, like, you know, everything personal is going in there.

Why do you trust it? The reason I think you trust it, at least the reason I trust it, is because it has a very simple mental model on how it will act on your behalf or how will it'll act in general.

Email in, reply out. It's not, um, go-- it's basically not trying to do something sneaky under you that is unpredictable. It's a very simple mental model. Your trust of that service is correlated with whether you understand how it behaves on your behalf.

So imagine tomorrow, if some unknown email service that you've been using forever says, "Oh, you know, for some of your emails that I have confidence in, I can auto-reply on your behalf."

And you're like, "Okay, well, first of all, that might be true, but what is the worst case action you can take?" Maybe you'll, like, reply to my boss, like, something nasty, [chuckles] and, like, I don't want that to happen.

And, like, that's, like, once the action space becomes powerful enough and unpredictable enough, you get uncomfortable with using a service that you're not fully in control. Um, and it can get, like, worse, right?

Like, uh, companies have to monetize in a million ways. And so what if, like, you're using, like, like, an online service and they suddenly are like, "Oh, you know, every time you ask for a shopping query, we're gonna, like, start making the agent only buy from, like, stuff that gives us kickbacks or something."

So, like, I think, like, your personal agent is so personal to you and so intimate that I feel like ultimately you want to be in control, uh, on many aspects that you might not have control on eventually when you have to, like, trust, uh, an online service.

So that's, like, one of the biggest reasons, like, why I feel I wanna build a personal agent that's local, uh, to myself. The second is decentralization. Like, I mean, you already see all these ecosystems that are walled gardens and, like, fighting with each other and don't allow each other to interoperate in various ways.

And if you build one of, like, your personal life, uh, your personal agent around one ecosystem, like, is that something-- Like, it, it works fine for compartmentalized things like maps and email and various things, but, like, is that something that you wanna really subscribe into for, like, an agent that can take so many different kinds of actions

on your behalf, uh, in your day-to-day life? That's, like, the other reason I, I, I feel like you should try to-- we as a world should try to get to, like, local personalized agents as the norm.

Um, and the third one is, um, for various reasons. Okay, this is what I had called, uh, this is what I call, um, are you going to be punished for your thought crimes, right?

Like, okay, you have a thought, and it is not a good thought, and, like, you know, should you be punished for it? And usually, like, the answer is no.

Now, if you have a personal agent that is effectively augmenting you in such an intimate, personal way, you might be asking it stuff that you generally wouldn't say out loud ever.

Um, and in those cases, like, do you really want to take the risk of, like, like, putting it out into, like, some provider? Because, like, you know, you, you can act-ask Perplexity, like, enterprise-grade cloud API contracts that, like, are, like, um, enterprise grade, not consumer grade, where they, like, get sloppy.

Even they have to, like, do a bunch of, like, legally mandated logging and then safety checks and stuff. So there is a possibility that, like, you might or might not wanna take a risk on.

But for me, I'm like, I don't want to ever get into a scenario where, like, I will be, like, prosecuted or persecuted for my thought crimes. [chuckles] And, like, that I think is, like, another really powerful argument for myself at least to focus on, like, local agents for my most personal, um, Uh, augmentation.

So now coming to, um... Well, I hope you're convinced that yes, yes, we actually like, if you're gonna build a personal AI agent, it has to be local and private.

Uh, well, okay, what's the problem? Well, let's go to the technical challenges first. First, like, okay, you gotta run this stuff, right? Um, there are, like, great open source projects to run a bunch of local models that are one of the key components of these agents.

Uh, vLLM and SGLang are pretty great. Uh, they're both built on top of PyTorch. [laughs]

So effectively, uh, this is one time, like, we wrote a bug in PyTorch and, um, a bunch of us, uh, had a Tesla car, and Tesla uses PyTorch. [laughs] And we were like, "Man," like, "this is so scary because, like, are we writing bugs on ourselves?" [laughs]

That's an essay. Um, it was totally fine. The bug was not that bad. [laughs]

Um, so yeah, vLLM, SGLang are great. Um, but local model inference is still, as of today, slow and limited. It's not as fast as, like, you know, if you just use, like, a cloud service.

Even if you spend, like, a b- like, enough money on a beefy machine, I think that's also rapidly changing. Like, for example, locally, if you're using, like, a 20 billion or distilled model of some sort, it actually runs pretty fast.

Uh, but if you want to use, like, the latest R1, like, full unquantized, then it runs, like, super-duper damn slow. Um, I think this, like, is in a state of, like, it will fix itself, so you, you probably wouldn't get to run the latest and greatest.

Um, and I think, like, the challenges are not so much the technical and infrastructural challenges. Like, they will kind of get to a place where they're fine. I think there's some challenges around, like, both the research and product that, um, people need to think a bit more about.

I think there's a bit of a gap, and this is just an open challenge for this room for all of you AI engineers. Um, one is, like, the open multimodal models are good but not great.

I mean, they're not great in a couple areas. One is, like, just computer use. Even the closed models, like in the latest and greatest APIs that you can just pay money for, they're not that great for computer use.

They break all the time. So that needs to definitely get, like, into a better state. The other thing, like, I noticed is, like, if I ask a model to do shopping for me, from clothes to shoes to furniture to whatever, it'll basically give me the most boring shit, right? [laughs]

Like, it's like the basic stuff. And if I ask it... If I'm like, "Look, I'll tell you my tastes," and my tastes can get very f- like, specific and fine-grained.

Like, the more specific I get, the more, like, bullshit it gives me. [laughs] Like, it's like, it's like the same, "Oh, you asked for, like, a red velvet sofa with oak wooden legs.

Uh, here's a green sofa that has velvet, um, and it doesn't have, like, oak wooden legs." You know, like, they're not very good at identifying actually visually what you're asking for.

They mostly rely on, like, a bunch of text, uh, matching. Um, the other thing, uh, you will notice, and this is a big one, is we don't have good catastrophic action classifiers.

What I'm-- What do I mean by, uh, catastrophic actions is

there's many actions an agent can take. A lot of them are reversible or harmless. Like, even if it takes the action and that's not the action you wanted it to take, it's like whatever.

Oh, it had to go to, like, that particular Wikipedia link, but it went to this other one? Okay. Big deal. Whatever. It'll just backtrack and go. But there's some actions that are actually catastrophic.

For example, you ask it to go purchase, like, uh, a renewal of your Tide Pods, and then it goes and, like, purchases a Tesla car. You know? [laughs] This is not the best thing for you to do.

Uh, and some of these are called catastrophic actions. And I don't think there's a lot... Like, there's some open research around, like, how to really get agents to get good at identifying catastrophic actions before taking them and then maybe, like, notifying the users instead.

Uh, but there's not enough. And so if you wanna really trust your agents, personal or in cloud, I think we've gotta get a bit better at these things. Um, so that's, like, a big one.

And I think open source voice mode is barely there. Uh, I feel like when I need a personal local agent, I definitely want voice mode because sometimes I wanna talk to it, uh, and not actually type out everything I wanna say.

Um, but still, why am I bullish about this whole thing? I am, uh, because one, I see open models are actually, like, compounding in intelligence, like, faster than closed models, like, based on how many resources are being put on them.

Like, what do I mean by that? Like, OpenAI is only improving their own model. Anthropic is only improving their own model with all the billions they have or whatever.

But open models are improving themselves, like, in coordination across board. Um, and, you know, people didn't really b- believe it until Llama came out, and they didn't, they didn't really believe it until Mistral came out, and then they didn't really believe it until Grok came out.

Uh, and then they didn't really believe it until DeepSeek came out. Like, basically, like, people are like, "Oh, you know, like, open models, you know, will not really win," but I think they are.

Like, basically, in open source, like, I've worked in open source, like, all of my life, um, there's a starting coordination problem. Like, initially, you don't have enough of a critical mass to coordinate with each other.

But once you have a critical coordinated mass, open source kind of starts winning in- Like an unprecedented way, and you see that with Linux, you see that with a bunch of projects.

Uh, so I am pretty bullish that open models will actually start getting, like, better than closed models, um, um, like per dollar of in- investment into open models. Um,

and that's what I said. Well, okay, I have some plugs. Uh, this is like, uh, GR.inc from, uh, my friend, uh, Ross Taylor, who worked on this model called Galactica, which got a lot of s- like, um, criticism when it was released, uh, out of Meta.

It was this open science model before ChatGPT released. Now, like doing science with like LLMs is pretty common, but like [chuckles] they got a lot of shit when they released.

Uh, and he like quit, uh, he, it like unreleased Galactica, and he quit doing like, uh, a bunch of stuff publicly. But then, like he's working on like plugging the, the reasoning gap between open models and closed models, and they released a bunch of open reasoning data, uh, that will help.

So just a nice quick plug. The other quick plug is I work on PyTorch. PyTorch is working on enabling local agents, especially the technical challenges that I talked about, uh, and we're hiring.

So if you are more than an AI engineer, if you're an AI engineer who's also like a systems engineer, then like PyTorch is hiring. Um- [laughs]

Well, [laughs] that's what we got. [laughs] The other thing obviously is I welcome you all to come to Llama Con, which is happening on April twenty-ninth, and save the date. It's gonna be very exciting.

Lots of Llama stuff will happen there. That's it. Uh, I think it's in California. I, um, I actually didn't look it up. [laughs] [upbeat music]