# What does it take to build a personal, local, private AI Agent that augments you deeply?

Soumith Chintala, Meta PyTorch

AI Engineer Summit 2025

[Talk page](https://ai.engineer/talks/jMoAaZP_Kkw-what-does-it-take-build-personal-local) · [Video](https://www.youtube.com/watch?v=jMoAaZP_Kkw)

## What does it take to build a personal, local, private AI Agent that augments you deeply?

A useful personal agent needs more than intelligence: it needs the right context, dependable actions, and an owner who controls what it knows and does.

### From keeping up with AI to delegating everyday work

[0:21](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=21s)

How do you keep up with AI when following its developments is part of your job? For Soumith Chintala, that problem comes with working on the infrastructure beneath many AI applications. He co-founded PyTorch and works at Meta, which he describes as the project's majority funder. Although he also works a little on **Llama**, this talk is about personal agents, not model-release announcements.

Chintala reports previously spending three to five hours a day checking AI sources for work. [AI News](https://news.smol.ai/), swyx's news aggregator, made that daily information gathering much easier. Its usefulness prompted a larger question: where else could AI become a deeper part of everyday life?

![Slide titled “Why I'm thinking about Personal, Local Agents?” above an AI News webpage screenshot.](https://ai.engineer/api/talk-essay-media/1b16913ced26a30ba1c79c98b24860d797848bd5737ad40d73080a60738721a3)

AI News as an introduction to personal, local agents.

Aggregation is only the starting point. AI News gathers information; an agent can act on it. Chintala's other motivation comes from robotics: he wants home robots that handle errands. Bringing those two ambitions together creates a demanding design problem. A deeply useful personal agent would hold extensive context about your life and have authority to act on your behalf. **That combination makes local ownership and privacy central requirements.** What follows is a sketch of how to get there, not a finished system.

### Intelligence cannot recover context it never received

[3:01](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=181s)

Chintala defines an agent by its ability to act in the world. Merely collecting and processing context does not meet that definition. But the ability to act is not sufficient either: a highly intelligent agent without the right context can still be useless.

Suppose your agent can read Gmail, WhatsApp, and your calendar. You ask whether your prescription was renewed, and it answers no. The renewal confirmation actually arrived as a CVS text in iMessage—a source the agent cannot access. Its answer looks like a lie, but the underlying failure is missing evidence. A similar failure occurs when an agent sees one bank account but misses money received through Venmo.

The practical damage extends beyond one wrong answer. If you cannot predict when the agent has enough information, you have to investigate its answers yourself. **Delegation only saves effort when reliability is predictable enough to stop routinely rechecking the work.** More reasoning over an incomplete view does not remove that dependency on context.

### A practical place to gather context

[4:46](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=286s)

Context collection is a separate problem from choosing an inference provider. Whether the agent uses an OpenAI API, another service, or a local model, it still needs access to the parts of your life relevant to its tasks. A wearable could, in principle, see what you see and hear what you hear. Chintala considers battery life the obstacle to making that continuous capture practical.

A phone is the next obvious candidate because so much personal activity already happens there. But an agent continuously running in the background and watching the screen conflicts with the execution restrictions Chintala encounters in Apple's ecosystem. The limitation is restricted background execution, rather than a blanket inability to perform asynchronous work.

His practical alternative is an internet-connected Mac mini at home:

1. Keep the computer plugged in and online.
2. Log into the personal services the agent needs.
3. Run the agent asynchronously on that machine.

This removes the wearable's battery constraint and provides a desktop environment for accessing services. Chintala also describes this approach as connecting with Android ecosystems, which he considers more open. It is a feasible host for an agent, not a claim that every personal data source becomes automatically accessible.

![Projected slide titled “Universal context*” showing a small silver desktop computer and the label “Desktop,” with the speaker below.](https://ai.engineer/api/talk-essay-media/ba38c5ab7811ef258bce3d9634c1773442a8e5cf8d5b2810832eb8b2473e26d8)

A desktop computer under “Universal context*.”

### Trust changes when software acts for you

[7:13](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=433s)

Why not subscribe to a cloud agent and let it run your life? After all, cloud email already holds sensitive information, including tax documents. Chintala's distinction is behavioral predictability. Email has a simple mental model: messages arrive, and you send replies. You can understand what the service does on your behalf.

Now let the email service automatically reply whenever it feels confident. The question immediately becomes what its worst possible action might be. An offensive reply to your boss is consequential in a way that receiving or displaying a message is not. As the action space grows more powerful and less predictable, direct user control matters more.

Control also includes whose incentives guide a decision. Chintala imagines a shopping agent whose provider changes its behavior to buy only from vendors that pay kickbacks. The agent could still complete a purchase while no longer serving the user's preferences faithfully. Local ownership is his proposed way to retain control over an assistant intimate enough to make such decisions.

### Avoiding dependence across an entire life

[9:44](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=584s)

The second reason for local agents is decentralization. Walled gardens already restrict interoperability between services. Accepting an ecosystem's limits for a compartmentalized tool such as maps or email is one decision; placing an agent that takes many kinds of everyday actions inside that ecosystem is a much broader commitment. Chintala wants local, personalized agents to become the norm rather than another layer of dependence on a single provider.

The third reason concerns private deliberation. A deeply personal assistant may receive questions you would never say aloud. Chintala calls his concern “thought crimes”: the possibility of prosecution or persecution for thoughts shared with an intimate assistant.

He invokes enterprise cloud API contracts, mentions Perplexity, and contrasts enterprise arrangements with consumer services. His concern is that even stronger contracts can leave exposure through logging and safety checks, which he describes as legally mandated. That is his personal risk rationale; the talk does not establish a provider-specific legal obligation. For his most intimate augmentation, he does not want to take that risk, which brings him back to local and private operation.

### Running the model is only the first technical challenge

[12:20](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=740s)

The technical assessment belongs to the talk's February 2025 [AI Engineer Summit](https://ai.engineer/summit/2025) context. Chintala points to [vLLM](https://github.com/vllm-project/vllm) and [SGLang](https://github.com/sgl-project/sglang) as strong open-source projects for running models, both built on PyTorch.

That dependency prompts an infrastructure anecdote. After introducing a PyTorch bug, Chintala and colleagues who owned Teslas worried about the possibility of writing bugs that could affect themselves, because Tesla uses PyTorch. He immediately qualifies the story: things were fine, and the bug was not that bad. The anecdote illustrates how close foundational software can sit to the systems its developers depend on.

At recording time, Chintala describes local inference as slower and more limited than cloud services, even with an expensive machine. His qualitative comparison is between a roughly 20-billion-parameter or distilled model that can run fairly fast locally and full, unquantized **[DeepSeek-R1](https://github.com/deepseek-ai/DeepSeek-R1)**, which he describes as very slow. He supplies no hardware configuration, task, or latency measurements for that comparison.

He expects the infrastructure to improve, while accepting that local users may not always run the newest, most capable model. The harder remaining questions are research and product questions: can the model operate software dependably, understand your preferences, and recognize when an action is too consequential to take?

### Understanding a screen—and a particular sofa

[14:12](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=852s)

Chintala finds open multimodal models promising but insufficient for dependable computer use. Even leading paid closed models, in his experience, frequently break while operating software. Better access to applications does not help much if the agent cannot reliably navigate them.

Shopping reveals a different weakness. Recommendations for clothes, shoes, and furniture tend toward generic choices. Giving increasingly precise preferences does not necessarily produce increasingly precise matches. His example is a request for a red velvet sofa with oak wooden legs that returns a green velvet sofa without oak legs.

| Attribute | Requested | Returned in the example |
|---|---|---|
| Color | Red | Green |
| Upholstery | Velvet | Velvet |
| Legs | Oak wood | Not oak wood |

Matching one attribute is not enough to satisfy the request. Chintala's diagnosis is that the models rely heavily on text matching rather than actually identifying the requested visual properties. A personal shopping agent needs to preserve the conjunction of preferences, not merely retrieve something nearby in description.

### Recognize dangerous actions before taking them

[15:28](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=928s)

Another missing component is a dependable **catastrophic action classifier**. Not all mistakes deserve the same response:

- **Recoverable navigation:** The agent follows the wrong Wikipedia link. It can backtrack and open the intended page.
- **Consequential purchase:** The user asks to replenish Tide Pods, but the agent buys a Tesla. The mistake has escaped the realm of harmless exploration.

The distinction is about what happens when the action reaches the world, not just whether the model selected the wrong next step.

The required mechanism is detection **before execution**: identify a potentially catastrophic action and, where appropriate, notify the user instead of taking it. Chintala acknowledges existing research but considers it insufficient for the trust people need. Local operation does not solve this problem by itself; both local and cloud agents need better judgment at the action boundary.

Voice is another practical gap. Chintala describes open-source voice mode as underdeveloped, yet considers it necessary for a personal assistant. Sometimes the natural interaction is to speak, not to type every request. A useful local agent needs that everyday interface alongside better perception and action safety.

### Why open models could compound faster

[17:05](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=1025s)

Despite those gaps, Chintala is optimistic about open models. He contrasts OpenAI and Anthropic improving their respective models with improvements shared across an open ecosystem. In his account, successive releases from Llama, **Mistral**, **Grok**, and **DeepSeek** have repeatedly challenged skepticism about what open models can achieve.

The proposed mechanism is coordination. Early in an open-source project's life, there may not be enough contributors to coordinate effectively. Once a critical mass exists, participants can build on one another's work, and progress compounds. Linux is his analogy for what becomes possible after that threshold. Chintala forecasts that open models will outperform closed models per dollar invested; he presents this as an expectation grounded in open-source coordination, not an investment-normalized benchmark.

### The work extends from reasoning data to systems engineering

[18:36](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=1116s)

A concrete example closes the research agenda: [General Reasoning](https://www.gr.inc/), associated with Ross Taylor. Taylor worked on **[Galactica](https://arxiv.org/abs/2211.09085)**, an open science language model released before ChatGPT. Chintala recalls the criticism surrounding its release and the subsequent retreat from its public rollout, contrasting that reception with the later acceptance of using language models for science. This should not be read as the disappearance of all model access: the [Galactica repository](https://github.com/paperswithcode/galai) still documents models and weights. Taylor's newer work, Chintala says, includes open reasoning data intended to narrow the reasoning gap between open and closed models.

The infrastructure work remains concrete too. Chintala says PyTorch is working to enable local agents and, at the time of the talk, is hiring engineers who combine AI expertise with systems engineering. He ends with an invitation to LlamaCon on April 29, 2025, promising more Llama developments. He tentatively places it in California while acknowledging that he has not checked the location. The closing invitations point toward two complementary contributions: improving the intelligence available to local agents and making that intelligence practical to run.

## Resources

- [AINews](https://news.smol.ai/) — Daily summaries of AI discussions across Discord, Reddit and X.
- [vLLM](https://github.com/vllm-project/vllm) — Open-source inference and serving engine for language models.
- [SGLang](https://github.com/sgl-project/sglang) — Serving framework for language and multimodal models.
- [DeepSeek-R1](https://github.com/deepseek-ai/DeepSeek-R1) — Official repository for the DeepSeek-R1 reasoning model family.
- [Galactica: A Large Language Model for Science](https://arxiv.org/abs/2211.09085) — Ross Taylor and collaborators' paper on language models trained for scientific knowledge and tasks.
- [GALACTICA model API](https://github.com/paperswithcode/galai) — Installation instructions, model sizes and inference examples for GALACTICA.
- [Apple background execution strategies](https://developer.apple.com/documentation/BackgroundTasks/choosing-background-strategies-for-your-app) — Apple's guidance on scheduling background work and handling runtime limits.

## Slides

### Who’s this guy?

The speaker introduces himself as a PyTorch co-founder and says he works on the project at Meta.

[0:23](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=23s) [Introduction slide with the PyTorch logo.](https://assets.ai.engineer/sites/aiecode2025/sha256/d83680a68bf924e70d151215f0af23fbff02f401a185e10be8723968571ea8b9.webp)

Who’s this guy?

PyTorch

2

AGENTS AT WORK

### Why I’m thinking about personal, local agents

AI News aggregates updates from AI communities, saving the speaker time he used to spend checking multiple sources to keep up with AI.

[1:11](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=71s) [Slide showing the AI News aggregation service as a source of inspiration.](https://assets.ai.engineer/sites/aiecode2025/sha256/d28157b3b555c4bf75bd07fa5801877898ed938d4507d31e3e04fa410a8e4849.webp)

Why I’m thinking about Personal, Local Agents?

3

AI News

We summarize top AI discords + AI reddits + AI X/Twitters, and send you a roundup each day! See archive for examples.

"Highest-leverage 45 mins I spend everyday" - Soumith

"best AI newsletter atm" and "I'm not sure that enough people subscribe" - Andrej

"genuinely incredible" - Chris

"surprisingly decent" - Hamel

A smol service by @swyx and other Latent.Space friends!

You can pay for a customizable version here, and see a chunked/embedded search from Airtrain. Thanks to Pieter Levels for the Lex Fridman feature!

### Why I’m thinking about personal, local agents

The speaker’s goal of building home robots led him to study agents that act in the world. The slide shows robotic hand demonstrations including unstacking bowls and opening a bottle cap.

[2:10](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=130s) [Five images of a robotic hand handling bowls, a cup, a joystick, a bottle cap, and a book.](https://assets.ai.engineer/sites/aiecode2025/sha256/1a51cb3d2e574a9166ae8339cc3a1fac78155e7ccee343da9089a61ced259e45.webp)

Why I’m thinking about
Personal, Local Agents?

Bowl Unstacking
Cup Unstacking
Joystick Movement
Bottle Cap Opening
Book Opening

4

### Key takeaway

Personal agents can act on your behalf and need life context to be useful, so the speaker argues for keeping them local and private. He plans to sketch an approach, while acknowledging it is incomplete.

[2:39](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=159s) [Slide argues that personal agents should be kept local and private because they can act and need life context.](https://assets.ai.engineer/sites/aiecode2025/sha256/c87b45299883b8a12ea1c52a9d8845c1ade8060c515038f481b68093201f20ff.webp)

Key takeaway?

(Personal) Agents have so much possible agency, and need so much of your life context, that you’re better off keeping them local and private.

And this is how you do it.

5

### Agency

A controller connects a model and memory to tools and environments to get context and act. The speaker defines an agent by its ability to act in the world.

[3:06](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=186s) [A controller connects a model, database or memory, and tools or environments to receive requests, get context, and act.](https://assets.ai.engineer/sites/aiecode2025/sha256/931b0ff6de7fe010c29f1c0fb84c79a7bb45db7ce05d4817b7e7508ae3d44040.webp)

Agency

ask
Controller
Get context
Act
Model
Tools / Environments
Database / memory

AIE
AGENT ENGINEERING
HTTPS://AI.ENGINEER

### Agents need context

An intelligent agent needs the right context to be useful. In the speaker’s example, an agent misses a prescription update because it cannot access the text message.

[3:20](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=200s) [A statement compares an intelligent agent without the right context to a bag of rocks.](https://assets.ai.engineer/sites/aiecode2025/sha256/4689d82e49a748dc47d64f29b183639429ae34a582514d1b478e7f2ca90920e7.webp)

A highly intelligent agent without the right context is as useful as a bag of rocks

Interface text:
AIE
AGENT ENGINEERING
HTTPS://AI.ENGINEER

### Missing prescription context

The agent says the prescription has not been renewed, but a CVS text says it is ready for pickup. The agent has access to Gmail, WhatsApp and calendar, but not the iMessage containing the update.

[3:35](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=215s) [An agent says a prescription has not been renewed, while a separate message indicates it is ready for pickup.](https://assets.ai.engineer/sites/aiecode2025/sha256/aa7e4992a8c2be46adbb894e9c337498947c8c86432986af5de6571106f2215c.webp)

11

S: Did my prescription get renewed?
No, not yet!

Your prescription is ready to be picked up. Please pick it up before 11/21.

Interface text:
Sprint LTE
4:08 PM
75%
Messages
I-CVS
Details

### Reimbursement example

If an agent can see only one bank account, it may miss a reimbursement paid through Venmo while reporting that none was found.

[4:05](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=245s) [An agent says no reimbursement was found, with service icons and a Venmo logo below.](https://assets.ai.engineer/sites/aiecode2025/sha256/89c67cf2cb1576c0eca20ddc44f4f35d5d482feacd2a97ae3365ac6dec98676c.webp)

A
Did my work expenses from last month get reimbursed?

No, there's no reimbursement found.

venmo

12

Interface text:
AGENTS AT WORK

### Personal agents need context

Without enough context, a personal agent can give answers you have to check yourself, making it hard to know when to trust it.

[4:19](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=259s) [A slide states that a personal agent without nearly all of the user's context will be erratic, irritating and unproductive.](https://assets.ai.engineer/sites/aiecode2025/sha256/8e8569201950d31542e740f093626292efeb65a6eeae47eb7ef2d5498cab79ad.webp)

A personal agent without nearly all context in your life is going to be erratic, irritating and unproductive

### Personal

The talk turns to how an agent can receive context personal to its user.

[4:53](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=293s) [Section slide labeled 02 Personal.](https://assets.ai.engineer/sites/aiecode2025/sha256/19d87ae08babc03092778f9471d3a08d0b2389c58005f5ed1a8f6c9f39f2295e.webp)

02
Personal

### Universal context: Wearables

Wearables could let an agent see and hear what you do, but the speaker says battery life makes that impractical. The slide depicts a battery kicking a person.

[5:32](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=332s) [A battery character kicks a person beside the label Wearables.](https://assets.ai.engineer/sites/aiecode2025/sha256/321cd1111dae44179f6ae6c2de19c8b0ffcb85c4365a9ecf9f9d4843a16c6f14.webp)

Universal context*

Wearables

16

### Universal context: Phones

A phone agent could gather context by watching the screen in the background. The speaker says Apple’s restrictions make this impractical for him.

[5:41](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=341s) [Images of an iPhone and a Google phone beside the label Phones.](https://assets.ai.engineer/sites/aiecode2025/sha256/dafb63ea020cc24f0ea426367a6522e1e7909e93f1161fe5030d4e80c7832841.webp)

Universal context*

Phones

Phone screen: [illegible]
68°

17

### Universal context: Desktop

The speaker suggests keeping an internet-connected Mac mini at home, signed into your services, to run agents asynchronously without battery life concerns.

[6:26](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=386s) [A compact desktop computer beside the label Desktop.](https://assets.ai.engineer/sites/aiecode2025/sha256/82d8ff634540c06d7d946d21433a959a665cd7d4feeac0d09a1e7b08841cd0a4.webp)

Universal context*

Desktop

19

### Local & Private

The speaker introduces the question of why a personal AI agent should run locally and privately instead of through a cloud service.

[7:13](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=433s) [Section 03: Local & Private.](https://assets.ai.engineer/sites/aiecode2025/sha256/46d7afecf93c8e70a770f53dd4136e4623446978957e721f79b90d09862f77d0.webp)

03
Local & Private

Event signage: AGENTS AT WORK

### Why Local?

The speaker frames the case for a local agent around its action space, decentralization, and privacy and security, asking why someone might choose it over a cloud service.

[7:32](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=452s) [Why Local? Powerful action space, decentralization, and privacy and security.](https://assets.ai.engineer/sites/aiecode2025/sha256/ba593e3ca4e9619cd4aa5688a4b46eff0224e7bfe8375e02e915fe10de17b240.webp)

Why Local?

1. Powerful Action Space
2. Decentralization
3. Privacy & Security

21

### Why Local?

Why trust Gmail with personal information? The speaker says his trust in cloud email comes from a simple mental model of how it behaves.

[7:45](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=465s) [Why do you trust Gmail?](https://assets.ai.engineer/sites/aiecode2025/sha256/d47d598e97e6597bb809232d7209f9d045c4b810a27f95e0d925e038b6ed19ce.webp)

Why Local?
22

Why do you trust gmail?

AIE
AGENT ENGINEERING
HTTPS://AI.ENGINEER

### Trust in a cloud service

The speaker says he trusts cloud email because its behavior fits a simple mental model: email comes in and a reply goes out.

[7:54](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=474s) [An email enters Gmail and a reply comes out.](https://assets.ai.engineer/sites/aiecode2025/sha256/e8a41c8dad9a8a23b4bb468bffb22de2d830967a5c6d45a68bc440380ae5d64c.webp)

Trust in a cloud service
23

email in → [Gmail logo] → reply out

AIE
AGENT ENGINEERING
HTTPS://AI.ENGINEER

[8:03](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=483s) [A Gmail icon between arrows labeled email in and reply out.](https://assets.ai.engineer/sites/aiecode2025/sha256/3ac05245acbcb15be94ea73dddd27394a00e9c791526ef556151c6841b185a4b.webp)

Trust in a cloud service
23

email in → [Gmail logo] → reply out

AIE
AGENT ENGINEERING
HTTPS://AI.ENGINEER

### Trust in a cloud agent

The slide links trust to understanding an agent’s behavior. The speaker imagines an auto-reply going wrong, such as sending an unwanted message to your boss.

[8:23](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=503s) [An email flow diagram above a statement linking trust to understanding how an agent behaves on your behalf.](https://assets.ai.engineer/sites/aiecode2025/sha256/967ad65cb9c372a065b93ed56b1423820f0a79dc633971ea2d3adb1ae957eb6b.webp)

Trust in a cloud agent
24

email in
reply out

Your trust is correlated with whether you understand how it behaves on your behalf!

Event branding: AGENTS AT WORK

### Decentralize

Google, Apple and Microsoft app icons illustrate separate ecosystems. The speaker questions tying a personal agent that acts across daily life to one ecosystem when these walled gardens limit interoperability.

[9:53](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=593s) [Groups of Google, Apple and Microsoft application icons.](https://assets.ai.engineer/sites/aiecode2025/sha256/b3fd9707b423e92d604d5ccec93322ac466f7674e49d6d1529557b415f93a6be.webp)

Decentralize

25

31
JUN 29
W  X
N  O  A

### The Challenges: Technical

The speaker turns to the technical challenge of running local models, a key component of personal AI agents.

[12:11](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=731s) [Section 03: The Challenges: Technical, projected above the speaker.](https://assets.ai.engineer/sites/aiecode2025/sha256/0e2991aaaee318b9fad5d3138a0279ef0e12ccb5e9f69fac49fdafa7e7d2344b.webp)

03
The Challenges: Technical

### The Challenges: Research & Product

The speaker expects technical and infrastructure challenges to improve, while research and product challenges need more thought.

[13:57](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=837s) [Section 04: The Challenges: Research & Product.](https://assets.ai.engineer/sites/aiecode2025/sha256/f9e9ecf23d161968dfca8e304ff55a67436b93ad5ca8ebb1a003d80d79280aad.webp)

04
The Challenges: Research & Product

AIE
AGENT ENGINEERING
HTTPS://AI.ENGINEER

## Full transcript

[0:00](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=0s) [upbeat music] So, uh, why do I-- Well, first of all, who am I?

[0:21](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=21s) Uh. [laughs] Do you know this thing called PyTorch? Um, a lot of people in AI used to know it, but now a lot of people just use high-level APIs and don't know what's, uh, powering things underneath.

[0:33](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=33s) Uh, but, like, PyTorch is, uh, the software probably powering your AI APIs. Um, so I work on it. I co-founded the project. And it's, uh, it's a big project that is, uh, majority funded by Meta, where I work at.

[0:50](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=50s) Um, and so I'm talking about, uh... So I'm not talking about Llama at all. [chuckles] I work on Llama a little bit, but, uh, unfortunately, uh, I am not in charge of Llama to try to sneak in some secrets for you guys.

[1:04](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=64s) Um, I'm not gonna tell you when the next Llama is gonna come or anything like that. So, uh, why am I thinking about personal local agents? Well, the-- A-as AI kinda started becoming more and more useful, one of the things that saved my time the most every single day was swyx's AI News.

[1:26](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=86s) And it's basically, like, I have to keep up to date with all of what's going on in AI. That's my job. And now, instead of basically spending three, four, five hours a day, um, looking at a bunch of sources, like, the AI was aggregating a bunch of news for you, and I thought that was, like, one of

[1:45](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=105s) the s- like, one of the first applications that I thought was, like, mind-blowingly, uh, personally, like, effective for my own productivity. And that's when I started going into like, "Hey, I'm gonna, like, augment, uh, AI within my day-to-day in a deeper way."

[2:02](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=122s) Um, that's not an agent, though. AI News is more like a, an aggregator. Uh, but that's how it kinda started. The other thing is I also work on robotics.

[2:11](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=131s) Um, and robots are essentially agents. They act in the world. So, um, I, I... My goal is to build home robots so that I don't need to do any errands.

[2:22](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=142s) Um, and so, um, as part of that journey as well, I've been, like, kind of getting into like, okay, how do I get into understanding AI agents more deeply?

[2:32](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=152s) Um, the key takeaway I'm gonna really, like, drill down, uh, to you today is agents, like, especially personal agents, have so much possible agency in taking actions on your behalf and stuff, and they have so much of your life context to actually be useful to you that you're better off keeping them local and private.

[2:53](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=173s) And I'm gonna try to, like, sketch out a plan on how to do it, but I don't think I have a complete solution either. Um, so first, like, agent.

[3:01](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=181s) What is an agent? Like, and why did I say swyx's AI News is not an agent? Well, an agent is something that can act in the world. Like, an agent is something that has agency.

[3:10](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=190s) It can actually, like, take an action in the world. Uh, anything that can only get context and do things, but then eventually can't act in the world is not an agent.

[3:19](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=199s) That's how I think about it. And what I think is, like, a highly intelligent agent without the right context is as good as a bag of rocks. It's, like, really useless.

[3:29](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=209s) I'll give you a couple of examples very quickly. Uh, let's just say I build a personal agent. It has, uh, access to my Gmail, my WhatsApp, my calendar. And I was like, "Did, did I get my prescription renewed?"

[3:40](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=220s) And it's like, "No, not yet." And, like, it's totally lying, uh, except it didn't know because, like, I got the text from CVS on my iMessage, and it didn't have access to, uh, that source.

[3:54](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=234s) And it was doing the best it can with the information it has, but if it didn't have the context, like, it's not gonna, like, know how to do better.

[4:03](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=243s) Uh, similarly, I mean, you can make up like a hundred examples like this where, like, you have access to one bank account, but, like, it, your money came into a Venmo, and you're like, uh, the agent lied to you.

[4:15](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=255s) What happens is, like, a personal agent that doesn't have the right context, it's largely gonna be irritating to use. It's like you don't know when it is useful and when it is not useful, so it's essentially not useful.

[4:28](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=268s) Um, like, even when it gives you some answer, you're like, "Hmm, is this actually right? I'm gonna have to go dig in." Right? So unless it hits a certain level of, like, reliability and predictability that you know it is right, uh, it's not gonna be actually useful to you.

[4:46](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=286s) Um, so now, like, why am I talking about personal agents specifically? Um, and how do you, like, how do you get all this context to the agent? So let's just say you have, like, your OpenAI API or some other API or some local LLM.

[5:06](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=306s) What is all the context in the world that is personal to you, and how do you give it to the agent? Well, like, the number one thing that you probably want to do is, like, just have wearables, right?

[5:17](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=317s) You're just like, you can, uh, the... Your AI should see everything you see and listen to everything you hear. Um, and that is, like, obviously the best case of providing context to your AI agent, except, like, there's no battery life for any of these wearable things, so that's not really practical.

[5:36](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=336s) Maybe one day when you have, like, crazy batteries, but that's not really gonna work. The other thing it could be, like, okay, like, most of my life is on my phone, uh, in ways that I care about from, like, an agent perspective.

[5:51](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=351s) Uh, what about just, like, uh, running an agent on my phone? It's running in the background, and it's just, like, always, like, watching my screen or something. Well-

[6:00](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=360s) You know, that's where, like, Apple kicks you because, you know, they don't let you run a bunch of stuff and, and, like, on your phone asynchronously. If you-- even if you do, they have a lot of restrictions.

[6:10](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=370s) So, like, the ecosystem's kind of, like, kill you and not allowing you to do that. And unfortunately, I use, like, Apple. Um, so that's, that's out. [chuckles] So the next one is like...

[6:23](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=383s) Okay, actually, like, the thing that I found, like, relatively useful is, like, if you use, like, Apple in your daily life, uh, you can actually get a Mac mini and, like, just put it somewhere in your home, connect it to the internet, and you can run your agents asynchronously.

[6:38](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=398s) There's no battery life issues. You can just log into all your services on your, like, Mac mini. And, um, it also connects with all the Android ecosystems because Android is actually open.

[6:50](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=410s) Um, [chuckles] so I work at neither of these companies, so I can say whatever I want. [audience laughing]

[6:57](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=417s) Um, so I, I think that's, like, what I think is a feasible, um, um, device to use to, like, run your AI agent right now. Um, the next thing I wanna talk about is, like, okay, why are you talking about local and private?

[7:13](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=433s) Why can't you just, like, run this in the cloud, like, just subscribe to one of the large tech companies', uh, agent services and run your life out of it?

[7:23](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=443s) Well, I wanna give you, like, a few points here. Uh, first is, um, I wanna talk about how this is different from you using other digital services. And I think it is different meaningfully, and I think it's also easy to understand.

[7:42](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=462s) So let's just think about, like, a lot of you in this room probably use, like, a cloud email service that is free for all of your life. All your taxes are going in there, like, you know, everything personal is going in there.

[7:53](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=473s) Why do you trust it? The reason I think you trust it, at least the reason I trust it, is because it has a very simple mental model on how it will act on your behalf or how will it'll act in general.

[8:07](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=487s) Email in, reply out. It's not, um, go-- it's basically not trying to do something sneaky under you that is unpredictable. It's a very simple mental model. Your trust of that service is correlated with whether you understand how it behaves on your behalf.

[8:24](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=504s) So imagine tomorrow, if some unknown email service that you've been using forever says, "Oh, you know, for some of your emails that I have confidence in, I can auto-reply on your behalf."

[8:36](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=516s) And you're like, "Okay, well, first of all, that might be true, but what is the worst case action you can take?" Maybe you'll, like, reply to my boss, like, something nasty, [chuckles] and, like, I don't want that to happen.

[8:51](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=531s) And, like, that's, like, once the action space becomes powerful enough and unpredictable enough, you get uncomfortable with using a service that you're not fully in control. Um, and it can get, like, worse, right?

[9:07](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=547s) Like, uh, companies have to monetize in a million ways. And so what if, like, you're using, like, like, an online service and they suddenly are like, "Oh, you know, every time you ask for a shopping query, we're gonna, like, start making the agent only buy from, like, stuff that gives us kickbacks or something."

[9:27](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=567s) So, like, I think, like, your personal agent is so personal to you and so intimate that I feel like ultimately you want to be in control, uh, on many aspects that you might not have control on eventually when you have to, like, trust, uh, an online service.

[9:44](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=584s) So that's, like, one of the biggest reasons, like, why I feel I wanna build a personal agent that's local, uh, to myself. The second is decentralization. Like, I mean, you already see all these ecosystems that are walled gardens and, like, fighting with each other and don't allow each other to interoperate in various ways.

[10:02](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=602s) And if you build one of, like, your personal life, uh, your personal agent around one ecosystem, like, is that something-- Like, it, it works fine for compartmentalized things like maps and email and various things, but, like, is that something that you wanna really subscribe into for, like, an agent that can take so many different kinds of actions

[10:25](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=625s) on your behalf, uh, in your day-to-day life? That's, like, the other reason I, I, I feel like you should try to-- we as a world should try to get to, like, local personalized agents as the norm.

[10:37](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=637s) Um, and the third one is, um, for various reasons. Okay, this is what I had called, uh, this is what I call, um, are you going to be punished for your thought crimes, right?

[10:51](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=651s) Like, okay, you have a thought, and it is not a good thought, and, like, you know, should you be punished for it? And usually, like, the answer is no.

[11:00](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=660s) Now, if you have a personal agent that is effectively augmenting you in such an intimate, personal way, you might be asking it stuff that you generally wouldn't say out loud ever.

[11:12](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=672s) Um, and in those cases, like, do you really want to take the risk of, like, like, putting it out into, like, some provider? Because, like, you know, you, you can act-ask Perplexity, like, enterprise-grade cloud API contracts that, like, are, like, um, enterprise grade, not consumer grade, where they, like, get sloppy.

[11:34](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=694s) Even they have to, like, do a bunch of, like, legally mandated logging and then safety checks and stuff. So there is a possibility that, like, you might or might not wanna take a risk on.

[11:47](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=707s) But for me, I'm like, I don't want to ever get into a scenario where, like, I will be, like, prosecuted or persecuted for my thought crimes. [chuckles] And, like, that I think is, like, another really powerful argument for myself at least to focus on, like, local agents for my most personal, um, Uh, augmentation.

[12:09](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=729s) So now coming to, um... Well, I hope you're convinced that yes, yes, we actually like, if you're gonna build a personal AI agent, it has to be local and private.

[12:20](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=740s) Uh, well, okay, what's the problem? Well, let's go to the technical challenges first. First, like, okay, you gotta run this stuff, right? Um, there are, like, great open source projects to run a bunch of local models that are one of the key components of these agents.

[12:35](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=755s) Uh, vLLM and SGLang are pretty great. Uh, they're both built on top of PyTorch. [laughs]

[12:41](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=761s) So effectively, uh, this is one time, like, we wrote a bug in PyTorch and, um, a bunch of us, uh, had a Tesla car, and Tesla uses PyTorch. [laughs] And we were like, "Man," like, "this is so scary because, like, are we writing bugs on ourselves?" [laughs]

[13:01](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=781s) That's an aside. Um, it was totally fine. The bug was not that bad. [laughs]

[13:10](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=790s) Um, so yeah, vLLM, SGLang are great. Um, but local model inference is still, as of today, slow and limited. It's not as fast as, like, you know, if you just use, like, a cloud service.

[13:21](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=801s) Even if you spend, like, a b- like, enough money on a beefy machine, I think that's also rapidly changing. Like, for example, locally, if you're using, like, a 20 billion or distilled model of some sort, it actually runs pretty fast.

[13:37](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=817s) Uh, but if you want to use, like, the latest R1, like, full unquantized, then it runs, like, super-duper damn slow. Um, I think this, like, is in a state of, like, it will fix itself, so you, you probably wouldn't get to run the latest and greatest.

[13:55](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=835s) Um, and I think, like, the challenges are not so much the technical and infrastructural challenges. Like, they will kind of get to a place where they're fine. I think there's some challenges around, like, both the research and product that, um, people need to think a bit more about.

[14:12](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=852s) I think there's a bit of a gap, and this is just an open challenge for this room for all of you AI engineers. Um, one is, like, the open multimodal models are good but not great.

[14:25](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=865s) I mean, they're not great in a couple areas. One is, like, just computer use. Even the closed models, like in the latest and greatest APIs that you can just pay money for, they're not that great for computer use.

[14:37](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=877s) They break all the time. So that needs to definitely get, like, into a better state. The other thing, like, I noticed is, like, if I ask a model to do shopping for me, from clothes to shoes to furniture to whatever, it'll basically give me the most boring shit, right? [laughs]

[14:55](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=895s) Like, it's like the basic stuff. And if I ask it... If I'm like, "Look, I'll tell you my tastes," and my tastes can get very f- like, specific and fine-grained.

[15:04](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=904s) Like, the more specific I get, the more, like, bullshit it gives me. [laughs] Like, it's like, it's like the same, "Oh, you asked for, like, a red velvet sofa with oak wooden legs.

[15:15](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=915s) Uh, here's a green sofa that has velvet, um, and it doesn't have, like, oak wooden legs." You know, like, they're not very good at identifying actually visually what you're asking for.

[15:28](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=928s) They mostly rely on, like, a bunch of text, uh, matching. Um, the other thing, uh, you will notice, and this is a big one, is we don't have good catastrophic action classifiers.

[15:40](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=940s) What I'm-- What do I mean by, uh, catastrophic actions is

[15:45](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=945s) there's many actions an agent can take. A lot of them are reversible or harmless. Like, even if it takes the action and that's not the action you wanted it to take, it's like whatever.

[15:56](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=956s) Oh, it had to go to, like, that particular Wikipedia link, but it went to this other one? Okay. Big deal. Whatever. It'll just backtrack and go. But there's some actions that are actually catastrophic.

[16:07](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=967s) For example, you ask it to go purchase, like, uh, a renewal of your Tide Pods, and then it goes and, like, purchases a Tesla car. You know? [laughs] This is not the best thing for you to do.

[16:21](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=981s) Uh, and some of these are called catastrophic actions. And I don't think there's a lot... Like, there's some open research around, like, how to really get agents to get good at identifying catastrophic actions before taking them and then maybe, like, notifying the users instead.

[16:38](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=998s) Uh, but there's not enough. And so if you wanna really trust your agents, personal or in cloud, I think we've gotta get a bit better at these things. Um, so that's, like, a big one.

[16:52](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=1012s) And I think open source voice mode is barely there. Uh, I feel like when I need a personal local agent, I definitely want voice mode because sometimes I wanna talk to it, uh, and not actually type out everything I wanna say.

[17:05](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=1025s) Um, but still, why am I bullish about this whole thing? I am, uh, because one, I see open models are actually, like, compounding in intelligence, like, faster than closed models, like, based on how many resources are being put on them.

[17:19](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=1039s) Like, what do I mean by that? Like, OpenAI is only improving their own model. Anthropic is only improving their own model with all the billions they have or whatever.

[17:27](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=1047s) But open models are improving themselves, like, in coordination across board. Um, and, you know, people didn't really b- believe it until Llama came out, and they didn't, they didn't really believe it until Mistral came out, and then they didn't really believe it until Grok came out.

[17:45](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=1065s) Uh, and then they didn't really believe it until DeepSeek came out. Like, basically, like, people are like, "Oh, you know, like, open models, you know, will not really win," but I think they are.

[17:57](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=1077s) Like, basically, in open source, like, I've worked in open source, like, all of my life, um, there's a starting coordination problem. Like, initially, you don't have enough of a critical mass to coordinate with each other.

[18:10](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=1090s) But once you have a critical coordinated mass, open source kind of starts winning in- Like an unprecedented way, and you see that with Linux, you see that with a bunch of projects.

[18:21](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=1101s) Uh, so I am pretty bullish that open models will actually start getting, like, better than closed models, um, um, like per dollar of in- investment into open models. Um,

[18:36](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=1116s) and that's what I said. Well, okay, I have some plugs. Uh, this is like, uh, GR.inc from, uh, my friend, uh, Ross Taylor, who worked on this model called Galactica, which got a lot of s- like, um, criticism when it was released, uh, out of Meta.

[18:55](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=1135s) It was this open science model before ChatGPT released. Now, like doing science with like LLMs is pretty common, but like [chuckles] they got a lot of shit when they released.

[19:04](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=1144s) Uh, and he like quit, uh, he, it like unreleased Galactica, and he quit doing like, uh, a bunch of stuff publicly. But then, like he's working on like plugging the, the reasoning gap between open models and closed models, and they released a bunch of open reasoning data, uh, that will help.

[19:23](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=1163s) So just a nice quick plug. The other quick plug is I work on PyTorch. PyTorch is working on enabling local agents, especially the technical challenges that I talked about, uh, and we're hiring.

[19:35](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=1175s) So if you are more than an AI engineer, if you're an AI engineer who's also like a systems engineer, then like PyTorch is hiring. Um- [laughs]

[19:49](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=1189s) Well, [laughs] that's what we got. [laughs] The other thing obviously is I welcome you all to come to LlamaCon, which is happening on April twenty-ninth, and save the date. It's gonna be very exciting.

[20:04](https://www.youtube.com/watch?v=jMoAaZP_Kkw&t=1204s) Lots of Llama stuff will happen there. That's it. Uh, I think it's in California. I, um, I actually didn't look it up. [laughs] [upbeat music]
