AI Engineer World's Fair 2026
State of the Union: Why Local, Why Now
Read the talk
Why Local AI Now: From Downloadable Models to Systems You Can Control
Always-on agents make privacy, inference cost and control harder to ignore, while better models and deployment tools make local AI increasingly practical.
From a talk by Nader Khalil, Alex Cheema, Matthew Berman, Ahmad Osman and Joseph Nelson
Before you start: Familiarity with language models and basic inference concepts is helpful; no experience running models locally is required.
When the agent stops turning off
How closely do you need to watch a coding agent? Nader Khalil opens with a striking change in the answer. He recalls Andrej Karpathy warning in November that agents still required close supervision, then describing difficulty keeping up with their capabilities three months later. Both assessments could be right: the models and the harnesses around them were improving together. Khalil’s practical response is to use AI a little more each day, so experience can keep pace with capability.
That progress also changes the workload. A chatbot generates a burst of text in response to a question. A reasoning model first spends compute generating tokens the user does not directly consume. An agent can keep working beyond either interaction—and, given enough useful work, its owner may not want it to stop at all.
The more useful these systems become, the more sensitive context people want to give them: enterprise intellectual property, medical records, health data and home-camera footage. Continuous generation also makes token costs harder to ignore. Local inference brings the data boundary and the compute budget under the operator’s control. Khalil describes this as a plateau in token spending on owned compute, rather than an ever-growing metered bill, and says the room’s demonstrations keep their processing on the devices present. That is a bounded-capacity deployment proposition, not a claim that hardware and operation are free.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The first experience of owning the model
The panel approaches that proposition from several directions. Alex Cheema introduces himself as co-founder and CEO of EXO Labs and creator of Local.ai, with more than two years spent making local inference accessible. His opening comparison runs from an early large-Llama experiment on two MacBooks to Nemotron Ultra on four Sparks. Matthew Berman brings the perspective of an enthusiast who tests AI for videos and a newsletter. Ahmad Osman, Osmantic’s founder and a LocalLLaMA moderator active in the space since 2022, jokes that an interpretation of his name as Open Source Man became a company.
A passing reference to Fable returning introduces another reason to care about local access: a product built on frontier intelligence depends on that intelligence remaining available. Joseph Nelson, Roboflow’s co-founder and CEO, adds a physical constraint. Computer vision often needs to run alongside the camera producing the images, with sufficiently low latency to keep up with video. For vision developers, deployment near the data is already familiar.
For Berman, the first turning point was Llama: intelligence he could download and run in his office. Years of building and overclocking PCs made that possibility immediately compelling. Osman’s equivalent moment came with Llama 2 on an RTX 4090. Sampling parameters, configuration choices and inference engines became things he could inspect and change. The appeal was both practical and educational: local execution made the black box something he could experiment with, while leaving room for cloud systems to coexist.
Khalil recalls an approximately 2023 airplane experiment in which a phone model took 20 minutes to finish a sentence. It was painfully slow, but it answered without an internet connection. Osman contrasts that experience with Qwen3.5-4B on an iPhone, which he describes as offering GPT-4o-equivalent quality. The panel supplies no matched evaluation or phone configuration for that comparison; its significance here is Osman’s assessment of how far pocket-sized inference has progressed.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A model that fits still has to be useful
Cheema separates the excitement of fitting a model onto local hardware from the experience of actually using it. Llama 405B narrowed the capability gap with closed frontier models, but his dense-model experiment was too slow for practical use. DeepSeek V3 and DeepSeek R1 marked another turning point: he credits mixture-of-experts models with making very large models more practical on hardware such as a Mac Studio or Spark. The distinction is between storing a large model and paying the full dense-model computation cost for each generated token.
| Milestone in Cheema’s account | What changed | Remaining qualification |
|---|---|---|
| Local Llama 405B | Narrower capability gap | Cheema reports two tokens per second, too slow to be useful. |
| DeepSeek V3 and R1 | Practical progress with MoE models | Performance depends on the deployment. |
| A model Cheema calls GLM 5.2 | Described as Opus-level on a DGX Station | A qualitative assessment, without an evaluation setup. |
The under-desk DGX Station example extends the same trajectory: increasingly capable models on hardware someone can control directly. Cheema expects compression and lower memory requirements to continue moving that boundary, eventually making local inference the default.
Nelson’s turning point was less about model size than a particular failure. On a flight, a passenger with limited vision was repeatedly photographing the cabin to understand the seating and how to buckle in. Apple’s accessibility feature described the seat back as something like a printer. The passenger knew from context that the answer could not be right.
Nelson tried the recently released LLaVA on a photo of the tray in front of him, and it correctly described an airplane seat tray. LLaVA connects a vision encoder with a language model through visual instruction tuning; the original paper appeared in April 2023. In this particular comparison, a broadly available model supplied a useful description where the phone’s built-in feature had failed. For Nelson, that was evidence that useful intelligence need not be monopolized by the largest technology companies.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Connect intelligence to the task
The camera is essential to Nelson’s example: it gives the model access to the situation it must explain. Khalil connects that to the wider improvement in harnesses. A CLI can expose a business system to an agent; a coding environment can expose the filesystem. His earlier workflow involved choosing snippets, pasting them into ChatGPT, then copying the answer back. Cursor changed the division of labor by letting the agent inspect files and decide which context it needed. A better harness changes what the model can observe and do.
Vision also offers a lesson in specialization. Nelson contrasts language workloads that can often reach data-center compute with cameras and robots operating on limited hardware or weak connections. Those constraints encourage a model that performs well in a particular domain, rather than prioritizing world-scale generality. Language systems are following a similar path through coding-specific harnesses and last-mile adaptation for work such as tax or legal preparation.
Specializing the task does not eliminate the separate work of optimizing inference on the device. It does, however, change the question from which single model should do everything to which model belongs in each part of the system. Khalil describes NVIDIA’s expectation as a multi-model world, with routing important enough to receive its own later summit panel.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Spend intelligence where it matters
Berman already uses a mixture: a top-end model he calls Fable, Sonnet as a workhorse, and local models for work less sensitive to latency. He cites a Brian Armstrong post describing Coinbase’s rising token consumption with flat costs through a mixture of models. That is the budget problem in miniature: an enterprise wants more useful AI work without pricing every task at the most expensive tier.
A concrete division of labor is to let the strongest model plan the architecture, then assign coding subtasks to smaller, less expensive models. Local models can perform that execution while a frontier service handles the top-level plan. This gives the operator more control over the workflow and reserves expensive inference for the part expected to benefit most from it.
Cheema sees market demand for that flexibility. His illustrative objection is paying $50 per million tokens for a giant model when the workload does not require one. Enterprises also want sovereignty: the ability to replace a model and continue operating if a provider withdraws access or changes its terms. Those requirements make model choice an architectural concern, not merely a dropdown in a chat interface.
Routing creates its own engineering problems. The selected model needs the right context; a planner’s output must become workable sub-agent assignments; the system needs a framework for coordinating those assignments. Version control matters too. If model behavior changes, an operator needs to know which version ran and should be able to choose when to adopt an update. Local ownership can support that control, but the harness still has to implement it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Collect the evidence for specialization
The sovereignty discussion extends across the stack: hardware, software, weights, model versions and fine-tuning. If AI becomes civilizational infrastructure, access to an endpoint is only one layer of control. The panel’s proposed destination is smaller models adapted to a user, use case or workflow, rather than one undifferentiated service for every need.
Osman’s immediate recommendation is to start collecting workflow evidence. A business cannot decide which tasks belong on which models merely by inspecting model names. It needs employee interactions, execution traces and feedback about whether the work succeeded. A practical sequence follows from his proposal:
- Capture traces from real workflows and the feedback attached to them.
- Separate the recurring tasks and use cases represented in those traces.
- Use that evidence to decide which models or endpoints suit each task.
- Continue collecting feedback as the routing and workflows change.
Agents could help analyze and improve this process, but the traces provide the basis for the decisions.
Osman then describes a more ambitious form of recursive self-improvement: a model obtaining compute, training new checkpoints and deploying its next version. This is a proposed progression beyond improving prompts or workflows. Khalil supplies a narrower observation from Brev: agents are increasingly requesting GPUs directly, which has led the product to treat agents as a first-class audience. Obtaining compute is one component of the proposed loop; it does not by itself establish autonomous model improvement.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Extract more from the hardware already available
The hardware discussion begins with a collaboration: Cheema wanted to extract more performance from DGX Spark, and Khalil offered a conference room at NVIDIA where relevant experts could join the effort. Osman connects that work to home labs. Enthusiasts trying to maximize the value of limited hardware often face the same problem as the enterprises where they work: obtain useful inference within a budget. He points to quantization as a central example of that optimization mindset.
The scale can range from a DGX Station serving a small or midsized business to a larger cluster. Osman’s definition of local is correspondingly broad: on-premises hardware, colocated machines and rented clusters can all participate, including compute obtained through Brev. The common thread is control from hardware and software through endpoints and weights, together with the ability to collect data for more efficient specialized models. Physical possession and end-to-end control are related, but they are not identical requirements.
Khalil introduces the Spark result as an exercise in assembling existing solutions. Cheema recounts the sequence: a Thursday dinner, rapid email coordination and a Monday start at NVIDIA headquarters. Teams moved through the room, including Nemotron, data-center and vLLM-for-Spark specialists. He calls this an example of Jensen Huang’s swarming approach; Khalil likens it to children converging on a soccer ball.
Cheema reports a 10× performance improvement over roughly three weeks relative to NVIDIA’s existing DGX Spark playbook using Hermes Agent. The panel does not define the workload or performance metric behind that multiplier. The described work used vLLM as the inference backend, quantized and tuned models for local operation, and adjusted configurations designed for data-center deployments.
Shared architecture made that adaptation valuable. Cheema points to Grace Blackwell and existing optimized kernels as a foundation the local system could reuse. The work was therefore less about inventing new computer science than identifying which settings and optimizations needed to change on Spark. Better compression then compounds the benefit by fitting more capable models onto smaller devices.
Cheema reports Nemotron 3 Ultra, a 550-billion-parameter model, running on four Sparks at 30 tokens per second. He does not give the prompt length, concurrency, quantization or timing method for that room demonstration. It is a separate reported result from the 10× playbook improvement.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make the first useful interaction easy
Osman compares the state of local AI to Linux in the 1990s: promising capabilities, an active community and substantial infrastructure still to build. His Osmantic Deployment System, or ODS, bundles open-source tools and configures the services and agents needed for a local deployment. The goal is to package an entire working stack, with optimizations appropriate to each hardware target—from phones through Sparks and Stations to data centers.
Berman sets a demanding usability target: starting local AI should be roughly as easy as opening Cursor or installing and opening Codex. He considers the current sophisticated setups beyond what most people, including many enthusiasts, can manage. Setup is only half the problem. Users also need to know what their combination of model, harness and hardware can usefully do, whether the compute is at home or rented.
Documentation alone cannot deliver that experience. Osman wants the system to figure out the configuration; Berman wants it to work inside familiar tools without exposing infrastructure details. For an ordinary user, the interaction should be simple: start the application, send a message or give an agent a task. The technical choices still exist, but they should not all become prerequisites to the first useful result.
That leads to Khalil’s product-versus-plumbing question about routing. Berman places model selection inside the user’s front door to AI—the harness that understands the task and can choose the appropriate model. Routing alone is only one component of that experience. Osman illustrates the broader approach with Carter’s reported ODS onboarding: it first downloaded a two-billion-parameter model so he could begin interacting, then downloaded the next model suited to his device. The useful state arrived before the final download finished, without requiring the user to choose quantization formats or extensions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When should knowledge move into the weights?
Specialization makes a new demand on the builder: know what the model is specializing in. A broadly capable model can accept an open-ended request; a specialized system needs a defined task. Osman contrasts the aggregate feedback available to a cloud provider with the particular interaction patterns and workflows inside one business. Training compute and storage for customized weights make that distinction consequential. Each entity needs evidence about its own work.
Osman predicts that continual learning will move some agent memory from expanding collections of Markdown files into model weights. His concern is that accumulating skills and memories eventually creates an inefficient context burden and makes it harder for the agent to maintain the right files. He argues for local weight updates as the next step. That remains a proposed direction in the discussion, rather than a demonstrated replacement for file-based memory.
A reference to Swyx’s question about why fine-tuning as a service has not taken off exposes the difficulty: customization itself is hard. The panel points to Nemotron releases that include data, weights and recipes as a way to make that work more accessible. But the central trade-off remains unresolved: how often does a smaller tuned model justify its development and maintenance cost when a strong general model already performs well with suitable context?
Cheema’s answer is that an open ecosystem can explore several paths at once. He uses speculative decoding as an example: a smaller model proposes tokens to help a larger model generate faster, with the larger model responsible for verification. He recalls several recent advances, including work from DeepSeek and work involving Modal, SGLang and DFlash draft models for Qwen. The Modal account credits Z Lab with DFlash’s architecture and describes Modal’s Qwen speculators and SGLang integration.
Reusable specialization changes who must do the hard work. Someone with a particular need can adapt a model and release it for others; early image-generation LoRAs provide a familiar example. A managed service could similarly handle fine-tuning for a business. The user’s test remains whether the system solves the task within budget, while keeping model operations out of the work they actually want to do.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use broad models to build a narrow deployment
Nelson makes specialization concrete with a small thought experiment: thinking of any object is harder than thinking of an object in your fridge. Real deployments often resemble the second request. They operate in a bounded environment, even when the models used to understand that environment have much broader knowledge.
His example is the Monterey Bay Aquarium Research Institute, or MBARI. Nelson reports that the institute used Roboflow to process deep-sea submarine footage in work associated with discovering a new fish species. He describes Segment Anything 3 and language models acting as judges in an automatic labeling pipeline. SAM 3’s broad visual knowledge is useful during preparation, but the deployment only needs to recognize the underwater subjects relevant to the research.
The pipeline he describes separates broad interpretation from narrow inference:
- Run large, general models over collected footage to propose annotations.
- Use language-model judges to assess agreement among those interpretations.
- Turn the accepted annotations into a specialized dataset.
- Build a task-specific model from that dataset for the submarine and later video processing.
Nelson describes the resulting specialized model as suitable for real-time submarine inference and faster post-processing. The large models contribute knowledge during dataset preparation without all becoming part of the deployed inference path.
This also changes which model should be adapted. Nelson cautions against automatically choosing to fine-tune SAM 3 simply because it helped label the data. If the destination has a fixed class list, its open-vocabulary breadth may no longer be the capability worth carrying into deployment. He proposes using DETR or another specialized model, avoiding what he calls SAM’s expensive autoencoder portion. The architectural lesson is to select the deployed model for the target task; his claim that fine-tuning sacrifices open-vocabulary capability is a caution about that choice, not an established property of every SAM fine-tune.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Discover the use case before training
Who actually operates that specialization pipeline? Nelson says Roboflow supplies the platform and recipes so customers can do it themselves, with optional paid forward-deployed engineering help. Osman connects this back to workflow traces: collect data while the business uses AI, then use that evidence to build smaller models for recurring tasks and improve token efficiency.
Cheema expects more systems to use powerful frontier models to bootstrap efficient open-source deployments. His recommendation is to use those models during the transition, rather than requiring an immediate break from them. He also observes that preventing this kind of bootstrapping has proved difficult. The practical pattern is to spend broad capability on discovering and preparing the work, then move repeated execution to a system better matched to its constraints.
Khalil connects that reversal to Swyx’s writing about the AI engineer. In his account, the older sequence was to train a model and then search for a use case. Large general models made it possible to start with the use case and pursue additional machine learning only when the application justified it. A working application can produce the evidence for specialization. The specialized model becomes a response to an understood workload, rather than a prerequisite to discovering whether anyone needs the product.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The remaining problems are technical and institutional
An audience member asks for the biggest open problems in local AI. Osman returns to inference optimization, easy startup and extracting useful performance from a constrained budget. He points to hardware-specific deployment systems and to the continued usefulness of older GPUs: his RTX 3090s, once used for Llama 2 and Dolphin fine-tunes, now run newer Qwen models. He mentions Qwen 3.6 and 27-billion-parameter models with excellent performance, without providing a numerical measurement. The opportunity is not limited to developing new hardware; it includes making existing hardware easier and more efficient to use.
The intended beneficiaries span individuals, midsized businesses and enterprises. Osman’s standard is that efficient local AI should solve more problems than it creates. Khalil names the current technical agenda: quantization to fit models into smaller memory footprints, routing to allocate work appropriately, and continued model development. As those bottlenecks change, the questions organizing future summits should change too.
Nelson adds two challenges that cannot be reduced to tokens per second. One is the persistent trade-off between simplicity and customizability: local ownership enables changes and optimizations, but a hosted product can arrive with those decisions already made. The other is continued access to open models. The ability to use, change, adapt and experiment with a model depends on that access remaining available, which he argues requires advocacy as well as technical work.
Khalil frames open source as a competitive environment through which useful ideas reach more people. His analogy is Silicon Valley’s tension between profit-seeking businesses and people who want to share software freely. Osman emphasizes that the two can coexist: a project can serve consumers and sell to businesses. Cheema closes by naming Right to Intelligence as a route for people who want to advocate for open-model access without contributing technically. The freedom to operate and adapt the system is part of the infrastructure the community still has to build and preserve.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Connect Macs and workstations into a local inference cluster with automatic discovery and model distribution.
A local AI deployment stack combining inference, chat, agents, workflows and hardware-aware setup.
Official model card with usage examples, evaluation information and inference settings.
Meta's SAM 3 implementation, with inference and fine-tuning code, checkpoints and example notebooks.
Further reading
NVIDIA's deployment and benchmarking guide for a four-Spark cluster using NVFP4 weights, vLLM and multi-token prediction.
- Speculation Is All You NeedArticle
Modal explains its DFlash speculators for Qwen models and performance engineering with SGLang.
The original LLaVA paper explains how visual instruction tuning connects a vision encoder with a language model.
Read the complete timestamped transcript
- 0:00
[upbeat music] Can you guys hear us?
- 0:13
Pretty good time.
- 0:14
Soundcheck. All right. Give it up for local AI, everyone. [clapping] Woo.
- 0:23
I hope you guys are excited as we are. Um, this is-
- 0:26
Woo.
- 0:26
Woo. This is the Local AI Summit. So we're gonna be here all day talking about local AI. And the reason why is we hit an inflection point this year.
- 0:35
Not only did the models get really good, but the harnesses got really good. And this happened really fast. It's been, I think, a struggle for anyone here to keep up.
- 0:44
I felt that most when I saw one of Andrej Karpathy's tweets. In November, he tweeted that you can't really trust these coding agents alone yet. You have to monitor them with an eye like a hawk.
- 0:53
Three months later, he tweets that he's struggling to keep up with the capabilities of how good this all has gotten. And the thing is, both times he was right.
- 1:01
This space is progressing really quickly, and it's wild how much you can do. And honestly, the only thing to do is just try to use it a little bit more today than you did yesterday.
- 1:10
That's how to keep up with this space, and that's exactly what you guys are doing right here. And so I'm really excited about this panel. The way that we use AI has also changed.
- 1:18
I'm not just using chatbots. I'm not just asking simple questions. When we got reasoning models, the, the profile of how the AI model responded changed. Not only is it bursty and responding to me, but before the burst, it kind of plateaus for a bit.
- 1:33
It's reasoning. It's churning on tokens that I'm not consuming.
- 1:38
And then we got agents, and suddenly, I don't even want these agents to turn off. I want these always-on agents. They can always be productive if we set them up.
- 1:47
And so we have enterprises that want to put a lot of their IP into this because it becomes more useful. We have consumers with the same thing. I want to give it my health data, my medical records.
- 1:58
I want to give it footage from my home camera. And both enterprises and consumers, we don't want that stuff to leak. And as you also have a profile of co- uh, tokens continuously generating, suddenly costs matter.
- 2:11
So local is amazing for both of those things. You get to make sure that you are plateaued on the costs, uh, for the tokens that you're generating. And also, uh, everything sits in that room.
- 2:22
So we have amazing demos here after these talks where the y- everything that's being run stays on those devices. It stays in this room. And so that's a really nice guarantee.
- 2:31
So as we turn over the panels, um, first, do you guys want to introduce yourselves?
- 2:36
Yeah. Should I start?
- 2:37
Please. You got the mic.
- 2:38
Oh, yeah, your mic'd up. [laughs] [laughs] Um, so yeah, I'm Alex. I'm the co-founder and CEO of EXO Labs and also the creator of Local.ai. So we're-- we've been working on local AI for over two years, which feels like a lot longer in this space.
- 2:54
And, you know, from the days of, you know, running Llama Four hundred and forty B on two MacBooks to now where we have, you know, demo there of Nemotron Ultra running on four Sparks.
- 3:04
You know, our mission is to make AI more accessible and, you know, it's crazy to see what is possible now. Um...
- 3:12
Cool. Hey, everybody, I'm Matt. Um, I make content, uh, videos. We have a newsletter all about artificial intelligence. I'm an AI enthusiast. Uh, and yeah, thanks for having me.
- 3:25
Hey, everyone. Um, my name is Ahmad Osman. Uh, I am the founder and CEO of Osmantic. Uh, somebody jumped into my DMs, asked, "Hey, what does OS Man stand for?"
- 3:36
Open Source Man, so that became a company. [laughs] And, uh, uh, I also moderate Local Llama, the subreddit. I have been in this local AI space since twenty twenty-two. And, uh, yeah, let's, uh, let's make open source and local AI one.
- 3:50
Love it.
- 3:51
Woo.
- 3:51
Woo. [clapping] [laughs] Open Source Man gets the round of applause. [laughing]
- 3:58
Woo.
- 3:58
That's, that's appropriate. [laughing] Today is, uh, I think, a pretty momentous day for the panel. Um, you know, Fable just came back, but it's a good reminder of why we need to have access to frontier intelligence to be able to build anything.
- 4:10
Uh, I'm Joseph. I'm co-founder and CEO at Roboflow. We do all things vision. I kinda like to joke that vision is like the original local AI. [laughs] 'Cause, like, everything needs to run probably concurrent with, like, low latency alongside your video, uh, where the images and data is being captured.
- 4:25
Uh, looking forward to the discussion today and all things why local AI is the future of making AI useful for everyone. Everyone, let's give Joseph and everyone a round of applause. [clapping]
- 4:33
Woo. So one thing that I'm really excited about for all the panelists is you guys were all early to the space in your own way, uh, and, uh, you know, very early.
- 4:45
And I think as we all feel this inflection point now, uh, when do you guys feel like you felt the inflection point?
- 4:52
Yeah, I, I can go first, sure. I mean, look, when I first saw Llama, right, I got really excited. The idea that I can actually download this intelligence, run it from my local computer, it was very exciting.
- 5:03
I'm a tinkerer at heart. I'm a builder at heart. I, I've overclocked PCs for years, so this just felt like so good. It felt so interesting to be able to have this crazy alien intelligence running on, on my device in my office.
- 5:19
Uh, and that, that was the first time where I didn't even think it was possible, to be honest, till I saw Llama, and I was like, "Oh, wow, this is incredible."
- 5:27
Um, so that was really the point at which I was very turned on to it.
- 5:30
I, I think I, um, it's kinda the same for me. Uh, yeah, it's Llama Two, I think, that, um, finally on my forty ninety, RTX forty ninety, I'm like, "Wow, I can actually understand this black box and customize it and play with parameters and, like, you know, sampling parameters and all those, uh, configurations and see how inference
- 5:49
engines work." Uh, that, that, that made me feel like, you know, something clicked. It's like, oh, magic. I can be a wizard now with this thing. I can control how it, how it plays to my ways of thinking.
- 5:59
Um- And from there it was, you know, the rest is history. Basically, I've been very vocal about why local and open-source AI must coexist with cloud and how, how we are supposed to like, you know, push for that and try to understand it as much as possible and teach people about it.
- 6:15
Totally. I actually felt this as well. I was on a plane, I think it was in 2023, and I had, uh, a model running on my phone, and it was awful.
- 6:23
It took twenty minutes to complete a sentence. But I was-- I didn't have internet, and so I was able to like ask, uh, essentially intelligence something there in the plane.
- 6:31
And, and that was my first feeling of like, "Hmm, this is gonna be something really cool."
- 6:35
Do you know that you can run the equivalent of GPT-4o on your iPhone now?
- 6:39
Really?
- 6:39
Yeah.
- 6:40
Amazing.
- 6:40
It's Qwen 3.5, the four parameter-- four billion parameters. Y- it, it's basically the same quality as something that used to be served in data centers, and it's in a device in your pocket.
- 6:52
Yeah, that's crazy. [laughs]
- 6:54
Yeah, I think if you just think about like how crazy that GPT-4o moment was, and now you can run it on the phone, it's like... I think-
- 7:01
Massive
- 7:01
... a lot of this stuff was just, you know, you had to see the vision of like where things were going because it was a toy, right? It was like, you know, the, the first-- I think there were a few key points for me.
- 7:11
One was like running Llama 45B locally, right? That was like the first really big open source model. And I remember like the gap, it closed the gap quite significantly with the frontier closed models and open.
- 7:24
Uh, but it ran at two tokens a second. [laughs]
- 7:27
So it wasn't useful, right? Um, and then, you know, another big moment was DeepSeek, um, both V3 and R1. De- DeepSeek V3 was like a massive MoE. So like, you know, Llama 45B was dense, so super slow.
- 7:39
MoE seemed to unlock the performance. So it was like, "Oh wow," you know, with the devices I already have, like with a, you know, a Mac, a Mac Studio or Spark, I can run, you know, this massive model at actually like decent performance, which is comparable with, you know, what you can run in the cloud.
- 7:57
And then, uh, I think j- just recently, to me, GLM 5.2 is a big moment as well. Uh, 'cause again, it's closing the gap and it's like, you know, Opus level.
- 8:06
Um, and, you know, we have it running over there on a device that like literally can fit under your desk, uh, the DGX Station. So this is, to me, just like, you know, a, a trend and it's gonna keep increasing, right?
- 8:20
Like there's gonna be smaller and smaller devices with less memory, better compression. We'll be able to run more capable models locally and, you know, soon it'll be the default.
- 8:32
I'll keep with the, uh, the theme of, of airplane stories. [laughs] [laughs]
- 8:36
So I, I agree that there's been like multiple moments over, over time of what's been, what's been going on, but one personal-
- 8:41
I have a presentation for that after this, so stay around please. [laughs]
- 8:44
All right. Uh, one example moment where this like came up for me was, uh, I was on a plane. I was sitting next to someone, man, this is maybe two, three years ago, and they were hard of sight and they were using-- if you, on Apple, you know, if you take a photo of something, you can use
- 8:58
the accessibility settings, and it'll describe the photo to you.
- 9:01
Mm-hmm. Mm.
- 9:01
And we were seated on the plane together, and they were taking constant photos to understand where were we seated and how to get buckled in and these sorts of things.
- 9:08
And I remember they took a photo of the seat back in front of them, and the Apple accessibility described the photo, described it as like a printer or something. [laughs]
- 9:16
It was like, "Oh yeah, like you're seated like by a printer." And of course, the individual knew from the context that couldn't be right. Uh, and I was like, you know, I wonder.
- 9:22
LLaVA had just come out, a multimodal model- [laughs] ... that like kinda just described things with like the pivot to text base rather than like natively multimodal, but still kind of a useful example.
- 9:31
And I was curious, I was like, "Well, let's see how LLaVA would do on the same thing." So I took a photo of like the seat tray in front of me, and it aptly described it as like, "Hey, you're on an airplane.
- 9:37
That's a seat tray." And I showed the person sitting next to me, who had never seen local models and never experienced local AI, and what that stood out to me is I was like, man, a company that's a trillion-dollar-plus market cap business shipping the latest intelligence on their phones for describing visual settings is inferior to something that's
- 9:54
broadly accessible and available to anyone. And that was such like a watershed clear moment that like even the largest companies don't have a monopoly on the frontier of intelligence.
- 10:03
And so increasingly making that be accessible to others, um, is, I think, gonna be critical for un- understanding all the massive impacts that tech will have. And that was LLaVA. [laughs]
- 10:15
Yeah, it's interesting to hear you-
- 10:16
That's like, that's like twenty twenty-two, twenty twenty-three, right?
- 10:18
Around there.
- 10:18
Yeah.
- 10:19
Yeah.
- 10:19
Yeah, that was a while ago. It's interesting to hear you say about the, the accessibility of that, right? The, uh, taking this frontier intelligence, what made it useful was giving it peripherals, in this case, a camera, so that it could access the data in front of you.
- 10:31
And I think that's where the inflection point this year was so much more than just models, but also these harnesses and what you can give it access to. It was CLIs so that you can give business systems, uh, and plug those right into your agent.
- 10:43
It was, uh, you know, 4o was really exciting because I was coding with it, right? I was taking code snippets from my code base, and then I'd go to ChatGPT, and I'd paste it.
- 10:51
I'd give it as much context as I thought I should, and then I would take the result, and I'd go back into my code base and paste it. And what was really great about tools like Cursor was it essentially was this harness that said, "What if I just had the full file system?"
- 11:03
And I can let these agents reason on what files they needed and so on. And so the way that, you know, you get more use out of all of this local, uh, out of all of these local s- models and systems is how does it, how can it interact with the real world?
- 11:17
I'm curious, um, Joseph, so you guys got the start in vision, and it seems like vision had to learn the hard way what a lot of LLMs are dealing w- uh, discovering right now.
- 11:28
What do you think is like a big lesson that the language world is discovering right now that, uh, we could look to vision for?
- 11:34
I think, um, one of the things that language has an advantage of is it's inherently a human construct. So language generally exists where people exist, and that also means usually you can use, uh, infinite amounts of, of computer, what is available in a, in a data center.
- 11:49
Whereas a lot of vision, uh, you're compute constrained. You're running maybe where there's low internet connectivity, you're running on a device, you're running on a robot, and the amount of compute you have available to you is, is what, um-
- 12:01
And so what did that do? That created, I think, an emphasis on specialized learners faster.
- 12:05
Mm-hmm.
- 12:06
Because it's like, okay, I need this model to run in a limited context. Um, and I'm not gonna prioritize full world-scale generalizability. Instead, I'm gonna have specialized in-domain context work.
- 12:18
And what's interesting is I think you're actually seeing language follow a similar thing. You're describing it in the context of, like, harnesses for a given context. Um, but, you know, tools like, uh, any sort of coding agent where you have a really good harness and is specialized for doing those sorts of tasks.
- 12:31
But you're also seeing that even in like, you know, tax preparation or legal preparation of last-mile fine-tuning adaptation specialized learners. And so in some ways, I feel like the pendulum's swinging back to actually having specialized models, even in language just as much as vision, I think is one thing.
- 12:47
Um, and then, you know, there's a whole different thing around how you get the most out of compute optimization-
- 12:52
Yeah
- 12:52
... when running on, on an edge device. Um, but I know that probably the Exo folks can talk, speak more to that.
- 12:57
Absolutely.
- 12:59
Um, but yeah, I think specialized models being... You remember there was like one model will rule them all world-
- 13:04
Yeah
- 13:04
... was what everyone kind of thought, and I feel like the pendulum has swung back to people realizing specialized-
- 13:07
Specialized models. I think we definitely-- At NVIDIA, we see that it's gonna be a multi-model world.
- 13:11
Yeah.
- 13:12
We, we definitely agree with that. Um, there's actually one of the panels that's coming on later, I think at 3:20 PM, is all about model routing, and so that's, that's really exciting.
- 13:19
I guess as an enthusiast, uh, and as you use some of this, is that, uh, how your usage pattern looks? Are you using many models? What does that kinda look like?
- 13:27
Yeah, absolutely. Everything from obviously, like the top models, Fable, uh, uh, to kind of the more workhorse models with Sonnet, local models for things that maybe don't depend as much on low latency.
- 13:40
Um, y-yeah, like the multi-model world s-seems like a no-brainer at this point, uh, especially as you're seeing all of these enterprise companies come out and look at their budgets and, and think, "Okay, well, I, I need to continue to increase the, the total number of tokens that I'm consuming as a company, but I also don't wanna just
- 13:59
completely blow out my budget." I know Coinbase and Brian Armstrong just came out with that just great post the other day talking about, uh, how their tokens are, are exploding, yet their costs are staying flat and, and that is because they're using a mixture of different models.
- 14:14
You don't need the top model for every single use case, and in fact, most use cases you don't. Uh, I think the most obvious application is let the top model plan, uh, the, the, the architecture, whatever the kinda the top-level plan is, and then the actual execution of the code can go to, uh, a, a more reasonably
- 14:30
priced smaller model.
- 14:31
Your most intelligent should provide you with the overall plan and then sub-tasks for your smaller execution, like a executioner models, and that's exactly the future.
- 14:43
Yeah, uh, especially like, you know, we're talking about local. Um,
- 14:47
local models are great at writing code, but maybe we offload the actual top-level planning to one of the frontier models, and you, you save a bunch, you control more of the workflow.
- 14:56
It's, it's a, it's a nice pattern.
- 14:59
Yeah. I, I think the market wants this. So it wasn't clear like a few years ago when this stuff was just starting to play out, you know, would there just be one big model that everyone is using?
- 15:10
But clearly that is not what people want. That is not what enterprises want. They don't want to be told what they can do by Dario. They don't wanna be paying for the same model for all their workloads when some mo- some workloads don't actually need, you know, a giant, gigantic model that costs fifty dollars per million tokens.
- 15:28
Um, they want control, they want sovereignty, they want the ability to switch out models. They don't want to get rug pulled, you know, uh, from one day to the other, uh, because of some safety risk or whatever.
- 15:41
Um, and, you know, that's really what's driving a lot of this progress. So, you know, I'm more optimistic than ever, I think, about sort of, you know, where things are going with Local.
- 15:51
I think that, you know, the market is basically pulling a lot of this stuff, um, out of, you know, startups, out of enterprise are building solutions around this and, you know, it, it's, it's, yeah, it's, it's happening, right?
- 16:03
Totally. I feel like this is a new frontier where as we go multi-model, that becomes really difficult. Of, of course, you need to route between the models, but then you have to figure out how to provide necessary context to whichever model you're routing to.
- 16:14
If you are making a plan and then breaking it off into sub-agents, what is the framework to do so? How should you do this? These are all open problems and, you know, I think just being playful is, uh, kinda the best way to, to approach this.
- 16:25
But I like the way that you worded that of like, uh, the market's kind of pulling for these answers. It's reaching in this direction. We're looking for startups and companies to fill this gap.
- 16:32
Um, makes a ton of sense. You know what's funny? Um, as any, any enterprise consumes any s- any piece of software, it matters a lot. If you're gonna build a foundation, you wanna know what the versions are.
- 16:43
You wanna know what your model is, which-- whether that version changed in order to see changed behavior later. So having, uh, also more control over all of this so that you know exactly what version it is, it can't be changed, uh, n-not even because of any sort of, you know, re-regulation, but just simply if there are updates,
- 16:59
you choose when you're opting into whatever it is on your entire stack. Yeah. No, so [clears throat]
- 17:06
when you think about, um, this wave that we're seeing here of like this civilizational infrastructure that is called AI, you have to consider the, the potential of, you know, things being taken away from you and your sovereignty and, uh, how can you be in charge of, you know, this thing full stack end-to-end.
- 17:31
That's hardware, software, and everything in between. That's the model weights. That's the specific version of the model, as you were saying. That's how you fine-tune it. And picking up of, mm, Joseph's, you know, thing, um, [clears throat] statement here about small and specialized models, I have been v- like, you know, I have been a proponent of that for quite
- 17:51
some time. I have tweets about that in twenty twenty-four saying small and specialized models are the future. And it is gonna be per user, by u- per use case, per workflow.
- 18:02
For businesses, you know, to, to decide on their niche domains, and you need to start onboarding them now because you wanna collect the data points, the traces. That's what you're asking about, Nader.
- 18:11
You were asking about how do we get there? How do we decide which mo- model gets routed to? How do we decide which use case gets... goes to which model or which, which endpoint?
- 18:21
You need to collect data, and you need to be ready to basically break down, um, the, the traces that you've collected from your employees, from everybody in your organization, and decide how can we make the most optimal use case of these data points to which models, and collit- collect feedback as well.
- 18:39
And you can actually automate that with agents as well. So like, this is like something that it's re- you know, the, the whole thing about RSI right now, recursive self-improvement.
- 18:48
Mm-hmm.
- 18:48
It also applies to even agents and harnesses and to, to workflows and to use cases and to enterprises very well.
- 18:55
Do you wanna describe that for everyone too? Maybe for folks who don't know RSI.
- 18:59
Uh, recursive self-improvement. It means that, uh, a model would basically rent its own compute and then start training its own checkpoints and then, y- you know, uh, deploy, uh, its next version so that it, uh, gets updated in certain ways, change its behavior in certain ways.
- 19:15
So basically, a model is training itself.
- 19:17
Totally. For our product Brev, which just makes it really easy to get a GPU-
- 19:20
Mm-hmm
- 19:20
... we've been focusing more on agents as a first-class audience because we're seeing this. So we're seeing growing usage from agents that just want to go grab a GPU, uh, directly.
- 19:29
Exactly.
- 19:29
So, yeah. You know what? It's funny. So the second thing you talked about was, uh, not just multi-model, but then all these optimizations to actually run it performant when you have less compute than in a, say, data center.
- 19:39
I think, Alex, this is some fun work that we did. So, um, Alex has actually set up, uh, a second headquarters inside of NVIDIA. [laughs] We got a conference room, and, uh, we were, we were having dinner, and he mentioned that he really was motivated to get to squeeze out every drop of performance we could on the DGX
- 19:56
Spark. And so we said, "Hey, let's get a conference room at NVIDIA. Come bring your team. And any time you need an expert across any pillar, let's just go and pull that person into the room."
- 20:06
And then, yeah.
- 20:06
I, I actually have a follow-up here. Um, so since I have been part of LocalLlama for quite some time, um, when I think about home labbers, I think about the b- individuals that are actually, uh, are within enterprises, on boardrooms, making decisions about, you know, the models that we're gonna host or w- you know, where we're gonna
- 20:24
get our AI from. Um, the things that these home labbers focus on are optimizations. It's basically how can I extract the most economical value out of the hardware, the constraints that I have, the software that I have.
- 20:36
You know, quantizations came to be because of that. When we're thinking about enterprises, it's the same thing. It's constraints. It's budget limitations.
- 20:43
Mm-hmm.
- 20:43
It's how can I make the most use of the number of tokens that I can get under the hardware that I have, whether that's a DGX station, you know, um, centrally in an, a small, middle-sized business or, you know, a DGX P300 cluster.
- 20:58
Yeah.
- 20:58
It's all about how can I optimize the software for the latest and greatest frontier open source model out there and get as much value as possible out of it under the economical constraints that we have.
- 21:08
100%.
- 21:08
I think it's, uh, it's crucial to see home labbers, and that's why, you know, this is local AI. It's local AI, but it can, it could be on premises.
- 21:16
It could be co-located hardware. It could be rented clusters. It could be from Brev-
- 21:21
Yeah
- 21:21
... basically. It's just the idea about controlling this thing end to end, from hardware to software to model endpoints-
- 21:27
Totally
- 21:27
... to model weights and, you know, collecting all of that data to train your specialized and smaller models that will be more efficient as, as you go. Um, yeah, the future is great, and the future is local. [laughs]
- 21:39
Yeah. Yeah, absolutely. Um, you know, making sure it's your dates, your, your weights, your compute. I think one thing that was really interesting, we did end up getting 10X performant improvements on the DGX Spark, and we sent an update to Jensen, to the executive staff, to the teams that we're helping.
- 21:54
And one line I really liked in that email was that we didn't solve any new computer science to do this. We actually took things that the experts at NVIDIA had already solved and was out there, and I think what we worked together to do really nicely was assemble it in a bouquet.
- 22:08
And I think that speaks to some of the, the usability here, right? We're talking about what is capable, where, what are the capabilities, but do you feel like it's just capabilities that are holding people back from adopting this?
- 22:18
Yeah. So just to tell the story a little bit what happened here. So I think we had dinner on Thursday of the week, and then, you know, an email got sent out on Friday.
- 22:29
So this idea came up, like, "Why don't we do a lab," right? Why don't we like... You know, this to me sounded crazy at the time. I didn't think this would be possible, but it was like, you know, let...
- 22:37
Like, Nader was like, "Oh, let's, like, get a bunch of people from NVIDIA to help you guys and work with you guys to improve the performance on the Spark."
- 22:45
Right? And I was like, "Okay," like let, yeah, let's... I mean, if we can, that'd be great. Email got sent out, I think, pretty much that night, and then on Friday, Nader told me, "Be here on Monday, um, at NVIDIA HQ.
- 22:59
Uh, we're gonna have, you know, teams of people here that are gonna work together with you on this." And, you know, turned up on Monday, and, uh, Jensen talks about this concept of swarming.
- 23:11
Uh, it's, you know, basically the idea that, uh, the, the whole company will, like, mobilize around something.
- 23:18
Have you ever seen little kids play soccer? There's a ball, and everyone just attacks it. [laughs] It's like a tackle.
- 23:22
Literally that.
- 23:22
It's just the mosh pit of the soccer ball.
- 23:24
Yeah. And I had heard about this idea, but, like, I'd never experienced it. And I can tell you, like, we were just back to back that whole day, people coming in and out of the room, you know, from all various teams, like the Nemotron team, um, you know, people working in, in, on data center stuff, people working,
- 23:39
um, on vLLM. I, I realized, you know, NVIDIA has a team for everything, so there's like a vLLM for Spark team, which is oddly specific. [laughs] But like, you know, they have all these teams, and, you know, we basically got to pull those resources in.
- 23:54
And, you know, you know, in those three weeks or so, as you mentioned, like, we basically got 10X performance versus, you know, what, um, NVIDIA had running on the Spark- Um, in their existing playbook, which was using Hermes Agent.
- 24:09
So we did a bunch of optimizations there, um, using VLLM, um, as the, the sort of inference backend there. Uh, doing a lot of work with the, like tuning the models, like quantizing the models, uh, to, you know, be fit for local.
- 24:25
So what you'll find is, I think one of the great things NVIDIA has done with the local hardware is it's the same architecture that's running in the data center as running on the Spark.
- 24:36
So it's Grace Blackwell. Um, so the hardware is like fundamentally the same, meaning you actually get a lot of things for free. So for example, like the kernels are already really good.
- 24:45
However, there is like a lot of tuning and a lot of like configuration that is right now designed specifically for the data center. So a lot of the work that we did was not like inventing anything new, but it was literally just tweaking things to work, uh, more performantly on the Spark.
- 25:03
And, you know, the hardware is extremely capable. It's like, you know, it is data center level hardware that literally can sit on your desk. Um, so it's just about like, you know, how do we activate that?
- 25:13
And I think this has been one of my learnings from the last few years, is like we already have the hardware. Like the hardware is already really good and, you know, the models are getting better.
- 25:24
Um, they're getting a lot better at compression, so you can fit more on a smaller device. So like, you know, I think it really is just a matter of more people looking at the space and working in it and, you know, we will be able to do amazing things.
- 25:36
Like we have, you know, Nemotron 3 Ultra. It's a 550 billion parameter model running on four Sparks over that, 30 tokens per second. So like, that's-
- 25:45
That's huge
- 25:46
... pretty easy.
- 25:46
I want to say something here. Like it's, it's so great to see the community being proactive. I think of Alex Cheema right here with Exo as somebody who's a member of the community who's trying to push the frontier of open source intelligence to the next level.
- 26:00
I love how NVIDIA is working with them on that. When I think about, you know, this space, I think, uh, we're in the '90s of the Linux operating system-
- 26:08
Totally
- 26:08
... and we are like just starting. The infrastructure is not there yet. We need so much more. Um, I was even telling Alex I want to see how we can collaborate.
- 26:16
We have like an open source, uh, uh, sys- deployment system called ODS that is a bunch of open source tools that we deploy in each hardware. It fig- configures stuff, configures agents, and basically, um, gets you set like, you know, get, gets you set up with the entire infrastructure end to end that you need for your agents
- 26:33
locally. This kind of stuff, we need that optimized for every piece of hardware out there so that we can get more and more people onboarded, have the open source adoption, you know, just go to the moon.
- 26:47
That's what I want right now. This is why we're here. We want everybody to know that local, local and open source AI can run on anything, starting from your phone to your DGX Spark, as you were saying, Alex, to DGX stations, to the next level-
- 27:01
Right
- 27:01
... in the data centers, and it will deliver you frontier level intelligence.
- 27:05
Yeah, absolutely. I think this is a really good question for you, Matt. You know, you test everything for a mainstream audience. You're hearing about all of this technology, you're using this aggressively.
- 27:14
Where does it still fall short for the, uh, average user for AI?
- 27:18
So I, [clears throat] I think about two things, the average kind of personal user, then the average enterprise using, uh, open source. I mean, it, it, it, it needs to basically be as simple as opening Cursor.
- 27:31
Yep.
- 27:31
It needs to be maybe slightly more complicated than that, or slightly more complicated than just installing Codex and opening it up. Right now, it, to be fair, it is quite far from that.
- 27:41
It, like the, the stuff these guys are doing is incredible, but it is more sophisticated than what most people, including myself, are gonna be capable of. [laughs] Um, y- you know, let alone a, a, a business.
- 27:54
Uh, I, I think-
- 27:54
Need a full-time job to just do this kind of stuff.
- 27:57
Totally. Yeah.
- 27:57
That's why we need to automate it. [laughs]
- 27:58
And it, it really does need to be point and click.
- 28:01
Yeah.
- 28:01
And once it gets there... And there's, there's a lot of great open source projects, there's a lot of great projects in general that are getting there, but we're still not quite there.
- 28:09
Uh, the other thing to make it widely adopted is to allow people to better understand what use cases are appropriate for whit- what type of model, for what type of harness, what type of hardware.
- 28:21
Knowing exactly the use case that I can get out of my home system or something that I'm, I'm, you know, renting from a, a service center, uh, that, that is inc- incredibly important as well.
- 28:34
And I, I think it shouldn't be just in documentation. I think that's where open source becomes so difficult for people. I think it needs to be a point and click, and it figures it out on its own.
- 28:43
Yes.
- 28:44
That's what ODS is about. That's what I think Exo is about. That's what, you know, as we grow more and more and, and building this infrastructure, we need to be thinking about the user experience for your average everyday user, not us as technical folks here.
- 28:57
Because this is AI engineering, um, you know, summit, and I'm pretty sure that we all can manage our way around this stuff. But your ChatGPT users, your Claude users, whatever out there, they want this to be an alternative that is just click, play, send a message, use an agent, and done.
- 29:15
Yeah. Most, most people really don't wanna know about the details. [laughs] They, they just want it to work. And even if it worked seamlessly within Cursor or Codex or any of these other things, and it just worked and they didn't have to think about it, th- that is number one-
- 29:28
Yeah
- 29:28
... uh, for, for the vast majority of people.
- 29:30
So interesting on that, in a multi-model world where we have to pick a model, do you think that that's something that is a product, um, as we're... or is that, is that plumbing?
- 29:39
Is that something-
- 29:39
Yeah. Nader, you talked about the harness.
- 29:41
Yeah.
- 29:41
It, it's exactly that. It should be some, like whatever the kind of the front entry point to somebody's AI experience is, that should be what is choosing w- which model at the right time for the right use case.
- 29:52
This is a very difficult problem. There are, uh, you know, open source projects, closed source, uh, companies. Building routing, that's only one piece of it.
- 30:00
Yeah.
- 30:00
Right? So, go ahead.
- 30:02
No, like, uh, Carter from NVIDIA, he's, he's gonna be a moderator here.
- 30:05
Hey. [laughs]
- 30:05
Hey, Carter. Uh, looking good, man. So, um, he, uh, I, I, I shared ODS with him, and he told me, "I love how, like, it immediately downloaded that two billion parameter models, allowed me to start playing with it, and then it started downloading the next model that would work perfectly on, on my device."
- 30:20
That's the kind of experience we need to be giving these users when they-- we're onboarding them. Don't make them sit down and have to think about all these quantizations and all these extensions and all these weird things.
- 30:30
That is too much work for your average user. That's how we lose them.
- 30:34
Mm-hmm. Yeah, it's interesting, right? Like, the-- When you're not having to deal with multimodal and you have just this generally good but big model, you can kind of ask it anything.
- 30:43
You don't have to be as sophisticated, or you don't have to understand your use case too well. But as we talk about specialized models, if you're gonna specialize in something, you need to know what that something is.
- 30:52
Yeah.
- 30:52
And so the amount that... I mean, can you guys speak to maybe what it's like to move into a specialized model or to build one? Or have you, have you-
- 31:00
I, I can start with that. Uh, on the cloud, uh, you're basically getting the normal distribution. So every time that they are training something, they are taking the average feedback from everybody when they are being happy or sad or like, you know, um, they are okay with the answer or, "Hey, no, this is not what I meant.
- 31:15
Go back or change the model, and answer with a different model." That's the kind of feedback that allows you, you know, cloud providers to fine-tune the next model. But if we're talking about small and specialized models for use cases, that's a lot of compute to train models.
- 31:28
That's a lot of, um, storage for these weights. Uh, that does not happen without each use case or each business entity, et cetera, focusing on their own, um, patterns and use cases and, like, how they handle agents and how their employees, uh, message these agents and what workflows they are interested in.
- 31:47
So it's really, you know, it's per c- use case, it's per entity. It, it's not something that we can just generalize, and that's why, you know, continual learning is not being talked about by the frontier model, uh, as much.
- 31:59
It's coming though. It will happen, and it needs to be running on local hardware for it to happen. That's how you can have something that is so optimized that is not just plotted markdown files sitting in one REPL and, you know, you think that that agent is not gonna lose track of which scale to update or what
- 32:16
to edit or which memories to update. Because at some point, context lens becomes inefficient, and that's one issue. That's the current paradigm of agents is basically just saving to markdowns.
- 32:27
The next one would be updating the weights, and that needs to happen locally.
- 32:31
Yeah. There's-- It's funny too 'cause, you know, uh, there was a tweet I think by [REDACTED:username], uh, that was like, "Why has fine-tuning as a service not taken off?"
- 32:37
It was, like, a couple months ago. Um, and I think a big reason for that is that model customization itself is also a very hard problem. And then what models can you hack on?
- 32:46
Uh, that's-- I mean, that's why NVIDIA releases Nemotron. It's a open source model, but everything from the dates-- Sorry, the data to the weights to the, uh, you know, uh, the recipes for how to do so and then, of course, you know, the final model, everything is open sourced, um, just so that it is a model that
- 33:00
you know you can safely, uh, use and customize. You know, it's interesting. I keep kind of flip-flopping on this point. I, I, I think, like, people look at how good the generalized models are, and you give it the right context.
- 33:12
Is that gonna be better than, than having a fine-tuned model and all the work that comes with that? And I, I keep going back and forth 'cause there is a lot of value in being able to have kind of a smaller, very specialized model and, and maybe a bunch of them working in unison to accomplish whatever task
- 33:28
you have. Um, but I, I-- Yeah, I'm not sure. What, what do you, what do you guys think about that?
- 33:32
Yeah.
- 33:32
I think, um, if I can chime in on this one, like, I think the, the beauty of, like, the open source ecosystem is that, like, all of these paths kind of get to be explored.
- 33:40
Yeah.
- 33:40
And then whatever wins, like, you know, it's-- it-- it's like what happened with speculative decoding. Um, you know, there's all these, like, different ways of doing speculative decoding, which is this idea that you can use a smaller model to basically approximate a larger model to speed it up.
- 33:59
Um, and, you know, you've seen, like, recently, literally I think in the last week, there's been, like, three different, like, quite, you know, seem to be breakthroughs in, like, speculative decoding that have just come from, like, different places, one from DeepSeek.
- 34:14
Um, there was some work that was done by, like, Modal and, uh, the SGLang team to, like, build, uh, DFlash-
- 34:21
Yeah
- 34:21
... draft models for, like, various Qwen models that are, are like a big improvement on the previous ones. Um, there's, like, all this work being done. And so, like, we kind, kinda get to see, like, all of it and, you know, basically, like, whatever ends up being the best thing will just be the thing that kind of
- 34:35
wins, right?
- 34:36
That's kind of-- That's a great point where maybe it's not users and consumers actually customizing their own models and using the specialized models, but someone who has a need does so and does so in an open source fashion so that someone else can just adopt it.
- 34:46
Yeah.
- 34:47
Right? If I'm gonna start to do image gen for a particular use case, I can just go find a model. Actually, we saw this a lot also with, like, early LoRAs on, like, a lot of image gen models.
- 34:56
I feel like that was really big as, you know, in the 4.0 era as well. Um- I think ultimately what the end user cares about is, does it solve my use case, and can it do so within my budget?
- 35:07
Yeah.
- 35:07
And if there's another service that can manage the entire fine-tuning process, and, and I don't have to think... I know I keep saying I don't have to think about it.
- 35:13
Sometimes I do think about things. But, like, in this specific use case, um-
- 35:17
I was thinking about too many things already to add one more thing to it.
- 35:20
Yeah. There's-- Like, I, I care about my business. I don't wanna think about the, uh... I want it to be abstracted away from what I'm worrying about day to day.
- 35:28
Totally.
- 35:29
I could, uh, describe maybe a common flow that we see that's, like, distillation or, uh, general model, specific model. So if you think about it, um, in a lot of cases, it's like if the, the challenge I like to pose that's, like, a thought experiment is, like, if you po- tell someone, "Think of any object," and it's
- 35:45
like, "Think of an object in your fridge," the latter of those is actually easier, and the latter of those is actually where a lot of models kinda get deployed into a real world setting.
- 35:52
So, so for example, um, you know the Monterey Bay and the Monterey Bay Aquarium Research Institute, MBARI they're called? They discovered a new fish species recently, and they built with Roboflow to process all of the underwater, um, data ...
- 36:05
footage that they capture from their deep-sea, um, submarines. And they use large models like Segment Anything 3 and LMs as Judge to basically say, "Hey, let's take all this video footage, and let's build a auto label pipeline to understand as many things as we can about all the video footage that we've collected."
- 36:21
But then ultimately, you know, Sam knows everything about, uh, you know, things from fish to maybe architectural diagrams to, um, items in your fridge and everything in between. But they only care about things that are, you know, in this case, underwater deep-sea exploration.
- 36:35
And so a very common flow that we see for, like, the fine-tune last mile distillation is, okay, let's take the large context of an array of models, have those maybe all have a pass at understanding something, use LMs as Judge to say, "Hey, we agree with some amount of consensus," and now we have a specialized dataset.
- 36:51
Then we can use that specific model and actually run that on the submarine in real time and also post-processing for faster video. And if you think about it, like, a lot of problems are of that shape, where it's like, yeah, I want, like, general as much intelligence to understand the thing, and then ultimately the problem I'm solving
- 37:05
is specific enough, even if not fully unbounded. And a mistake I see pretty frequently is people thinking, like, Sam 3, which is an awesome model and a great family of models.
- 37:14
It's like, okay, well, I should take Sam 3 and then maybe just fine-tune Sam 3. And in some ways that actually doesn't make as much sense because you lose the thing that makes Sam 3 awesome, which is the open vocabulary capabilities.
- 37:24
What might be better is, like, if you know you're distilling down to a specific fixed class list, then you can actually drop the large, expensive autoencoder portion of Sam and use a specific maybe, like, debtor or more specialized model, depending on the task that you're solving, and get all the benefits of, you know, speed up and accuracy,
- 37:41
uh, while still having the general knowledge of preparing and curating your problem. And I think a lot of problems are of that shape.
- 37:47
Are you, are you managing that pipeline for your customer?
- 37:51
The-- Yeah, the tooling makes it so they can do that. Um-
- 37:55
Do you do it on their behalf?
- 37:57
Uh-
- 37:57
Or do you give them the tooling to do it?
- 37:59
Give them the platform to do that, and then there's, like, recipes-
- 38:02
Yeah
- 38:02
... where someone can go and do that. And then like any good AI company, there's an FDE that if you want, I can sell you. [laughs] We would get around back here, and they can do it for you.
- 38:09
Uh-
- 38:10
That, that's a great example basically of the use cases and workflows that I was talking about when you're trying to find like, you know, collect the data as you go for your business, for your enterprise, and then decide how you're gonna fine-tune a model, a small specialized model on those use cases as you grow.
- 38:28
That's how you become more token efficient.
- 38:30
Yeah.
- 38:31
Yeah. I, I, I think th-this year and, and next year you're gonna see a lot of using these, like, monster frontier models to bootstrap, you know, like a more efficient setup that runs on open source.
- 38:42
Yep.
- 38:42
And I think, you know, that's great. Like, I think this is how a lot of the open source models have been built right now, and it's, it's, it's proving to be, like, quite hard to, you know, stop that.
- 38:53
And I would just enc- [chuckles] you know, encourage people to just, you know, move away potentially from like, you know, these, uh, frontier models, but like, you know, use them.
- 39:01
Use them for what-- for, for bootstrapping that, right? So like, um-
- 39:05
Love that you said that actually because that's how the word AI engineer got coined, right? When Swyx released that blog post that coined it, the way that we used to build AI products was we would start with the m- with the machine learning.
- 39:16
We would start by training a model, then we would go try to discover a use case.
- 39:19
Yeah.
- 39:19
And what these big models allowed us to do is actually flip it. We said we could start with discovering a use case and then get into ML if it makes sense.
- 39:28
And that was actually, uh, honestly an amazing foresight from Swyx because he kind of defined that pattern, uh, for us a few years ago and gave this amazing conference as well.
- 39:37
Yeah, it's crazy how far this conference has come in just three years.
- 39:40
Yeah, totally. I have a question. Do you guys have questions in the, in the crowd? We have a few more minutes, and I know we were talking about opening this up if you guys wanna ask the panelists directly.
- 39:49
Just, just shout.
- 39:51
Yeah. [laughs]
- 39:52
Please go ahead.
- 39:52
Um, what are the big open problems in local AI? I feel like you're the luminaries of the field. You know more, you have a line of sight on, like, the biggest open problems.
- 40:01
You wanna repeat the question?
- 40:02
I'll repeat that one.
- 40:03
He just asked, what are the open problems in local AI?
- 40:05
I, I can repeat it.
- 40:06
Yeah.
- 40:06
It's what are the biggest open problems in local AI? Um-
- 40:10
Um, it continues to be optimizations and, uh, for inference. It continues to be getting things easily kickstarted, which is-
- 40:19
Yeah
- 40:19
... you know, what ODS is, XOS for specific hardware. It continues to be how can I make the most out of my budget constraints and hardware constraints, and how can I squeeze the most performance out of that?
- 40:33
Uh, you know, the, the models that-- You know, I still have the same thirty-nineties that I used to run Llama two and like Dolphin fine-tunes on, and now they're running Qwen 3.5, Qwen 3.6, twenty-seven billion parameters with excellent performance.
- 40:46
You know, more on that in my next presentation.
- 40:48
Yeah.
- 40:48
So, you know, um, it, it comes down to the optimization, and we are still very early that there is space for so many players, for so many contributors. We need all the help we can get to make this thing the success that it needs to be, that we need to make local AI the default.
- 41:04
This has been my stance for years now. I have been saying open source AI must win for s- like since forever, and the way we do that is by giving the people, whether that's individuals at home, middle-sized businesses or enterprises, an easy way to use these models in a very efficient way that doesn't give them headaches more
- 41:24
than solve their problems.
- 41:25
Totally. And if you look at the panels that are, that are happening today, those are what we believe to be the biggest open-ended questions, which is why we tried to assemble this panel.
- 41:32
And so the-- you know, we have quantization, so all about talking about the footprint so that the models do fit on these, uh, uh, on these smaller hardware footprints.
- 41:40
Uh, there's model routing, there's models generally. Uh, so those are kind of the-- what we feel like are the big problems now. And as we do future local AI summits, um, I-- you know, every time the panels should change to be the topics du jour that are kind of holding the...
- 41:54
Or, or we're gonna be-- gonna usher the next chapter in.
- 41:57
Can I add one thing to that? One of the biggest challenge-- So I think there's two, two big challenges. One is what we've been talking about of basically the trade-off of simplicity versus customizability.
- 42:07
It's local, it's yours. You can do different things, optimization. If it's hosted, it's built out of the box the way. That trade-off is, is always difficult. The second, which I actually encourage people in this room to help solve, is the importance of open models is becoming increasingly in question.
- 42:22
And I actually think that, like, if you think Local AI is important, then you think open-source AI is important, and it's actually really important to be an advocate for being able to use, change, adapt, and toy with models.
- 42:32
And so I think that that's a problem that, um, could increasingly be something that we feel less control over absent advocating.
- 42:41
That's a great point. I think everyone here feels very passionate about open source. That is why we actually have access to the space at all. That's why the space has, has progressed.
- 42:49
It's a, it's a necessary competitive environment. It allows for the best ideas to make their way to everybody. Um, so definitely when there's talks about that being a threat, you know, we need to invest and advocate for open source.
- 43:00
And open source is so much bigger than AI, right? Like, there's a reason why computation-- like compute was invented in, on the East Coast, but Silicon Valley happened here, and it's because hippies realized that they could share ideas for free and software. [laughs]
- 43:13
I think Silicon Valley is this kind of tension between the capitalists that wanna make money and hippies that wanna give these ideas away for free, and that tension is what creates such an incredible environment here.
- 43:22
They can coexist, by the way. [laughs]
- 43:23
And they, they do.
- 43:24
You can have consumers, and you can sell to businesses.
- 43:26
And that's when the space works, works best, and so-
- 43:28
Ca- can I just say, like, if you care about this, and you wanna be more of an active participant, but maybe, you know, you don't wanna get involved in the technical side or you're not technical, then there are ways to get involved.
- 43:36
So there's a website that just came out called RightToIntelligence.org, and this is a way for you to, like, get involved and actually, like, advocate for open source and to ensure that, you know, we maintain freedom of intelligence.
- 43:49
Awesome. So we're out of time. I wanna thank you guys so much. Local AI Summit's gonna be a ton of fun. Thank you guys for this incredible way to kick this off.
- 43:56
We're gonna have amazing demos. We're running foundational intelligence models here inside of this room. Uh, we're gonna have a couple of amazing-- uh, a, a few more panels. It's gonna be an exciting day, um, and we're all lingering here, so ask questions and please participate.
- 44:09
Thank you. [audience applauding] [upbeat music]