State of the Union: Why Local, Why Now

How closely do you need to watch a coding agent? Nader Khalil opens with a striking change in the answer. He recalls Andrej Karpathy warning in November that agents still required close supervision, then describing difficulty keeping up with their capabilities three months later. Both assessments could be right: the models and the harnesses around them were improving together. Khalil’s practical response is to use AI a little more each day, so experience can keep pace with capability.

That progress also changes the workload. A chatbot generates a burst of text in response to a question. A reasoning model first spends compute generating tokens the user does not directly consume. An agent can keep working beyond either interaction—and, given enough useful work, its owner may not want it to stop at all.

The more useful these systems become, the more sensitive context people want to give them: enterprise intellectual property, medical records, health data and home-camera footage. Continuous generation also makes token costs harder to ignore. Local inference brings the data boundary and the compute budget under the operator’s control. Khalil describes this as a plateau in token spending on owned compute, rather than an ever-growing metered bill, and says the room’s demonstrations keep their processing on the devices present. That is a bounded-capacity deployment proposition, not a claim that hardware and operation are free.

The panel approaches that proposition from several directions. Alex Cheema introduces himself as co-founder and CEO of EXO Labs and creator of Local.ai, with more than two years spent making local inference accessible. His opening comparison runs from an early large-Llama experiment on two MacBooks to Nemotron Ultra on four Sparks. Matthew Berman brings the perspective of an enthusiast who tests AI for videos and a newsletter. Ahmad Osman, Osmantic’s founder and a LocalLLaMA moderator active in the space since 2022, jokes that an interpretation of his name as Open Source Man became a company.

A passing reference to Fable returning introduces another reason to care about local access: a product built on frontier intelligence depends on that intelligence remaining available. Joseph Nelson, Roboflow’s co-founder and CEO, adds a physical constraint. Computer vision often needs to run alongside the camera producing the images, with sufficiently low latency to keep up with video. For vision developers, deployment near the data is already familiar.

For Berman, the first turning point was Llama: intelligence he could download and run in his office. Years of building and overclocking PCs made that possibility immediately compelling. Osman’s equivalent moment came with Llama 2 on an RTX 4090. Sampling parameters, configuration choices and inference engines became things he could inspect and change. The appeal was both practical and educational: local execution made the black box something he could experiment with, while leaving room for cloud systems to coexist.

Khalil recalls an approximately 2023 airplane experiment in which a phone model took 20 minutes to finish a sentence. It was painfully slow, but it answered without an internet connection. Osman contrasts that experience with Qwen3.5-4B on an iPhone, which he describes as offering GPT-4o-equivalent quality. The panel supplies no matched evaluation or phone configuration for that comparison; its significance here is Osman’s assessment of how far pocket-sized inference has progressed.

Cheema separates the excitement of fitting a model onto local hardware from the experience of actually using it. Llama 405B narrowed the capability gap with closed frontier models, but his dense-model experiment was too slow for practical use. DeepSeek V3 and DeepSeek R1 marked another turning point: he credits mixture-of-experts models with making very large models more practical on hardware such as a Mac Studio or Spark. The distinction is between storing a large model and paying the full dense-model computation cost for each generated token.

The under-desk DGX Station example extends the same trajectory: increasingly capable models on hardware someone can control directly. Cheema expects compression and lower memory requirements to continue moving that boundary, eventually making local inference the default.

Nelson’s turning point was less about model size than a particular failure. On a flight, a passenger with limited vision was repeatedly photographing the cabin to understand the seating and how to buckle in. Apple’s accessibility feature described the seat back as something like a printer. The passenger knew from context that the answer could not be right.

Nelson tried the recently released LLaVA on a photo of the tray in front of him, and it correctly described an airplane seat tray. LLaVA connects a vision encoder with a language model through visual instruction tuning; the original paper appeared in April 2023. In this particular comparison, a broadly available model supplied a useful description where the phone’s built-in feature had failed. For Nelson, that was evidence that useful intelligence need not be monopolized by the largest technology companies.

The camera is essential to Nelson’s example: it gives the model access to the situation it must explain. Khalil connects that to the wider improvement in harnesses. A CLI can expose a business system to an agent; a coding environment can expose the filesystem. His earlier workflow involved choosing snippets, pasting them into ChatGPT, then copying the answer back. Cursor changed the division of labor by letting the agent inspect files and decide which context it needed. A better harness changes what the model can observe and do.

Vision also offers a lesson in specialization. Nelson contrasts language workloads that can often reach data-center compute with cameras and robots operating on limited hardware or weak connections. Those constraints encourage a model that performs well in a particular domain, rather than prioritizing world-scale generality. Language systems are following a similar path through coding-specific harnesses and last-mile adaptation for work such as tax or legal preparation.

Specializing the task does not eliminate the separate work of optimizing inference on the device. It does, however, change the question from which single model should do everything to which model belongs in each part of the system. Khalil describes NVIDIA’s expectation as a multi-model world, with routing important enough to receive its own later summit panel.

Berman already uses a mixture: a top-end model he calls Fable, Sonnet as a workhorse, and local models for work less sensitive to latency. He cites a Brian Armstrong post describing Coinbase’s rising token consumption with flat costs through a mixture of models. That is the budget problem in miniature: an enterprise wants more useful AI work without pricing every task at the most expensive tier.

A concrete division of labor is to let the strongest model plan the architecture, then assign coding subtasks to smaller, less expensive models. Local models can perform that execution while a frontier service handles the top-level plan. This gives the operator more control over the workflow and reserves expensive inference for the part expected to benefit most from it.

Cheema sees market demand for that flexibility. His illustrative objection is paying $50 per million tokens for a giant model when the workload does not require one. Enterprises also want sovereignty: the ability to replace a model and continue operating if a provider withdraws access or changes its terms. Those requirements make model choice an architectural concern, not merely a dropdown in a chat interface.

Routing creates its own engineering problems. The selected model needs the right context; a planner’s output must become workable sub-agent assignments; the system needs a framework for coordinating those assignments. Version control matters too. If model behavior changes, an operator needs to know which version ran and should be able to choose when to adopt an update. Local ownership can support that control, but the harness still has to implement it.

The sovereignty discussion extends across the stack: hardware, software, weights, model versions and fine-tuning. If AI becomes civilizational infrastructure, access to an endpoint is only one layer of control. The panel’s proposed destination is smaller models adapted to a user, use case or workflow, rather than one undifferentiated service for every need.

Osman’s immediate recommendation is to start collecting workflow evidence. A business cannot decide which tasks belong on which models merely by inspecting model names. It needs employee interactions, execution traces and feedback about whether the work succeeded. A practical sequence follows from his proposal:

1. Capture traces from real workflows and the feedback attached to them.
2. Separate the recurring tasks and use cases represented in those traces.
3. Use that evidence to decide which models or endpoints suit each task.
4. Continue collecting feedback as the routing and workflows change.

Agents could help analyze and improve this process, but the traces provide the basis for the decisions.

Osman then describes a more ambitious form of recursive self-improvement: a model obtaining compute, training new checkpoints and deploying its next version. This is a proposed progression beyond improving prompts or workflows. Khalil supplies a narrower observation from Brev: agents are increasingly requesting GPUs directly, which has led the product to treat agents as a first-class audience. Obtaining compute is one component of the proposed loop; it does not by itself establish autonomous model improvement.

The hardware discussion begins with a collaboration: Cheema wanted to extract more performance from DGX Spark, and Khalil offered a conference room at NVIDIA where relevant experts could join the effort. Osman connects that work to home labs. Enthusiasts trying to maximize the value of limited hardware often face the same problem as the enterprises where they work: obtain useful inference within a budget. He points to quantization as a central example of that optimization mindset.

The scale can range from a DGX Station serving a small or midsized business to a larger cluster. Osman’s definition of local is correspondingly broad: on-premises hardware, colocated machines and rented clusters can all participate, including compute obtained through Brev. The common thread is control from hardware and software through endpoints and weights, together with the ability to collect data for more efficient specialized models. Physical possession and end-to-end control are related, but they are not identical requirements.

Khalil introduces the Spark result as an exercise in assembling existing solutions. Cheema recounts the sequence: a Thursday dinner, rapid email coordination and a Monday start at NVIDIA headquarters. Teams moved through the room, including Nemotron, data-center and vLLM-for-Spark specialists. He calls this an example of Jensen Huang’s swarming approach; Khalil likens it to children converging on a soccer ball.

Cheema reports a 10× performance improvement over roughly three weeks relative to NVIDIA’s existing DGX Spark playbook using Hermes Agent. The panel does not define the workload or performance metric behind that multiplier. The described work used vLLM as the inference backend, quantized and tuned models for local operation, and adjusted configurations designed for data-center deployments.

Shared architecture made that adaptation valuable. Cheema points to Grace Blackwell and existing optimized kernels as a foundation the local system could reuse. The work was therefore less about inventing new computer science than identifying which settings and optimizations needed to change on Spark. Better compression then compounds the benefit by fitting more capable models onto smaller devices.

Cheema reports Nemotron 3 Ultra, a 550-billion-parameter model, running on four Sparks at 30 tokens per second. He does not give the prompt length, concurrency, quantization or timing method for that room demonstration. It is a separate reported result from the 10× playbook improvement.

Osman compares the state of local AI to Linux in the 1990s: promising capabilities, an active community and substantial infrastructure still to build. His Osmantic Deployment System, or ODS, bundles open-source tools and configures the services and agents needed for a local deployment. The goal is to package an entire working stack, with optimizations appropriate to each hardware target—from phones through Sparks and Stations to data centers.

Berman sets a demanding usability target: starting local AI should be roughly as easy as opening Cursor or installing and opening Codex. He considers the current sophisticated setups beyond what most people, including many enthusiasts, can manage. Setup is only half the problem. Users also need to know what their combination of model, harness and hardware can usefully do, whether the compute is at home or rented.

Documentation alone cannot deliver that experience. Osman wants the system to figure out the configuration; Berman wants it to work inside familiar tools without exposing infrastructure details. For an ordinary user, the interaction should be simple: start the application, send a message or give an agent a task. The technical choices still exist, but they should not all become prerequisites to the first useful result.

That leads to Khalil’s product-versus-plumbing question about routing. Berman places model selection inside the user’s front door to AI—the harness that understands the task and can choose the appropriate model. Routing alone is only one component of that experience. Osman illustrates the broader approach with Carter’s reported ODS onboarding: it first downloaded a two-billion-parameter model so he could begin interacting, then downloaded the next model suited to his device. The useful state arrived before the final download finished, without requiring the user to choose quantization formats or extensions.

Specialization makes a new demand on the builder: know what the model is specializing in. A broadly capable model can accept an open-ended request; a specialized system needs a defined task. Osman contrasts the aggregate feedback available to a cloud provider with the particular interaction patterns and workflows inside one business. Training compute and storage for customized weights make that distinction consequential. Each entity needs evidence about its own work.

Osman predicts that continual learning will move some agent memory from expanding collections of Markdown files into model weights. His concern is that accumulating skills and memories eventually creates an inefficient context burden and makes it harder for the agent to maintain the right files. He argues for local weight updates as the next step. That remains a proposed direction in the discussion, rather than a demonstrated replacement for file-based memory.

Berman recalls an unnamed person’s question about why fine-tuning as a service has not taken off, exposing the difficulty: customization itself is hard. The panel points to Nemotron releases that include data, weights and recipes as a way to make that work more accessible. But the central trade-off remains unresolved: how often does a smaller tuned model justify its development and maintenance cost when a strong general model already performs well with suitable context?

Cheema’s answer is that an open ecosystem can explore several paths at once. He uses speculative decoding as an example: a smaller model proposes tokens to help a larger model generate faster, with the larger model responsible for verification. He recalls several recent advances, including work from DeepSeek and work involving Modal, SGLang and DFlash draft models for Qwen. The Modal account credits Z Lab with DFlash’s architecture and describes Modal’s Qwen speculators and SGLang integration.

Reusable specialization changes who must do the hard work. Someone with a particular need can adapt a model and release it for others; early image-generation LoRAs provide a familiar example. A managed service could similarly handle fine-tuning for a business. The user’s test remains whether the system solves the task within budget, while keeping model operations out of the work they actually want to do.

Nelson makes specialization concrete with a small thought experiment: thinking of any object is harder than thinking of an object in your fridge. Real deployments often resemble the second request. They operate in a bounded environment, even when the models used to understand that environment have much broader knowledge.

His example is the Monterey Bay Aquarium Research Institute, or MBARI. Nelson reports that the institute used Roboflow to process deep-sea submarine footage in work associated with discovering a new fish species. He describes Segment Anything 3 and language models acting as judges in an automatic labeling pipeline. SAM 3’s broad visual knowledge is useful during preparation, but the deployment only needs to recognize the underwater subjects relevant to the research.

The pipeline he describes separates broad interpretation from narrow inference:

1. Run large, general models over collected footage to propose annotations.
2. Use language-model judges to assess agreement among those interpretations.
3. Turn the accepted annotations into a specialized dataset.
4. Build a task-specific model from that dataset for the submarine and later video processing.

Nelson describes the resulting specialized model as suitable for real-time submarine inference and faster post-processing. The large models contribute knowledge during dataset preparation without all becoming part of the deployed inference path.

This also changes which model should be adapted. Nelson cautions against automatically choosing to fine-tune SAM 3 simply because it helped label the data. If the destination has a fixed class list, its open-vocabulary breadth may no longer be the capability worth carrying into deployment. He proposes using DETR or another specialized model, avoiding what he calls SAM’s expensive autoencoder portion. The architectural lesson is to select the deployed model for the target task; his claim that fine-tuning sacrifices open-vocabulary capability is a caution about that choice, not an established property of every SAM fine-tune.

Who actually operates that specialization pipeline? Nelson says Roboflow supplies the platform and recipes so customers can do it themselves, with optional paid forward-deployed engineering help. Osman connects this back to workflow traces: collect data while the business uses AI, then use that evidence to build smaller models for recurring tasks and improve token efficiency.

Cheema expects more systems to use powerful frontier models to bootstrap efficient open-source deployments. His recommendation is to use those models during the transition, rather than requiring an immediate break from them. He also observes that preventing this kind of bootstrapping has proved difficult. The practical pattern is to spend broad capability on discovering and preparing the work, then move repeated execution to a system better matched to its constraints.

Khalil connects that reversal to Swyx’s writing about the AI engineer. In his account, the older sequence was to train a model and then search for a use case. Large general models made it possible to start with the use case and pursue additional machine learning only when the application justified it. A working application can produce the evidence for specialization. The specialized model becomes a response to an understood workload, rather than a prerequisite to discovering whether anyone needs the product.

An audience member asks for the biggest open problems in local AI. Osman returns to inference optimization, easy startup and extracting useful performance from a constrained budget. He points to hardware-specific deployment systems and to the continued usefulness of older GPUs: his RTX 3090s, once used for Llama 2 and Dolphin fine-tunes, now run newer Qwen models. He mentions Qwen 3.6 and 27-billion-parameter models with excellent performance, without providing a numerical measurement. The opportunity is not limited to developing new hardware; it includes making existing hardware easier and more efficient to use.

The intended beneficiaries span individuals, midsized businesses and enterprises. Osman’s standard is that efficient local AI should solve more problems than it creates. Khalil names the current technical agenda: quantization to fit models into smaller memory footprints, routing to allocate work appropriately, and continued model development. As those bottlenecks change, the questions organizing future summits should change too.

Nelson adds two challenges that cannot be reduced to tokens per second. One is the persistent trade-off between simplicity and customizability: local ownership enables changes and optimizations, but a hosted product can arrive with those decisions already made. The other is continued access to open models. The ability to use, change, adapt and experiment with a model depends on that access remaining available, which he argues requires advocacy as well as technical work.

Khalil frames open source as a competitive environment through which useful ideas reach more people. His analogy is Silicon Valley’s tension between profit-seeking businesses and people who want to share software freely. Osman emphasizes that the two can coexist: a project can serve consumers and sell to businesses. Cheema closes by naming Right to Intelligence as a route for people who want to advocate for open-model access without contributing technically. The freedom to operate and adapt the system is part of the infrastructure the community still has to build and preserve.