AI Engineer World's Fair 2026
Frontier results, on device - RL Nabors, Arize
Read the talk
Prototype Big, Deploy Small: Moving a Summary Feature On Device
A local model can replace a frontier call when the task is narrow, the acceptance criteria are explicit, and evaluation covers the whole application workflow.
From a talk by Rachel Lee Nabors (RL Nabors)
Before you start: Familiarity with JSON, language-model prompts, and basic application testing will help; no model-training experience is required.
Does this task need a frontier model?
What does every call to a frontier model cost your application—and does the task need that model at all? Rachel Lee Nabors approaches this question from web development: Firefox DevTools, W3C standards, Microsoft Edge, React, and three years consulting on web, AI, and UI before joining Arize. A seemingly small prompt edit or model migration can break an agent workflow, making evaluation as important as the initial model choice. Phoenix, Arize’s evaluation and observability tooling, provides the experiment environment used here.
Reaching for GPT-5 or Claude brings costs beyond the token bill. Remote inference sends data to another organization’s servers, introducing exposure, interception, and retention risks; Nabors points to business-data leaks as a consequence. Latency also changes the experience. Nabors cites research on response delays in LLM-powered virtual agents when describing four seconds as a believability threshold. That study concerns spoken interactions in VR, so it motivates a responsiveness budget rather than establishing a universal deadline for every AI interface.
The bill can grow even as individual tokens become cheaper: agentic and reasoning workflows make more calls and consume more tokens. Remote inference also makes connectivity a product dependency. An outage, unavailable Wi-Fi, or a deliberately disconnected secure environment can disable the feature entirely. These business, user, and environmental costs lead to the first design decision: identify the capability the task actually requires before choosing the model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Match the capability, then the device
A camera input does not automatically require a general multimodal model, and a microphone does not automatically require a language model. Nabors offers a practical starting map:
| Input or task | Candidate families |
|---|---|
| Camera and vision | MobileNet, YOLO, MediaPipe |
| Recorded audio | Whisper, wav2vec2 |
| Chat, translation, language analysis | Gemma, Qwen |
Choose a specialized model when the job is specialized. Choose a smaller language model when the application needs language generation or interpretation without the full breadth of a foundation model.
The boundary around small language models, or SLMs, is fuzzy. Nabors describes them as ranging from millions to billions of parameters, overlapping with the low end of larger models. The parameter-count dots in the presentation illustrate why this matters: the machine needed to hold and run a model changes dramatically with its size. A thread summary or hostile-behavior detector does not necessarily need broad historical and philosophical knowledge, or the ability to analyze images and audio together.
Quantization makes those smaller models easier to distribute. At FP16, one billion parameters requires about 2 GB for weights; 8-bit and 4-bit representations can roughly halve or quarter that storage. Runtime memory must also accommodate more than the weights, so the presentation’s device-fit grid is a sizing guide, described by Nabors as upper-bound estimates, rather than a guarantee that a particular model will run comfortably.
Nabors bought a Pixel 10 Pro specifically to explore its on-device model. The broader case for this deployment pattern appears in the NVIDIA-associated position paper Small Language Models are the Future of Agentic AI: many agent invocations are narrow enough for smaller, specialized models. Nabors’s illustrative energy chart places an SLM at about 25% of the LLM’s task energy and a task-specific model at about half the SLM’s amount. The paper supports the efficiency argument, but does not establish those fixed ratios for an identified workload and hardware setup.
Running locally removes the network round trip and the provider’s per-call inference fee. It also enables offline operation after the model is available and keeps model inputs on the device. Those advantages make local inference attractive, but actual speed still depends on the model and the hardware executing it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Can a local model reason about ichthyosaurs?
A side-by-side conversation makes the capability question concrete. Nabors runs Gemma in Goose, an open-source agent harness, beside Claude. The local response takes longer in this example—a reminder that removing network latency does not eliminate generation time.
The question is whether ichthyosaurs, marine reptiles with dolphin-like skeletons, could have used echolocation. A useful response must move beyond visual resemblance and reason about the biological structures involved. The local model, which Nabors tentatively identifies as Gemma 3, points to the lack of fossil evidence for a melon and the specialized sound-conducting bones found in dolphins. Nabors finds this a good response; Claude sometimes retreats to uncertainty about what can be known. This informal probe suggests that a local model can supply useful reasoning, but choosing a production model requires a repeatable task evaluation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Prototype the feature and capture acceptable outputs
Nabors’s model-selection framework, developed with Google and described in Choose a small, efficient model, begins with prototype big, deploy small. Start with a highly capable model to find out whether the feature is feasible. For handwriting or writing-style recognition, that might mean Gemini or a strong task-specific model. Success establishes a target that smaller candidates can then attempt to meet; it does not establish in advance which candidate will succeed.
The worked application is Mima, Nabors’s client for multiple social networks. Its summary feature addresses a familiar problem: waking up to 50 comments on a post and wanting to know what people are discussing—and whether they are angry. Nabors first prototypes the feature with Claude and judges its summaries good enough to make the feature useful. Those accepted outputs become the starting point for evaluation.
A golden dataset pairs representative inputs with the outputs the application should produce. Ideally, people curate and label those examples. Here, Nabors accepts Claude-generated summaries after reviewing their adequacy. The public thread records include author handles, message content, creation times, and a flag identifying messages written by Nabors. The dataset is exported as JSONL, with two output tasks for each thread: a short summary and a summary containing references deeper into the conversation. The dataset contains 14 threads and 28 examples across those two output tasks.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn product requirements into evaluation checks
Before comparing models, define what a usable summary must do. Some requirements are deterministic; others require judgment.
| Requirement | Evaluation |
|---|---|
| Parseable JSON | Attempt JSON.parse |
| Valid references | Check that referenced messages exist |
| Factual consistency | Compare summary meaning with the thread |
| Length compliance | Check the word count |
| Responsiveness | Measure P50 and P95 latency |
Parsing checks JSON syntax, not whether the result has the right fields or points to real messages. Factual consistency is a different question again: a conversation about cats should not become a summary about puppies. Nabors uses an LLM judge for that semantic assessment; human review is another option. P50 is median latency, while P95 describes the tail of the latency distribution, not its absolute worst case.
With those checks in place, test candidates from small to large. In Phoenix, a capability evaluation compares what each candidate can do against the reference outputs and acceptance criteria. Nabors introduces this with an illustrative comparison between Claude Opus and Gemma; the actual experiment then establishes a Sonnet baseline. Phoenix is presented as free and open source in the talk; its current distribution specifies the Elastic License 2.0.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Find the small and good enough model
The actual baseline uses Claude Sonnet. Nabors reports an average latency of 2.9 seconds and a cost of about $0.22 for 14 tasks. Extrapolating from personal Mima usage gives about $1 per day in inference spending, a meaningful subsidy if the application grows. The local candidates show zero provider inference charges because computation moves to the user’s device. That transfers the energy cost to the user’s battery and charging, rather than making computation free; battery drain was left for future evaluation.
The local shortlist spans several sizes. These are the parameter counts and disk sizes Nabors reports for the candidates:
| Candidate | Parameters | Disk size |
|---|---|---|
| Qwen 2.5 Instruct | 1.5B | About 1 GB |
| Qwen 3 | 1.7B | Not stated |
| Llama 3.2 | 3B | About 2 GB |
| Gemma 4 E2B | About 5B | 3.1 GB |
The later official Gemma 4 model card distinguishes effective parameters from total parameters including embeddings; it does not identify the exact artifact tested here. Gemma was the enthusiastic recommendation from several engineers Nabors consulted, but its performance in this application made that recommendation a poor substitute for measurement.
SAGE means “small and good enough”: select the smallest model that produces acceptable responses for the application. Qwen 2.5 initially looks attractive on speed. In the model comparison, Nabors reports roughly one-second P50 latency for Qwen 2.5, roughly three-second latency for Sonnet, roughly 90% accuracy for Llama 3.2, and roughly eight-second latency for Gemma 4 E2B. On the plot, these appear as the gray Qwen circle, blue Sonnet diamond, large green Llama circle, and orange Gemma square. Qwen’s weaker accuracy changes the decision; Llama offers the more useful balance.
Nabors tested each model three times and averaged the results. A different prompt might have helped Gemma, but under the tested prompts its slower behavior was consistent. These are application-specific measurements: hardware, runtime, quantization, exact cloud-model versions, and accuracy aggregation are not specified. The next check is therefore to open the Phoenix experiment and inspect raw outputs beside expected outputs. Nabors finds many Llama summaries effectively indistinguishable from Claude’s and selects Llama 3.2. Meta’s interest in social-network content offers a possible explanation for the fit, but that is Nabors’s interpretation rather than evidence about the cause of its performance.
The selection procedure is repeatable:
- Prove feasibility with a sufficiently capable model.
- Set the bar with representative inputs, acceptable outputs, and success criteria.
- Test from small to large, comparing each candidate against that bar.
- Select SAGE, the smallest candidate that meets the application’s requirements.
This gives model size a concrete role in the decision without allowing it to replace output quality.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Improve the prompt before replacing the weights
Selecting Llama leaves a quality gap to close. A distilled model is one possible route, but changing model weights carries a distribution cost: adding capabilities may require retraining and asking mobile users to download another large model. Data plans and download friction constrain that approach. An established team may also be committed to Gemma 3 until a later model update passes its evaluations. Prompt optimization is useful when the deployed model is difficult to replace.
Nabors focuses on JSON and reference structural validity, factual consistency, and latency. The displayed acceptance targets are at least 99% structural validity, at least 95% factual consistency, P50 latency at most 1.5 seconds, and P95 latency at most 3.5 seconds. The latency budgets leave room below the response-delay threshold that motivated the feature’s experience requirements. These targets turn “close the gap” into a set of checks for the next experiment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Test one prompt hypothesis at a time
The prompt experiment keeps the original baseline and adds four variants, each testing a specific hypothesis:
- Numbered input: Replace the JSON thread with numbered messages. The hypothesis is that a smaller model may track natural-language indexing more reliably than array offsets inside JSON.
- Few-shot examples: Add thread-and-output pairs. The hypothesis is that examples teach the desired format more effectively than rules alone.
- Strict rules: Add literal constraints, including no preamble and counting words before responding. The hypothesis is that explicit commands improve compliance.
- Chain of thought: Ask the model to identify key moments before writing the summary. The hypothesis is that this intermediate work improves grounding.
Nabors runs the variants locally with Llama 3.2, holding the model fixed so the prompt changes can be compared. The resulting measurements can also be taken into a conversation with Claude to explore the tradeoffs.
The baseline struggles to keep summaries short enough for their allotted interface space. Nabors reports baseline reference accuracy of 91.2%, factual correctness of 87.1%, and latency of about one second. Reformatting the input makes little difference, and explicit rules worsen the results. Chain of thought improves length slightly but adds about 600 ms of latency. Few-shot prompting improves length, references, and agreement with Claude while adding about 200 ms. Examples provide the best quality improvement for the added delay.
In the subsequent comparison, Llama 3.2 3B with the few-shot prompt reaches 91.7% structural validity, 92.9% factual consistency, and P50 latency below 1.5 seconds. Nabors considers this close enough to investigate the remaining failures individually. No statistical margin calculation is supplied, so the useful next step is to inspect which outputs received penalties and why.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Inspect the judge and enforce deterministic constraints
A factual-consistency score can hide disagreements about interpretation. Here, Claude Opus judges Llama’s output against Claude Sonnet’s reference summary. Nabors finds penalties for distinctions such as describing Jenna as “angsty” rather than “cross,” and interprets some of them as excessive strictness or favoritism toward the related model. The inspection establishes that the judge’s decisions deserve review; it does not establish that every penalty is wrong or that model-family bias caused it.
Reference validity and output length are more directly controllable in the application harness. Nabors checks references against the size of the thread and truncates overlong summaries. A count check can catch obvious problems, but each reference should also point to an existing message. The following JavaScript illustrates that boundary using a three-message thread about cats and a proposed summary containing an invalid reference. It validates the object shape, removes nonexistent message references, and applies a word limit:
javascript
const thread = [
{ id: 1, content: "My cat keeps waking me up." },
{ id: 2, content: "Mine wants breakfast early." },
{ id: 3, content: "An automatic feeder helped us." }
];
const raw = JSON.stringify({
summary: "People discuss cats waking them early and suggest an automatic feeder.",
refs: [1, 3, 9]
});
function normalizeSummary(raw, thread, maxWords) {
if (!Number.isInteger(maxWords) || maxWords < 1) {
throw new Error("maxWords must be a positive integer");
}
const result = JSON.parse(raw);
if (
result === null ||
typeof result !== "object" ||
typeof result.summary !== "string" ||
!Array.isArray(result.refs) ||
!result.refs.every(Number.isInteger)
) {
throw new Error("Invalid summary shape");
}
const validIds = new Set(thread.map(message => message.id));
const words = result.summary.trim().split(/\s+/).filter(Boolean);
return {
summary: words.slice(0, maxWords).join(" "),
refs: [...new Set(result.refs.filter(id => validIds.has(id)))]
};
}
const normalized = normalizeSummary(raw, thread, 8);
// {
// summary: "People discuss cats waking them early and suggest",
// refs: [1, 3]
// }
The example also exposes the limit of this repair: the word count passes, but truncation leaves an unfinished sentence. Likewise, an existing reference can still fail to support a claim. Deterministic enforcement should therefore remain separate from the factual and readability evaluations.
After adding post-processing, Nabors reports 100% JSON validity and 100% structural validity on this evaluation. Some factual disagreement remains, which Nabors attributes to the judge’s strictness. The completed workflow has reported P50 latency of about one second and P95 latency below 3.5 seconds. Nabors describes the result as meeting or beating Sonnet for this summary feature and saving about $1 per day in personal inference costs. The improvement belongs to the complete workflow—model, prompt, validation, and post-processing—not just the model weights.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep the gains and move one feature
Once the feature works, preserve those gains with regression evaluations. Run the same checks when changing a prompt or upgrading a model, much as CI/CD runs application tests. The failure modes are concrete: short summaries expand into paragraphs, references break, or unsupported claims appear. This returns to the opening problem of a seemingly harmless change disrupting an agent experience—an incident Nabors attributes to a founder friend.
The next opportunity is already in the application’s call history: which Claude calls could become Llama calls, and what would that save? Look for models already managed by the user’s platform. Chrome’s Prompt API exposes Gemini Nano, allowing Chrome to manage model distribution instead of requiring the application to ship its own weights. Current documentation still requires an initial model download, availability checks, and eligible hardware; these foundation-model APIs exclude mobile Chrome. Once downloaded and available, inference can run offline.
Local inference keeps the model’s processing on the device, reducing the need to transmit user information for that step. It does not determine how the rest of the application handles personal data. The practical migration can remain narrow: treat the existing frontier implementation as a prototype, choose one feature, establish its acceptance criteria, and test smaller candidates. Use prompt engineering to improve the selected model where needed, then retain the evaluations as the feature evolves. Nabors’s closing challenge is to make that one-feature conversion and measure the result.
For follow-up, Nabors points readers to the Nearest Neighbors site, Phoenix for trying evaluations, and the Mima beta for the application behind the case study.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Nabors’s guide to defining success criteria and selecting the smallest model that meets them.
Tools for tracing model calls, assembling evaluation datasets, and comparing experiments locally or on hosted infrastructure.
A position paper arguing for smaller models on specialized agent tasks and mixed-model systems where broader capabilities are needed.
- Mitigating Response Delays in Free-Form Conversations with LLM-powered Intelligent Virtual AgentsPaper1:38
A VR user study examining how response delays and conversational fillers affect perceived responsiveness and interaction quality.
Use Gemini Nano through Chrome, check device eligibility, and handle model downloads and session creation.
An extensible AI agent harness for working with language models and tools.
Updates since the talk
- Gemma 4 model cardDocumentation
Official model specifications, including the distinction between effective parameters and total parameters with embeddings for E2B.
Read the complete timestamped transcript
- 0:01
Hi there. I'm Rachel-Lee Nabors, and today I'm here to talk with you about how to use local models to stop paying for frontier models. Let's dig into it. So I've worked on standards that power today's web with Mozilla on Firefox DevTools and the W3C on web standards, and of course, on Microsoft's Edge browser.
- 0:21
I've even been on the React team. Now, I've spent the past three years consulting with AI startups and some of our favorite LLM and browser companies on all things web, AI, and UI.
- 0:33
And recently, I've joined Arize. Have you ever had your CTO ruin your agentic workflow with a slight change of prompt or an LLM migration? Have you ever been that CTO?
- 0:44
Well, you probably need Arize's observability platform for models and the agents who love them. Anyway, I'll be actually using one of Arize's open source projects today, Phoenix. We'll talk more about that later.
- 0:57
But today, specifically, I'm here to talk with you about how AI is costing you. Every time you reach for foundation models like GPT-5 or Claude, it's costing you, your users, and the environment.
- 1:10
Let's have a look at the costs of one-size-fits-all inference.
- 1:17
So first, there's security. Security costs trust. When you use a large LLM that's in the cloud, you're sending data to remote servers, and it always carries the risk of exposure, interception, and retention by third parties.
- 1:30
We have cases where the use of remote AI chatbots has led to sensitive business data being stored, breached, or leaked to the public.
- 1:38
Latency costs the user experience. Now, there's been research on mitigating response delays in LLM chats in VR and found that four seconds is the limit of believability for users, and many calls that you will make to large m- models are gonna take longer than four seconds, as we will see when we check Phoenix.
- 1:57
And it costs your business. Third-party inference costs are uncontrollable compared to API, uh, API costs. Agents see compounding levels of inference, uh, and this means even if the tokens are cheaper, you may be using more of them, or if you're using four of them, they may be more expensive.
- 2:14
And of course, if you aren't connected, remote models simply aren't going to work, which means that unless your software is connected to the web, nobody can use it. Big inference going offline costs productivity if there's an outage, if you're in a place where you cannot reach Wi-Fi, or you're in a very secure environment.
- 2:35
Now, token costs have been falling as of late, but total inference spend has been rising because agentic and reasoning workloads consume tokens way faster than prices are dropping.
- 2:48
But we can completely eliminate most of these costs, and it starts by asking ourselves exactly how much is this costing? Do we really need an LLM to do this job?
- 3:03
All right. You can use task-specific models, which are small in size and power consumption compared to an AI foundation. Uh, this is my little cheat sheet if you're looking for an expert model.
- 3:14
Is a camera pointing at something? Do vision, uh, you know, is, is there a, a vision component? You can use a model like MobileNet, YOLO, MediaPipe. Is it a microphone recording something?
- 3:25
There are audio models like Whisper and wav2vectu.
- 3:29
Chat translation or analysis, this is where you might use something like a small language model like Gemma or Qwen.
- 3:36
These, uh, small models are called SLMs, uh, [laughs] or smaller language models. The, the definition of small is up to debate here. Uh, these are great for times when we do need the language power of a generative pre-trained trans-transformer, a GPT, but we probably don't need the sum total of human knowledge in a black box at our disposal,
- 3:59
or we don't need multimodal capabilities. We're not gonna be analyzing images and audio at the same time. Now, SLMs, smaller language models, contain millions to billions of parameters. An LLM contains billions to, well, trillions.
- 4:16
So you see the big dot there? That's one of the smaller LLMs that's out there. But the little dot actually represents one of the larger SLMs. So you can see that there is a huge difference in parameter size, and this means that there is a vast difference in the size of machine you're gonna need to run one
- 4:32
of these models. The good news is you don't need most of what's in the, the big green dot there. You don't need history. You don't need philosophy. You don't need all those Reddit chats.
- 4:43
You don't need a lot of what the models have learned and been trained on. Most of us are using our models for things like summarizing a chat thread or detecting if this person's being a jerk right now, and it takes a surprisingly smaller amount of parameters to determine those things. [laughs]
- 5:01
Now, the nice thing about smaller language models is that they consume the same or less energy as large language models to produce correct, uh, awareness. We have, uh, uh, correct responses.
- 5:11
We have, uh, good, good research that shows this. Small language models, um, they come in all sizes and shapes. Most small language models, uh, for mobile and web are deployed with quantization.
- 5:25
That is to say eight bit, four bit, and that can halve recorder disk and memory requirements. One billion parameters fits on about two gigabytes in FP16. Uh, this, this grid, by the way, gives upper bound estimates, so these are just to give you an idea of what will and will not fit on different devices.
- 5:45
These are so lightweight that they can be put on devices. Um, speaking of, my Pixel Pro ships with one, and I, I, of course, bought the, the Pixel 10 Pro as soon as I could because I wanted to know what it would be like to work with an on-device model, uh, an SLM of my very own.
- 6:02
The good news is that small language models are production-ready. NVIDIA called SLMs the future of agentic AI. Uh, once again, great research paper from 2025 that found that SLMs are sufficiently powerful for running agentic task loads, and they consume less energy than language models.
- 6:22
Let me, let me give you, um, a bar chart here to take a peek. So let's say this is the total amount of, you know, energy consumed to perform a task that an LLM would take.
- 6:34
An SLM takes about 25% of that, and a task-specific model takes about half of that over. So as you can see, it's from an energy perspective, much cheaper to run the smaller language models than task-specific models.
- 6:50
Um, so what are the benefits? Once more, more secure, works offline, no fees, more efficient, and lower latency because it's on device, no round trips.
- 7:03
Now, I've been using, uh, local AI for some time. Uh, I, for instance, first started using local models. You see on the left you've got Claude, but on the right you have Goose, which is, uh, an open a- an open agent harness that has a, a really nice interface for chatting with models.
- 7:21
And this one here, it's running Gemma, and it's able to answer. It takes a little longer. Smaller models can be a little longer depending on what device you're running and how big the model is.
- 7:34
But one of my favorite questions to ask a model is, how likely is it that ichthyosaurs, marine reptiles from the dinosaur era, had echolocation? 'Cause if you look at their skeletons, they look a lot like dolphins, and it takes, uh, reasoning about biology to be able to determine whether or not that's possible.
- 7:54
And it's hilarious to me that even, I think this was Gemma 3, was able to come up with a good, a good response, um, that there is no evidence in the fossil record of their having developed a melon or, uh, the specialized bones required for conduction that dolphins have.
- 8:14
Of course, Claude hazarded it by saying, "Well, we just can't know," uh, about one out of every three times, which was interesting. Claude has always been, uh, a little nervous about, about coming forth with an opinion.
- 8:29
So how do you pick the best model for the job? This is the hard part. Uh, I recent... [laughs]
- 8:35
I... So how do you pick the best model for the job? I built a framework with Google that I use on my own projects.
- 8:48
You can find more about it at web.dev, but I'll run through it here. Now, first off, I like to think of this as prototype big, deploy small. Just repeat this to yourself.
- 8:57
Prototype big. Think big. Go big. Deploy small. Um, you wanna convert the parts of your system over to SLMs, uh, and specialized models for production. But you can prototype on a foundation model, no problem.
- 9:12
Uh, the first step in this process is to determine whether or not what you're trying to do is even possible at all. Uh, you use the largest, most capable model, and you, you just see if you can get it to do the thing.
- 9:25
See if you can get the model to recognize people, uh, by their handwriting or the way they, they write. If that is possible, then you know that another model can probably handle it too.
- 9:37
So you could use like a foundation model like Gemini, or you could use a really, really tough, uh, task-specific model for like handwriting recognition, for instance. Now, this is a feature of the product that I like to build on the side.
- 9:51
It's Mima. It is a, a client for all your social networks, and I built a feature for it that summarizes long conversation threads. Because I don't know about you, but sometimes I go to bed, and I wake up, and there's 50 comments on something I posted, and I just wanna know, are people angry at me?
- 10:09
And this is what, uh, uh, this is a feature that saves me, saves me a lot of heart attacks in the morning. So I first prototyped it out with Claude to prove that it's good enough.
- 10:20
So y- you could see the little, [sighs] you could see the little summaries there, and this does a pretty good job of, of saying who's talking about what. So first step for this was to collect a set of inputs and outputs.
- 10:34
That is to say, in, in my case, I wanted to collect a set of threads and then how I would summarize them, or in this case, how Claude summarized them seemed adequate to me.
- 10:46
So I, I exported a golden data set. Now, a golden data set is a curated high-quality collection of preferably human-labeled input-output pairs that you're gonna use as the ground truth to evaluate, validate, and benchmark your model.
- 11:02
This is what the, the data looked like. Um, this is all public knowledge, but, you know, you can see it's got the author handle, the content, created at, whether or not it came from me.
- 11:14
And I created a big JSONL of all of these. So there were 14 threads, and I evaluated each one for summaries and annotations because some of the summaries actually link deeper to ex-, uh, link deeper into the conversation.
- 11:31
So it was the same threads, but had two different outcomes for them. One was a short summary, and one was a summary with references. All right. So that means 28 examples all together.
- 11:43
Um, yeah. So these are the things that I was gonna be measuring. Um, you need to... Before you start doing anything, you need to know what it is that you're measuring.
- 11:56
What is success? In this case, was the JSON being output by the summarization process correct? Um, an easy way to test is to try JSON parse, and if it works- Yay.
- 12:11
If it doesn't, boo. Um, the reference structural validity, you know. If it's pointing to different parts of the conversation inside the summary, do those parts actually exist?
- 12:23
Factual consistency, that is to say that it's actually able to summarize the content of the threads within reasonableness. So if we're talking about cats, it doesn't give a summary saying we're talking about puppies.
- 12:37
Uh, this is, this is something that we need an LLM to judge, or humans, but the LLMs will be cheaper, I assure you. Um, length compliance, making sure it stays within a certain word count, and of course, checking the latency.
- 12:50
P50, this is, you know, the median across the eval set, and the P95 is the worst case scenario. All right.
- 13:01
So you collect these, uh, and now you're gonna test and do different models and compare how those models rank against the big models. You test from small to large.
- 13:15
I had to choose a selection of models, but first I needed to find a framework, a tool to do the measuring, and I chose Phoenix, which happens to be created by Arize.
- 13:25
It is open source, it's free, and, uh, the engineers who work on it are pretty darn awesome if I do say so myself. We're gonna perform first of all a...
- 13:37
what is known as a capability eval. A capability eval asks, you know, what can this agent do well?
- 13:44
Where we compare the, the com- performance of the large model against, uh, the performance of a selection of smaller models. We say, "This is what Claude Opus produced, and this is what Gemma 4 produced.
- 13:56
How do these two stack up against each other?"
- 13:59
Now, I used Claude, uh, Sonnet for the baseline, and you can see actually there in the bottom, number one, you can see the, the baseline. It's looking pretty good.
- 14:10
It's got an average latency of 2.9 seconds. It costs about 0.22, uh, 22 cents to run this, uh, to run 14 of these tasks. So I actually did some math, and it turns out I'm using about a dollar worth of inference every day using Mima.
- 14:29
I don't have the money to pay for that many teenage [REDACTED:gender], uh, using Mima every day. So we're gonna have to run this on-device. Good news is the total cost column for all these small local models is absolutely zilch, because that inference has been pushed to the consumer.
- 14:46
It runs on their device. They're the one who has to charge the phone so that it can draw energy from the battery to run the model. Of course, you want it to not suck, [chuckles] suck the battery dry, but that is another conversation for something that we could be testing and evaluating in the future.
- 15:02
Now, I chose... Let's look at our, let's look at our contestants here. We've got, uh, Qwen 2.5 Instruct weighing in at 1.5 billion parameters, only a handy one gigabyte on disk.
- 15:17
And its sister, Qwen 3, uh, 1.7 billion, uh, just a little bit bigger. Llama 3.2 weighing in at 3 billion parameters and a tidy two gigabytes on disk. And then there was Gemma 4 E2B, which is 5 billion parameters, a hefty 3.1 gigabytes, but this was the one that so many engineers I spoke with when I was
- 15:39
picking a model were like, "Oh, Gemma 4 is the best. You gotta use Gemma 4." And I think that's important here, because if I had just gone with what my buddies told me, I may have given the user...
- 15:50
Well, pardon, not may have, I would have given the user an extremely different experience, not a good experience.
- 16:00
You're gonna wanna select the smallest model that gives acceptable responses for your use case. Or as I like to call it, the SAGE model, the small and good enough model.
- 16:10
I'm trying to make this a thing. Bear with me. I hope we can make SAGE happen. Now, at first, I thought I wanted the, the, the Qwen 2.5 model because it was the fastest.
- 16:22
You can see that Qwen is all the way here. Let's see. Where'd it go? Yeah. 2.5 is the gray circle in the lower left-hand corner. It came in around one second total in latency, uh, on the P- P- P50.
- 16:35
That- that's amazing. Like, that's really fast, uh, for, for an AI summary. The problem was that its accuracy was pretty, pretty low compared to everybody else. The orange square is Gemma 4 E2B and the blue diamond is Claude Sonnet, which is our ceiling.
- 16:53
It's the most accurate, but it's also a little pudgy, a little slow. It's, uh, weighing in around three seconds in latency. So when we take accuracy into account, the winner was actually Llama 3.2, which is the big green circle that's weighing in right here around, uh, the 90% for accuracy.
- 17:15
And you can see that both, uh, both Claude and, uh, a- and Llama 3.2 are much faster than Gemma 4. Gemma 4 was coming in around eight seconds.
- 17:31
Now, maybe that was because Gemma 4 needed a different kind of prompt, but this was pretty consistent. I was testing each one of these three times and then averaging the results.
- 17:43
This is what the results look like. I recommend when you're running evals with something like, like, like Phoenix, you open up the experiment and you actually take a look.
- 17:52
You can see what the raw responses were, uh, what the expected responses were. You can actually compare them against each other. Um, I found that in many cases, Llama's response was so close to Claude's response as to be pretty much indistinguishable from one another.
- 18:11
So yeah, uh, Llama 3.2 was the ultimate winner, and I decided to move forward with them. And this makes sense because Llama 3.2 is created by Meta, and Meta has a strong interest in creating models that do a good job with human inputs and summarizing human things on a social network.
- 18:28
Makes sense, right? Mima is a social thing, benefits from a, a social model. So recap. Here's how you right-size your model in four steps. Number one, you prove it's possible.
- 18:41
You test whether what you're trying to accomplish is possible at all by using the largest possible model. This could be a foundation model like Gemini or a task-specific model.
- 18:50
And you set success criteria. You collect a set of inputs and outputs, um, that, that you wanna see coincide with one another. This will be the bar. And then you test from small to large.
- 19:03
You compare the outputs of small models against your test criteria, and you work your way up from the smallest model until you get within an acceptable range of that small and good enough model, that sage model.
- 19:15
That's when you select your sage model, the smallest model that gives acceptable responses for your inputs.
- 19:24
But you're probably wondering, what are we gonna do about that gap? I mean, 90% accuracy, that sounds like it could be a... You know, that could be a deal breaker.
- 19:33
Well, we can squeeze better performance out of smaller models with prompt engineering. This is important in cases where you can't control which model you're using. Some people might, for instance, create a distilled model that's been trained to do this one task really well.
- 19:48
But if you're working, for instance, with a mobile app, you might not wanna be using a distilled model because every time you add capabilities, you'll probably have to train the model slightly, and then you'll be shipping a new one or two gigabyte model every time to your user.
- 20:03
So this is a great example of like, "Ooh, I don't think I'll be able to control that model. Once it's on the user's device, I'm not gonna ask them to download new ones.
- 20:12
That would eat up their data plan." Um, you might join a team, and they're already committed to Gemma 3, and they're just not gonna be implementing Gemma 4 until they've rolled out an, an, an update that, uh, you know, addresses all of the evals.
- 20:28
So let's take a look at closing the gap between Sonnet and Llama, now the, the green and blue dots in that upper left-hand quadrant.
- 20:38
So the measures I, I honed in on, the ones that they really seemed to have different results with, were JSON referenced... Uh, JSON and referenced structural val- validity, factual consistency, um, P50 latency, and P95 latency.
- 20:53
And I decided that the P50 latency, the cutoff is, you know, one, one and a half seconds. For P95, it would be three and a half because remember, four seconds is that worst case scenario for people feeling disconnected from an AI, an AI-powered experience, as per the research. [lip smack]
- 21:13
So where do we start? Um, I recommend that you optimize one step at a time. You wanna isolate one variable per prompt variant to test whether what you're trying to accomplish is, uh, moving the needle when you're using the different prompts. [sighs]
- 21:34
So I created five prompts. Well, I created four prompts, and I had the original prompt as the baseline. The original prompt was pretty good. V2 used number input. Uh, the same prompt reformatted the thre- uh, the thread as numbered messages instead of using JSON, the hypothesis being that smaller models could track natural language indexing better than, uh,
- 21:54
array offsets in a heap of JSON. Uh, the s- the second one was a few shot. It added a couple of examples and outcomes to the prompt, the hypothesis here being that small models learn format from examples faster than from rules.
- 22:11
Then there was the strict rules version. This one was a house of no prompt. It had explicit negative constraints. No preamble. Don't count words before responding. I mean, do count words before responding.
- 22:23
And the hypothesis was that small models respond to literal commands and that they like to be bossed around a bit. And then lastly, there was chain of thought, which forced the model to identify key moments before writing, the hypothesis being that thinking out loud would improve grounding.
- 22:39
So ran the tests again with these new prompts and just, uh, just, just the Llama 3.2, and we discovered a couple of things. We, I mean myself. Uh, we just ran this locally on my machine and was able to compare the results.
- 22:56
You can even put the results into something like Claude and have a conversation about the trades if you like. So...
- 23:07
Stop that. All right, we're back. So in this case, uh, the baseline wasn't really good at, at the, the determining how short it should be, so it would fit within, um, a certain section of content.
- 23:23
The ref accuracy was 91.2%. It was factually correct 87.1% of the time, and the latency was one second. Reformatted input didn't really make a difference. Um, explicit rules actually didn't...
- 23:40
made things worse. Um, the model responded very negatively to being told what it couldn't do. Could and couldn't do. It, um, it was a, a naughty child, didn't like to take instructions.
- 23:51
Chain of thought, uh, didn't have that big... Uh, it, it did a little bit better on the length, m- and unfortunately, that came at increasing the latency by, uh, 600 milliseconds.
- 24:05
The best performing one was the few shot one that provided a couple of threads and a couple of examples. It was much better at, uh, getting the length right.
- 24:14
It was more accurate, um, and, uh, in its references, it agreed with the Claude model a little bit more about what was said, and it only increased the accur...
- 24:25
uh, the latency by 200 milliseconds. So that sounds like a pretty good deal, right?
- 24:31
So it was the few-shot trump, uh, prompt that won. It made the biggest improvement. Let's have a look at how it stacks up to the original. Bump, bump, bum.
- 24:43
So you can see the bar here, Claude Sonnet versus Llama 3.2 3B. Um, I actually did a couple of, of things at this point because I wanted to close that gap completely.
- 24:57
So Llama 3.2 B with a few-shot prompt was actually able to get within a reasonable era, area of, um, [lips smack] a reasonable er- margin of error.
- 25:13
The P50 latency went to less than one and a half minutes, so it was totally green. Uh, the structural v- validity, 91.7%. Factual consistency was also at 92.9%. Uh, the P95 latency was well under 750 milliseconds, uh, less than Claude.
- 25:36
In general, its latency was really good. But let's see about those, uh, those couple of, those 10 percents here between structural validity and factual consistency. This is why it's important to actually open up your evals and take a look at what's inside.
- 25:54
Uh, when it came to factual consistency, it turned out that Claude was just being a very strict judge. I was using Claude to judge the responses and comparing... You know, Claude Opus was comparing Claude Sonnet's response to, uh, Llama 3.2's response.
- 26:08
And of course, Claude was favoring its little sister and being like, "Yeah, um, I think, I think that, uh, you know, I don't think your interpretation of what Jenna said is accurate because you said she was being angsty, and, uh, she was actually being cross."
- 26:27
It was that sort of thing, and this is why it's important to crack them open.
- 26:32
Now, as for ref consistency and length, those actually could be handled inside the harness in post-processing. So making sure that it's, you know, got the right number of references in the thread, that's something very simple to look at.
- 26:45
You can just see how long the thread is, and if there are more refs than there are members of the thread, that's incorrect. Uh, when it comes to, you know, how long the actual, uh, summary is, if it's too long, you can just truncate it.
- 27:01
And so when we added the post-processing, we're actually able to close that gap pretty solidly. Now we've got, uh, 100% JSON validity. Structural validity was 100%. Factual consistency, there's only a little bit of a disagreement there, and it turns out that it was the judge being too darn strict. [sighs]
- 27:21
We got the P5, uh, P50 latency down to around a s- uh, one second, and P95, well, it was under three hundred and f- uh, three, three,
- 27:33
three and a half seconds. So that's pretty good. It actually ended up meeting and beating Claude Sonnet after doing this little bit of extra effort, and I'm saving about a dollar a day in inference costs.
- 27:47
So the important thing is, after you've done something like this, you don't wanna lose one ground. You'll want to make sure that, uh,
- 27:56
you can update the prompt in the future or upgrade the modal.
- 28:02
You don't wanna lose one ground. You wanna ensure that you can upgrade the pr- the model or change the prompt in the future, and it won't cause summaries to expand to be a paragraph long or v- to start hallucinating things that are untrue.
- 28:18
You'll wanna keep your evals running for that, and that would be a regression eval. And you run these sort of like you run, um, CICD tests. You know, uh, it's how you keep your CTO from blowing away your agentic experience by accident one morning.
- 28:34
True story. Happened to a founder friend of mine.
- 28:38
So where do you get started? Uh, how many Claude calls could be Llama calls? I, I challenge you to go home today and take a look at what you're sending to LLMs and ask yourself, "Is this something that a smaller model could handle, and how much money would I save if I did that?"
- 28:55
When working on AI projects, keep an eye out for SLMs and specialized models that might be on-device already. For instance, Chrome and the Prompt API, they access Gemini Nano, which ships with Chrome, um, natively, and that can be really useful 'cause that means you don't need to be shipping a model to anyone using the browser.
- 29:16
You can take advantage of what's right there.
- 29:20
Keep in mind that these models are more efficient, and they meet your users where they are. Their information stays on device. You don't have to worry about PII, and, you know, the energy consumption is, well, a, a blessing for the environment, so to speak.
- 29:38
Remember to prototype big and deploy small. You may want to prototype your system with a foundation model and then convert it, parts of it to small language models and specialized models for production.
- 29:50
Keep in mind, you wanna prove it, define it, test it, and then select your sage model that passes the tests.
- 29:59
You can use prompt engineering to help get better results from SLMs and close that gap with remote models.
- 30:08
So I challenge you to consider your current implementation a prototype and convert one feature to use a smaller local model. Go home, try it out for yourself, and you might be surprised.
- 30:22
Anyway, I look forward to seeing you out there on the web. If you enjoyed talking about small local models or the agentic web, you can follow me. I'm at nearestneighbors.com.
- 30:30
If you wanna try running some evals with your current setup, you can start testing with Phoenix at phoenix.arise.com. And lastly, you can sign up for, uh, the beta for Mima, uh, at mima.social.
- 30:43
I've been Rachel Nabors, and it's been awesome chatting with you today. Go forth and build your own inference stack.