AI Engineer World's Fair 2026
Same Model, Different Speed: Why Your Inference Provider Matters — FriendliAI
Read the talk
Same Model, Different Speed: Why Your Inference Provider Matters
Yunmo Koo explains how prefix reuse, cache placement and adaptive speculative decoding change the experience of running an open-weight model—and why agent completion depends on reliability as well as token speed.
From a talk by Yunmo Koo
At a glance
Ideas worth remembering
The same model and coding-agent workflow can have different completion times on different providers; the inference stack contributes to the user’s experience.
Prefix caching saves repeated prefill work, but distributed reuse also needs routing to the replica holding the prefix and memory management across GPU, CPU and colder storage.
Speculation trades drafting and validation overhead against accepted tokens. FriendliAI chooses between model-based and model-free methods at runtime and describes adapting its draft model to production traffic.
Evaluate providers on your own prompts and traffic, including errors and completed agent tasks: one failed model call can invalidate a longer workflow.
Two working games make cost the next question
A single prompt asks two models to build a tower defense game. Both produce working implementations: a map, turrets and enemies that the turrets shoot when they enter range. For Yunmo Koo, founding engineer at FriendliAI, this small experiment makes a practical question clearer than a leaderboard alone: once an open-weight model produces a useful result, what does it cost to keep using it?
The comparison pairs GLM 5.2 with Claude Opus 4.8. Koo describes their game structure and quality as similar. What does that similarity amount to? The maps and range-based turret behavior are the concrete features to compare in the demonstration; they establish a working game in each case, rather than a comprehensive evaluation of coding ability.
For this task, Koo reports that the GLM 5.2 implementation was more than 5.6 times cheaper than the Opus 4.8 implementation. Repeated agent iterations make that difference consequential: each additional attempt adds another inference bill, so a lower cost per attempt changes how much work a team can afford. The recording gives a task-specific comparison without a detailed billing breakdown, so the ratio should stay attached to this experiment.
The opening context is a shift in which models Koo considers usable. His audience mostly uses closed-weight frontier models, while he is moving more of his own workload toward open weights. Coding, reasoning and tool-use benchmark results motivate that move. His claim is scoped to many workloads: when model capability clears the required threshold, production speed, reliability and cost become the next decisions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Hold the model and agent fixed; change the provider
The next demonstration changes the provider while keeping GLM 5.2 and the coding-agent workflow the same. The task is a Candy Crush-style mobile game called Model Crush. FriendliAI, shown on the left, finishes within two minutes; the other provider takes more than three. Where does the difference become visible? The useful comparison is the progress toward a completed game on each side, because completion time is what the agent’s user experiences. This is one demonstrated run, rather than an established latency distribution.
Identical weights do not imply identical service. Latency, throughput and reliability depend on the system that runs those weights. Koo points to a provider comparison snapshot to reinforce that these differences are measurable, then opens up the stack behind them: cache systems, routing policy, scheduling, quantization and GPU kernels all contribute to the result.
FriendliAI’s engineering premise is that these parts must work together. Batching and scheduling determine how requests share execution; cache management determines how much previous work can be reused; speculative decoding changes how output tokens are produced. The game comparison does not isolate the contribution of any one optimization. It motivates examining the whole inference engine rather than treating the API endpoint as interchangeable plumbing.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reuse the long input—and route requests to where it lives
Agent requests often have an uneven shape: very long inputs and relatively short outputs. Koo describes inputs exceeding 10,000 tokens, with roughly 50% to 80–90% of tokens repeated across requests in many workloads he encounters. Under those conditions, processing the input can dominate compute cost. This input-processing stage is called prefill; it builds the key-value, or KV, cache used during generation.
A shared prefix creates an opportunity to avoid doing that work again. When a new request begins with a prefix already processed for an earlier request, the engine can reuse the corresponding cached tensors and skip that prefix computation. The immediate benefit is lower time to first token, or TTFT: less input work stands between receiving the request and starting its response.
Two separate mechanisms make that reuse practical at scale:
- Cache-aware routing. KV caches live on individual serving replicas. The router must send a request to a replica holding its relevant prefix; sending it elsewhere loses the reuse benefit.
- A memory hierarchy. KV caches consume substantial memory and, for large models with long contexts, can exceed the model weights in size. Hot cache stays in fast, limited GPU VRAM, warm cache in CPU RAM, and colder cache can move to NVMe or disk. Less-used blocks are offloaded and prefetched when needed.
Why are both mechanisms necessary? The diagram separates two placement decisions: which replica receives a request, and where cached blocks reside within the memory hierarchy. Routing makes an existing prefix reachable by the request; tiered storage preserves more cached work than scarce GPU memory alone can hold. Together, they reduce repeated prefill work for the long, repetitive inputs described here.
Request placement and cache placement solve different problems
Read the diagram as text
- Incoming request. routing — Shares a prefix with a previous request.
- Cache-aware router. routing — Selects the replica holding the relevant prefix.
- Replica with matching cache. routing — Reuses prefix tensors instead of recomputing that input.
- GPU VRAM. cache-storage — Hot cache; fastest storage, with limited capacity.
- CPU RAM. cache-storage — Warm cache.
- NVMe or disk. cache-storage — Colder cache.
- Incoming request → Cache-aware router: Route the incoming request.
- Cache-aware router → Replica with matching cache: Send to the replica with the prefix.
- GPU VRAM → CPU RAM: Offload less-used blocks.
- GPU VRAM → NVMe or disk: Offload colder blocks.
- CPU RAM → GPU VRAM: Prefetch when needed.
- NVMe or disk → GPU VRAM: Prefetch when needed.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Draft several tokens, then make verification worthwhile
After reducing the wait before generation begins, the next problem is generation itself. Standard autoregressive decoding produces one token per forward pass. That makes progress sequential, and Koo notes that the GPU is often underused during decoding. Speculative decoding proposes several candidate tokens ahead, then asks the target model to validate them in a single pass.
Accepted candidates let one target-model pass advance the output by multiple tokens, reducing time per output token. Koo calls accepted tokens “free”; the useful interpretation is that they avoid separate target-model passes, since drafting and validation still have costs. The target model remains responsible for validation, which is the basis of the talk’s claim that this acceleration preserves output quality.
The candidates can come from two sources, each with a different cost:
- Model-based speculation. A small draft model predicts future tokens. It can achieve a higher acceptance rate, but running that model requires extra compute.
- Model-free speculation. Patterns in the prompt or earlier output supply candidates. This has lower overhead and works best when the next output resembles existing context.
The fastest choice for one request may add undesirable work to the batch serving it. Validation has overhead, so speculation must earn back its cost through accepted tokens. FriendliAI uses a hybrid runtime that selects between model-based and model-free methods according to workload and request characteristics. The recording describes that decision at a high level without specifying selection thresholds.
There is also a second timescale of adaptation. Beyond choosing a method for the current request, Koo describes “self-evolving speculation”: the draft model adapts using real serving traffic so its predictions better match the workload actually running in production. This makes the speculator workload-specific over time; the talk does not detail its training or update procedure.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A fast call still has to finish
“Reliability is part of speed” becomes concrete when an agent needs several model calls to finish one task. A timeout, a server-side 5xx error or a 429 rate-limit response can interrupt the run. One failed call in the middle can invalidate the workflow, so low average response latency alone does not describe how quickly users receive completed work.
That changes the optimization target from token throughput alone to the full API experience, including error rate and uptime. Koo cites a provider snapshot for GLM 5.1 Pro reporting more than 99.9% uptime on FriendliAI. The recording does not establish the snapshot’s measurement window or workload, so that figure supports the presented model-specific service comparison rather than a general availability guarantee.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Test the prompts and traffic you actually serve
Provider speed claims are conditional. A benchmark uses particular inputs and operating conditions; an application brings its own prompt lengths, traffic patterns and latency expectations. Koo’s recommendation is to test those actual conditions and compare speed, reliability, cost and quality together. Long shared prefixes and context-like output patterns matter because they directly affect the cache reuse and speculation mechanisms described earlier.
The closing customer example is Kilo Code, which Koo describes as a coding-agent platform processing several trillion tokens each week. After testing multiple inference providers, its GLM speed tests reportedly found FriendliAI up to seven times faster, with a substantially lower error rate. This is a customer result relayed in the talk; the recording does not supply enough test detail to transfer that multiplier to another application.
The practical next step is to bring a real workload to the provider comparison. Hold the model and agent workflow steady, as in Model Crush, and judge the completed task alongside inference cost and failures. The tower defense example asks whether the model can produce useful work economically. The provider example asks how quickly and reliably that same class of work reaches the user.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Read the complete timestamped transcript
- 0:13
All right, let's get started. So, hello, everyone. My name is Yunmo. I'm a founding engineer at FriendliAI, uh, building a inference engine with our team to optimize inference performance. Uh, so today, I'm gonna talk about how we built the fast and reliable inference API for frontier open weight models. So . Oh, thank you . Okay, so before we dive in, let's do a quick show of
- 0:43
hands. So who here uses closed weight frontier models like GPT or Claude? Okay, most of you are already using it. And then who here mainly use open weight models?
- 0:59
Yeah, yeah, fewer than the closed weight models. So personally, I recently using a mix of both these days, but I'm gradually migrating more of my workload to open weight models. The reason is that open weight models, uh, have crossed the threshold these days. For example, in coding, reasoning, and agentic benchmarks, models like GLM five point two are competing in the same tier as the leading proprietary models. So as you can see from
- 1:29
this Code Arena re-leaderboard, uh, GLM five point two is ranked even higher than Claude Opus four point eight, so it is right, uh, below the Claude, uh, Fable five. So similarly, across various coding terminal and tool use benchmarks, uh, GLM five point two and MiniMax-M3 shows strong results. So the question is no longer, uh, only whether open weight models are good enough. For many workloads, now they are. So the
- 1:59
next question is, how fast and reliably we can run them in production. So benchmark numbers alone, uh, don't really give a clear sense of how well things actually works. So instead, I ran a, a simple hands-on experiment to see how these models actually perform. So I asked both GLM five point two and Opus four point eight, uh, four point eight to build a tower defense game like this. So as you can see, both models produce working
- 2:29
implementation with similar structure and quality. So both one has a map, and the turrets are shooting the, the enemies within the range. So yeah, both looks pretty great considering I made it with a si-uh, single prompt. But when we look at the cost, the difference becomes very clear. So GLM five point two was more than five point six times cheaper for this task. And in agent sly-style workflows where, uh, you need to iterate multiple times,
- 2:59
the, the, the gap compounds quickly. So this is not just about model quality anymore. It is about what you can afford to run at scale. So after the model quality and cost, the next que-next question is provider quality. And provider quality is really about performance. So even if the model weights are the same, latency, throughput, and reliability can vary significantly depending on where you run it. Uh, so this one has a OpenRouter snapshot,
- 3:29
which shows that, uh, these differences are measurable. So if you're building an AI product, you should not only ask which model to use, but you should also ask which providers gives you the best performance, reliability, and pricing. So I will show you another example. So here I use the same model as GLM five point two and the same coding agent workflow. So the task is to build a simple Candy Crush style mobile game, which we call Model Crush. So
- 3:59
when you see this, the left side is FriendliAI. It generates the, the, the game within two minutes. Another provider needs, uh, more than three minutes. So the model is the same, the agent ev-ev is the same, but the provider was different. So model developer tools and coding agent, this i- this expe- this is the experience that users actually feels. So from this simple example, we saw that even with the same model
- 4:29
and the same task, the experience can feel very different depending on the provider. The difference comes from the entire inference stack. So the model architecture, cache system, routing policy, scheduler, quantization strategy, GPU kernels, all those things come together to shape what the developer actually experiences. So this is the, uh, core of FriendliAI's value. So we are not just exposing a model API via, uh... So FriendliAI is built as a full stack
- 4:59
inference, uh, engine for GenAI models. This matters, uh, because fast and reliable inference is not just one optimization. It is batching, scheduling, quantization, spec decoding, mo- and kernels, all, all, everything should work together. So let me briefly explain, uh, how we built that and how we make, uh, the inference performance, uh, to the cutting-edge level. So in the
- 5:29
modern agent applications, inputs are often very long. It easily exceeds, uh, ten k tokens, while outputs are relatively short. And more importantly, a large portion of the inputs are repeated across requests. In many real workloads, like fifty percent to eighty to ninety percents of the tokens are identical across different requests. So from a system perspective, this means, uh, most of the compute cost is in the prefill stage, where
- 5:59
the model processes the entire inputs and build a KV cache. So if we recompute this k- uh, shared prefix every time, we are wasting both latency and GPU cycles. So when a new request share, uh, a prefix with the previous one, we can reuse those tensor and skip the re- corres- corresponding prefix computation and directly reduce the TTFT, and, uh, we can avoid recomputing large chunk of the input. And but however, making this work at scale requires two key che-
- 6:29
uh, mechanisms. So first one is cache-aware routing. So in the distributed system, KV cache is not centralized. It lives on individual replicas. So the router must send incoming request to the replica that already holds the relevant prefix in its cache. Otherwise, we ru- we lose the benefit of reuse. The second thing is cache hierarchy. So KV cache is extremely memory-intensive. For large models and long context, KV cache size can exceed
- 6:59
model weight. So, and as you know, GPU VRAM is the fastest, but it's also very limited resources. So, uh, we treat cache as a hierarchy, like hot cache stays in GPU memory, warm cache is in, uh, CPU RAM, and co- colder cache can be offloaded to NVMe or, uh, the disk spaces. The system dynamically offloads less used blocks and prefetches them when, uh, when they need it. So by combining cache elra- cache-aware routing and
- 7:29
hierarchical memory management, we significantly reduce the prefix cost, lower TTFT, and making, uh, long context agent workload much more efficient. So the next thing is interactive speed. So standard autoregressive decoding, it generates one tokens per forward pass that makes decode, uh, inherently a sequencer, and during the decode, the GPU is often underutilized. And speculative decoding changes the shape of this problem. So
- 7:59
instead of generating one token and stopping, it proposes multiple candidates tokens ahead. Then the target model validates those, uh, candidates in a single pass, and if the tokens are accepted, they are free. So this improves time per output token, uh, without changing model's, uh, output quality. And there are different ways to draft candidates. So one way is model-based speculation. It uses a small
- 8:29
draft model to predict the future tokens, and then, and the benefit of it is it can achieve higher accep- acceptance rate, but the cost is the, uh, extra compute. And another method is model-free speculation. It uses pattern from a prompt or prior output, so it has lower overhead, but it works best only when the output resembles the existing context. So the important part is the trade-off. So
- 8:59
speculation can improve per request latency, but validation adds overhead to the batch, so the best strategy is not static. And at FriendliAI, we take a hybrid approach, so the runtime can dynamically select between model-based and model-free speculation, uh, based on the workload and request, uh, characteristic. So we also think about speculation as something we can improve over time. So with self-evolving speculation, uh, draft model
- 9:29
can adapt based on the real serving traffic. So the speculator continuously improves its prediction of the workload, uh, actually running in the, in your, uh, production. So the next thing I want to mention is reliability. So speed alone is not enough. So production AI application, uh, reliability is part of speed. So if an API is fast in average, but frequently hits timeout, like five,
- 9:59
5XX error, 429 rate limit or fail, or that kind of failure, the user experiences, uh, speed bre- breaks. So this becomes even more important for agentic workloads. As you know, agent may need multiple model calls to complete one task, and if just one call fails in the middle, the entire workflow can be invalidated. So that's why we do not optimize only for the latency or throughput. We optimize the full API
- 10:29
experiences, uh, including the error rate, uptime, all those things together. And this OpenAI snapshot for, uh, GM- GLM 5.1 Pro, uh, shows that FriendliAI is not only the fast, but it's also high reliable with over ninety-nine point nine percent uptime. So what I want to say is many inference provider out here, of course, in this venue as well, they will tell you that they are fast, and some point of them,
- 10:59
uh, they are, but on cer- on certain benchmark, under certain condition. But for your application, the only benchmark that really matters is your own workload. So your prompt, uh, your token lengths, your traffic patterns, and your latency expectation. So our recommendation is simple. So please bring your workload to us and test the pu- free experience with us in terms of, like, speed, reliability, cost, and quality. And of
- 11:29
course, um, we build our inference service for exactly these production workloads, and we, and we believe that we can give you the best experience. So we have a booth in this venue, so bring your hardest problem to us, and let's solve it together. So to close, I want to share a testimonial from one of our customer. It's a Kilo Code. So Kilo Code is one of the largest coding agent platform, processing, like, several trillions of tokens every week.
- 11:59
So they have tested many different inference providers over past years, and in their speed test for, uh, GLM models, they were especially impressed by FriendliAI. So compared with many other inference provider, FriendliAI was over, like, f- seven time faster and had a sig- significantly lower error rate. So I believe you can get the same kind of experience with FriendliAI. So if you're building your frontier open weight models, building with your fr- frontier open weight
- 12:29
models, come try them on FriendliAI and bring your real workload, and let us show you what fast and reliable inference can do. So this is the end of my talk, and if you have any question, feel free to ask. Thank you.