AI Engineer World's Fair 2025
Hacking the Inference Pareto Frontier
Read the talk
Hacking the Inference Pareto Frontier
Inference efficiency depends on matching the model, cache, scheduler, and GPU fleet to the application’s quality, latency, and cost requirements.
From a talk by Kyle Kranen
Before you start: Basic familiarity with autoregressive language models and GPU inference is helpful; prefill, decode, and KV caching are explained in the article.
What makes an LLM deployable?
Can your model complete the application’s task well enough, fast enough, and cheaply enough? Choosing a capable model answers only part of that question. The model and the system serving it must meet the application’s constraints together. Kyle Kranen approaches this problem from operating large inference systems: he describes previously leading NVIDIA’s largest inference deployment, with a quarterly cloud bill in the multiple tens of millions of dollars. His subsequent work on NVIDIA Dynamo targets data-center-scale inference, using techniques such as disaggregation to meet better service-level agreements or reduce costs at an existing SLA.
Three requirements determine whether a deployment is useful:
- Quality: Can the model and surrounding application complete the target task with sufficient accuracy?
- Latency: Does the response arrive quickly enough for the user—or for a safety constraint, as in robotics?
- Cost: Is the expense per request low enough to support the application’s margins?
These are joint requirements. Excellent answers that arrive too late can fail the product just as surely as fast answers that are wrong.
A Pareto frontier describes the best achievable tradeoffs. The talk projects the problem onto two dimensions: tokens per second per GPU as an efficiency proxy, and tokens per second per user as a responsiveness proxy. Moving up and right improves both. Although the initial explanation briefly calls the GPU measure requests per second, the later graph explicitly uses tokens per second per GPU.
The deployment does not need every point on that frontier. It needs one operating point: the required quality and latency, delivered at the lowest feasible cost.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose the operating point from the application
Consider three applications with very different tolerances:
| Application | Dominant requirement | Accepted tradeoff |
|---|---|---|
| Personalized cancer cures | Quality of the result | Potentially substantial time and expense |
| IDE tab completion | Immediate responsiveness | A tight latency budget |
| Asynchronous code commits | Quality and cost | More time to finish the work |
In Kranen’s hypothetical cancer-cure example, spending millions of dollars to prove a single successful cure could be justified by its value. Cursor-style tab completion has a different contract: the next line or tokens must appear quickly enough to feel immediate. An asynchronous coding agent working alongside the user, such as the Agent mode example in the talk, has more room to spend time producing a good result. Whether the user is waiting in the loop changes the optimization target.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Combine techniques that move different constraints
The familiar optimization tools affect different parts of the problem:
- Quantization can accelerate inference and reduce memory requirements, enabling larger batches and lower serving costs.
- Retrieval-augmented generation adds retrieval work and context, generally increasing latency and cost in exchange for better answers.
- Reasoning spends additional generated tokens on thinking, increasing work to improve quality.
- Parallel configuration changes how the model runs across devices, altering speed and cost. Kranen also mentions possible quality effects for what he calls non-haloed context parallelism, without developing that variant’s mechanism.
These tools interact. Adding retrieval may produce the quality an application needs while making it too slow; quantizing the model afterward may recover some of that latency. The useful unit of optimization is the combined system. The remaining techniques exploit three properties of that system: scale, structure, and dynamism.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate prefill and decode, then balance them
Autoregressive generation repeatedly attends to the sequence produced so far. A KV cache retains each token’s key and value vectors so subsequent steps can reuse them instead of regenerating those vectors for the entire sequence.
This creates two distinct phases. Prefill processes the prompt and populates the cache. Decode produces output tokens and extends that cache. In an aggregated deployment, both phases run on the same set of GPUs. Disaggregation places them on separate workers and GPU sets so each phase can receive a different allocation of resources.
The phases have different bottlenecks:
| Phase | Resource pressure | Allocation opportunity |
|---|---|---|
| Prefill | Primarily compute-bound | Size workers to saturate compute |
| Decode | Can be memory-bound | Use a different GPU count and batch size |
Using DeepSeek-R1 as an example, Kranen describes compute saturating relatively early during prefill: fewer GPUs can handle smaller prefill batches, while more GPUs support much larger decode batches. Separating the phases permits this heterogeneous allocation.
It also removes a scheduling conflict. When one request needs prefill and another is already decoding on the same machine, the scheduler must arbitrate between admitting new work and advancing existing output. In-flight batching and chunk piggybacking help manage that contention, but retain the cost of scheduling both phases together. Separate workers simplify that decision.
Kranen reports that, for his Llama 70B comparison using the same total of 16 H100 GPUs, disaggregation achieves up to 2× tokens per second per GPU at a fixed latency. The graph plots GPU token throughput vertically and per-user token throughput horizontally. At that operating point, twice the throughput per GPU suggests roughly half the GPU serving cost per token; it does not establish a halving of total application expenditure. The talk does not specify the exact model variant, workload, concurrency, software version, or latency definition needed to reproduce that point.
The gain depends on where the workload sits. Short prompts require little prefill, leaving less contention to remove. At the high-throughput, high-latency extreme and the low-throughput, low-latency extreme, aggregated execution can converge with or outperform disaggregation. The middle of the curve is often more attractive for interactive applications; Kranen places many of these applications around 20–200 tokens per second per user.
Even there, the worker ratio matters. Too many decode workers leave GPUs waiting for prefill to supply work. Too many prefill workers feed decode faster than it can keep up, increasing load and queue depth. Each phase’s parallel configuration changes its capacity, so tuning the ratio also means exploring a broad—and expensive—configuration space.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Route for both cache reuse and available capacity
Disaggregation requires moving KV state between machines. Once state has a location, requests acquire an affinity for particular workers: a worker may already hold a reusable prefix from earlier requests. That cache can reside in GPU memory, host memory, or external storage.
Random routing ignores this affinity. Routing exclusively to the worker with the largest matching prefix has the opposite problem: popular prefixes can concentrate traffic on an already overloaded machine. Kranen corrects the labels and orientation of the routing illustrations as he walks through them; the underlying distinction is between random assignment, cache-only assignment, and a decision that considers both reuse and load.
A cache hit is valuable only if waiting for it does not erase the saved work. The routing decision should balance the prefix work already completed against the worker’s existing load.
A small Python example expresses that tradeoff using estimated queue delay and remaining prefill time. These teaching estimates put both terms in milliseconds; this is a scoring example, not Dynamo’s routing implementation.
python
from dataclasses import dataclass
@dataclass(frozen=True)
class Worker:
name: str
cached_prefix_tokens: int
queue_ms: float
prefill_ms_per_token: float
def estimated_start_ms(worker: Worker, prompt_tokens: int) -> float:
reused = min(prompt_tokens, worker.cached_prefix_tokens)
remaining = prompt_tokens - reused
return worker.queue_ms + remaining * worker.prefill_ms_per_token
workers = [
Worker("a", cached_prefix_tokens=800, queue_ms=100, prefill_ms_per_token=0.1),
Worker("b", cached_prefix_tokens=500, queue_ms=10, prefill_ms_per_token=0.1),
]
prompt_tokens = 1000
selected = min(workers, key=lambda w: estimated_start_ms(w, prompt_tokens))
print(selected.name)
Worker a has the better prefix match, but its score is 120 ms. Worker b has more prefill left to do, yet scores 60 ms because its queue is shorter. The selection favors b; it does not execute or dispatch an inference request.
Scale can increase the opportunity for reuse. Kranen argues that a larger fleet can hold more previously computed KV state, increasing cache hits and reducing prefill work. That benefit depends on requests actually sharing reusable prefixes and the system retaining and finding them. Routing aims to improve responsiveness and cost efficiency while preserving quality: it reuses work for the same model computation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use repeated reasoning as scheduling information
Agents introduce structure: moderately predictable patterns across successive and concurrent requests. Inference-time scaling is one example. Re-querying a model to reconsider an answer or reason further creates additional opportunities to improve the result.
The displayed comparison uses an 8B model in green, a 49B model in yellow, and a 235B model in red. Kranen reports that approximately three or four re-queries bring the 8B model near the 49B model’s quality and the 49B model near the 235B model’s quality. The specific model identities, task, quality metric, and prompting or answer-selection procedure are not established here, so the comparison illustrates a possible operating-point shift rather than a recipe for substituting arbitrary models.
There are two ways to evaluate the extra calls. Holding the model fixed, repeated queries spend more time and money to improve quality. Holding the required quality fixed, repeated calls to a smaller model may still be cheaper—and potentially faster—than one call to a larger model. Kranen’s comparison makes the latter case: additional inference work can reduce the overall expense if it enables a sufficiently smaller model.
The repeated-query structure also gives the serving system information it can use. In a reasoning example from NATURAL PLAN, Kranen compares runtime across concurrency levels. This example has short input sequence lengths and long output sequence lengths, so disaggregation alone offers only a small benefit: there is relatively little prefill work to separate.
The next changes target the workflow itself:
- Move re-query control into the router. A follow-up request no longer has to return to the external client before coming back into the serving system.
- Make the router and LLM scheduler aware of repetition. They can schedule with knowledge that the current request belongs to a sequence of related calls.
The first change removes external round trips. The second produces the additional improvement Kranen identifies as the red-to-green curve transition. His broader conclusion is that, at fixed quality, model selection plus structured scheduling can lower latency and raise throughput. The recording does not provide enough implementation or curve detail to quantify those scheduling gains.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep useful KV state through a tool-call pause
Workflow structure can also reveal when cached state will be needed again. Consider Kranen’s separate tool-call example: an LLM invokes a tool that takes about 30 seconds. During that pause, other requests consume GPU high-bandwidth memory, and the idle request’s KV cache may be evicted. The 30 seconds describe this example, not a universal cache timeout.
If the serving system knows the request will resume, it can preserve the expensive prefill work instead of letting eviction force recomputation.
The proposed lifecycle is:
- Prefill the prompt, then decode the output that initiates the tool call.
- Offload the reusable KV state to host memory while the tool runs.
- Restore the state to GPU memory near the expected completion time.
- Resume the LLM call with the tool result, reusing the prior context and processing the added context.
This requires knowing more than which cache entry was recently used. The workflow supplies a prediction about future use. Repeated reasoning and tool calls both expose that information. The intended benefit is faster, cheaper continuation with unchanged quality; Kranen explicitly corrects his initial suggestion that KV manipulation improves quality.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Specialize the fleet and rebalance it as demand changes
Dynamism begins with accepting that one worker configuration may not suit every request. Input sequence length (ISL) and output sequence length (OSL) determine how much prefill and decode work a request creates. Their joint distribution can guide a mixed fleet:
| Sequence-length regime | Illustrative configuration |
|---|---|
| Short input, long output | Aggregated workers with higher tensor parallelism |
| Middle input-length range | Disaggregated workers |
| Long context | Disaggregation with context parallelism |
These regions are model-dependent examples, not fixed boundaries. Specialization aims to improve speed and cost without changing quality because it changes resource allocation rather than the model’s executed mathematics.
The distribution itself changes after deployment. Suppose the initial configuration fits the input/output histograms of App A and App B. A change in the user mix shifts demand toward longer sequences. In the illustrated shift, both input and output lengths grow, but input length grows more. That creates relatively more demand for prefill capacity, invalidating the previous prefill/decode balance.
Prefill and decode capacity must adapt independently as the workload changes. Real-time autoscaling across the two worker types keeps one phase from starving or overwhelming the other. Dynamic load balancing is therefore essential to sustaining the speed and cost benefits of disaggregation, not merely an additional optimization after configuration is finished.
The implementation handoff is Dynamo, the open-source project introduced at the start. Kranen closes by inviting attendees to a San Francisco meetup the following Thursday, from 5–8 PM, for more detail on implementing these techniques. For a deployment, the ongoing work is equally concrete: keep matching worker specialization and phase capacity to the requests users actually send.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Distributed inference framework with setup instructions, disaggregated serving, KV-aware routing, cache offloading, and autoscaling.
Natural-language planning tasks and evaluation scripts for trips, meetings, and calendars.
Further reading
- Dynamo's launch architectureArticle
Kranen's coauthored introduction to Dynamo's planner, routing, cache management, data transfer, and launch benchmarks.
Original benchmark design and experiments on planning with supplied tool outputs.
Updates since the talk
Current development guide for configuring separate prefill and decode workers on Kubernetes.
Read the complete timestamped transcript
- 0:00
[on hold music] Hey there, everyone.
- 0:15
I'm Kyle Kranen, and today I'll be talking about how to break the inference Pareto pr-frontier in your advantage. Um, really, the thing that enables the success is that a good model and a good system that takes into account the actual constraints for what you need from your deployment is actually key to the success of both your deployment
- 0:34
and the application that is backed by it. Um, so who am I and why am I talking about this? Uh, as I said, my name's Kyle Kranen. Uh, previously, I work at...
- 0:44
Or currently, I work at NVIDIA. Uh, previously at NVIDIA, I was, uh, leading and GM-ing the largest inference deployment at NVIDIA with a multiple tens of millions of dollar quarterly cloud bill.
- 0:57
Uh, and now I'm an architect and lead for a project that we just released in open source called NVIDIA Dynamo that aims to do things like enable data center scale inference to manipulate your deployment and manipulate the Pa-Pareto frontier in order to achieve better SLAs or achieve lower costs for your existing SLAs with techniques like disaggregation or
- 1:17
more techniques that I'll talk about later in the talk. Dynamo is, uh, or the Dynamo meetup is linked right here. You can learn more about Dynamo there if you wanna look it up, and I'll also have that, uh, at the end of the talk as well.
- 1:29
Um, so the three things that we or I like to think about when I'm thinking about whether or not something can actually be deployed and used is really simple.
- 1:39
It's quality, whether or not your application and the system around your model is capable of, you know, completing tasks with some level of accuracy or quality. Latency, whether or not the task can be completed in a fast enough envelope for, you know, either the user to be happy or to meet safety guarantees, like for robotics.
- 1:57
Uh, and cost. Can the LLM complete the task cheaply enough per request in order for you to meet whatever, you know, margin requirements you have for your application? Um, and one of the ways that we generally compare these three things is through a Pareto frontier.
- 2:14
Now, uh, the Pr-frontier I'm showing here is two-dimensional. Uh, it's actually really hard to plot things in 3D on a 2D slide, so I'm gonna just show two dimensions.
- 2:22
Really what this looks like is you have this, like, edge that d- Oh. Is it working? Uh, we have this edge that sort of represents the best or to the top and rightmost points that we achieve for, uh, you know, a specific set of attributes.
- 2:39
So in this case, we have the TPS per GPU, which is effectively a cost metric. How many requests can you handle per GPU per second? And the user TPS, which is a responsiveness metric, right?
- 2:49
So this is the, the latency versus the cost. And for different applications, really, you want to enable your Pareto front-- e-enable... Or you actually really only want one point on the Pareto frontier.
- 3:01
It's, right, what is your operating latency? What is the operating quality you need? And how can you minimize cost for that? Now, this really actually depends on the application you're talking about.
- 3:10
So one of the most important things you're doing when you're thinking about breaking the Cr-Pareto frontier is you're thinking about your application. So for example, if we're talking about personal cancer cures, which is a topic that's talked about a lot in the context of generative AI, uh, in that situation, uh, latency and cost are pretty much no
- 3:28
object, right? You could spend millions of dollars on proving out a single cure, and if it works, the return on investment is so high that it doesn't really matter.
- 3:37
Um, to take a different example, tab completion like those that you see in popular IDEs like Cursor all are very, very dependent upon snappiness. The user expects that when they press Tab, they will see a recommendation for the next line or the next set of, you know, tokens very, very quickly.
- 3:56
And then to take an-another code example, with respect to async code commits, things like, uh, you know, uh, Cursor's, um, what's it called? It's Agent mode and, um, you know, other applications where the, the chatbot or application is working next to the user, there's not as much a consideration for latency, but there is a concern for both
- 4:18
quality and cost. Uh, and this sort of breaks down how... Or this sort of depends upon what the user expects from the application. Does it-- Do, do they expect it to be fast?
- 4:30
Do they expect it to be slow? Are they involved in the loop with this application?
- 4:35
Now, there are a series of, you know, techniques that are pretty commonly known about that all, you know, support the manipulation of this frontier. For example, quantization speeds up your latency, and it also decreases your cost because you can produce higher batch sizes.
- 4:49
Retrieval-augmented generation generally slows down your application, makes it higher latency, increases the cost, but also increases the quality. And reasoning, for example, similar, m-you know, you produce more tokens to think.
- 5:01
And changing the model config allows you to do any of these things. If you change how the model is represented in a parallel manner, uh, you can significantly change the characteristics of, uh, speed, cost, and theoretically quality if you're talking about non-haloed context parallelism.
- 5:16
Um, the thing that I wanna impart upon you before we jump into, like, a lot more of these advanced techniques is that these techniques can be compounded. So for example, if you have an initial application and has some required f-performance, you can actually stack, for example, retrieval-augmented generation in order to increase the quality but make the latency
- 5:34
worse. And you can also stack on top of that quantization of the model in order to speed up your latency. The point I'm, you know, trying to, trying to make here is that you really have this toolbox of a sets of large sets of tools that you can use together, and the tools themselves are not independent and
- 5:52
can be combined in very sometimes non-obvious ways in order to actually break your Pareto frontier or squeeze it in different directions in order to support your application. Um, so there are three things outside of those techniques that I, I, I tend to l- I tend to think drive, uh, you know, the how you can modify the Pareto
- 6:14
frontier, uh, w- going forward. Those three are scale, structure, and dynamism.
- 6:20
Um, so one of the things that is, you know, really relevant in the realm of scale is disaggregation. So for those that aren't aware, uh, KV caching is a technique by which you take the k- k-val-- key and value vectors that are associated with each token, and you cache them, uh, so that when you're doing autoregressive generation,
- 6:40
you don't have to generate the entire set of key and value vectors for the entire sequence up to this point. You can just generate new ones and put them back into the KV cache.
- 6:50
Uh, what this actually means is that we effectively have two phases of generation, one in which you're generating the prefill or filling up your KV cache, and one in which you're actually generating new KV cache as well as new tokens and producing output.
- 7:06
Now, um, disaggregation as a technique basically allows you to have these, you know, two phases which were typically used on the same set of GPUs, uh, onto multiple different s- workers and sets of GPUs.
- 7:19
And this provides a couple of key benefits that we'll go into right now. Um, the three really big benefits here are that you will-- you can really now take two, uh, sort of phases that have dir- very different needs.
- 7:32
Uh, prefill is very compute-bound, and decode, depending on the ap- application and model, can be very memory-bound. And it allows you to do a granular load matching between those two phases.
- 7:42
Wha-- And what this m-means is, uh, you know, compute saturates relatively early, uh, to use DeepSeeker One as an example. Compute satu-saturates relatively early, and you may use, uh, relatively few, uh, GPUs for your prefill instances and have them handle a lower batch size.
- 7:57
But handle a much larger batch size with many more GPUs for your decode instances. And this split and this heterogeneity between the two actually allows you to produce far more performance.
- 8:11
Um, the other thing is that, you know, one of the, one of the problems is that if you have in-flight batching, you have many tokens coming in at the same time that are in different phases of generation, right?
- 8:21
If you have a request that's doing prefill and a request that's doing decode on the same machine, you get scheduling conflicts, and the scheduler basically has to decide whether or not it handles new tokens.
- 8:29
Uh, there's some te-techniques to handle this, like in-flight batching and chunk piggybacking. Uh, or sorry, chunk, chunk piggybacking. But, um, generally, there is a cost to doing that mutual scheduling, so splitting this out makes the scheduling simpler.
- 8:42
Now, there's an asterisk to this, which is that... Uh, sorry, really quickly, and I'll, I'll go over the, the performance numbers. I'm gonna use Llama 70B as an example.
- 8:49
Right, right here, we have on our left axis or the-- or Y-axis, we have the tokens per second per GPU. On our X-axis, we have the tokens per second per user, right?
- 8:57
Up and to the right is better. Um, if we choose one operating point at latency, disaggregating on the same number of GPUs, sixteen total H100s, we can achieve up to two times, uh, the tokens per second per GPU at a fixed latency, which means that you're now paying two times less for your application.
- 9:16
Um, there are some constraints, though. Uh, the use case really does dictate performance for, for disaggregation. Uh, for example, low input length use cases have little to no speed up because, uh, you don't ha- actually have as much of the scheduling problem.
- 9:31
They're very prefill light, so you're basically just doing decode the entire time. Um, and then per the graph, disaggregation, and I'm... I'll go back to the graph, actually. Disaggregation is, is, is useful in, usually in the middle of the graph.
- 9:43
In very high, high latency, high throughput scenarios, which would be the left and top of the graph, and low latency, low throughput scenarios, the bottom right, uh, aggregated tends to reconfer- reconverge with disaggregated and produce a little bit more performance in those cases.
- 9:59
Uh, that being said, for a lot of user, you know, interactive applications, disaggregation makes the most sense because users tend to read that between or care about things in the realm of twenty to
- 10:13
two hundred tokens per second. Um, the other thing that's kind of a caveat about this is that configuration is really important. Um, since you're separating these two phases into prefill and generation, the balance between the number of workers for prefill and decoding dictates the performance.
- 10:30
So for example, if you have too many decode workers, you're pref-- you're, you're basically gonna have, uh, decode workers that are starving for work. And if you prefill workers, uh, they're going to be generating work for the decode workers, and the decode workers are gonna be being pushed down by, you know, just an inc-increasing amount of load
- 10:46
and increasing queue depth. Um, and the other thing is that mo-modifying this is kind of expensive and hard because the balance between prefill and decode depends on the paral-- the parallel configs of each.
- 10:58
So it's like this really wide configuration space. One other thing that we've talked about with respect to scale is routing. So we talked about how this KV is important.
- 11:08
Um, one of the things that we have to do for pref- uh, prefill decode disaggregation is that we need to actually transfer the KV between machines. And, uh, in some sense, there is actually an affinity for some me-machines to do some work since the KV cache of previous requests is actually stored on those GPUs or offloaded onto,
- 11:27
uh, system memory, host, or external storage, uh, during the course of, of inference. So, um, I actually labeled this wrong. Uh, this is not the smart router. Um, uh, this in, in, in, in a naive case, you would route pretty much exclusively randomly, right?
- 11:45
Um, or, or towards, uh, you know, s- anyth-anything, right? So in this case, we're biasing towards... We're, we're not biasing towards anything. We're, we're sampling randomly. Alternatively, uh, you know, if we're talking about this, uh, if routing to worker three in this case, it could also be that you're optimizing for purely your KV match.
- 12:02
Um- If you're doing... Uh, uh, this is actually inverted. If you're doing a, a KV-based router, um, uh, you, uh, may end up biasing towards machines that have too high KV load and therefore are not going to be able to handle the requests, you end up with queuing.
- 12:17
Uh, w- in a smart case, you actually wanna minimize the, uh, sort of this cost function that includes both the amount of prefix match that you can get from, uh...
- 12:28
Or m-maximize the prefix match that you can get from, uh, the work that's already been done on that node and the amount of load that it already exists on that node.
- 12:36
Um, and as you scale out, as you ge-get more and more GPUs, um, in a, in a d- in a deployment, you actually end up with more and more represented KV space that's local to those machines.
- 12:49
And because of that, having a larger and larger deployment means that you get an asymptotically increasing, uh, KV cache hit rate, which means that you're d-doing less and less prefill work over time.
- 13:01
So routing, you know, here to give it a report card, increases your speed and cost, and doesn't really have an effect upon quality because it's, it's doing the same work that it would normally do.
- 13:11
Um, now we talk about structure. Structure is really important because we have a lot of these workloads that you guys have probably seen at AI engineers world fair, like agents, for example.
- 13:20
Um, agents impart a structure on the workload in that they have moderately predictable usage patterns between concurrent requests. So an example here is inference time scaling. This is a cool graph we'll go over really quickly.
- 13:32
For example, we have three models here. In green, we have an 8B model. In yellow, we have a 49B model, and in red, we have a 235B model. We find that with inference time scaling, that is to re-query the model and to, you know, prompt it to reconsider its results or reason more about its results, we can
- 13:49
produce better and better results. And you actually see this really interesting trend where, uh, y- with about three or four times of re-querying, we can see that the 8B model i-is basically on par with respect to quality as the 49B model, and the 49B is almost on par with respect to quality as the two fif- thirty-five B
- 14:08
model. Um, and we note here that, like, the cost of querying that 8B model,
- 14:16
you know, uh, even querying it multiple times, is actually lower than querying the, uh, the larger model, right? And i-in this sense, you know, we, we ba- we basically see that inference time scaling can be considered sort of as, uh, you know, increasing quality at the cost of speed and at the cost of, uh, cost, right?
- 14:37
Because you're, you're re-querying it. But alternatively, if you keep la- quality fixed, um, you can basically get lower latency and lower cost by using a smaller model and re-querying it multiple times.
- 14:52
And the structure that we infer or from doing that re-querying allows us to do better scheduling. So in this graph, we have a series of curves that represent basically, uh, the runtime of a, of a given reasoning example, uh, from the natural plan dataset and the, uh, basically the, the, um, concurrency.
- 15:12
So this is like a graph of, like, how many you can r- how many in- concurrent instances you can run at once. And, uh, we sample across all this, uh, this concurrency.
- 15:20
We see that implementing disaggregation gives us a small benefit, uh, mostly because this dataset is very ISL, uh, sh- short ISL, long OSL. Um, you don't get a to- whole ton of benefit from disaggregation in this case.
- 15:32
Basically, making, uh, you know, removing a round trip by making the q-q- re-queries come from the router instead of coming from the user or the client on the outside allows you to really decrease these round trips with respect to latency.
- 15:45
And then on top of that, making the router aware and making the LLM scheduler aware that you are doing, you know, repeat work, you're re-querying it, actually gives you an increased benefit, that is the red line to the green line.
- 15:58
That is to say, amongst a wide variety of models, a- and if we assume that the quality is fixed, we can actually use inference time scaling and some smart techniques in order to significantly de-decrease latency and increase throughput while maintaining the same quality.
- 16:13
One last thing, and I'm gonna go through this really quick because I'm getting low on time, is manipulating K-K and V values, right? We've sort of talked about how before there's this work that we do in prefill that we don't wanna lose.
- 16:24
We d- we do routing to ensure that we, uh, don't lose this KV. And, you know, if we have like a workflow where we know the runtimes of things.
- 16:34
So for example, if we do a, you know, a tool call, for example, and we know this tool call takes a moderately deterministic amount of time, we basically end up with this, this, uh, KV eviction, right?
- 16:48
If we have a tool call that takes thirty seconds, the KV is gonna be swept out from HBM, and you're not, you're not gonna be able to use it in the future because it's no longer being cached.
- 16:56
But if we know that it's going to be used again, why not just offload it, right? Basically, inference time scaling gives you structure to manipulate your KV. Tool calling gives you structure to manipulate your KV.
- 17:09
So instead of doing another prefill, uh, the second time, you might, for example, do prefill once, do, uh, you know, the LLM call, the decode once, uh, move it to host memory, and then, you know, at the time at which you expect the tool to complete, you move it right back into me- into GPU memory, so that
- 17:26
it's ready for the next LLM call that will include this added context from the tool.
- 17:31
So KV manipulation, you know, again, increases your speed and decreases your cost while also, you know, improving your quality. Or not improving quality, keeping quality constant. Um, the last thing that we have to talk about here is dynamism.
- 17:44
Um, worker specialization is really important. As I said, since you have, uh, different cha- characteristics of disaggregation at different input sequence lengths and output sequence lengths, you actually want to have a mix of aggregated and disaggregated workers based on where you are in the OSL/ISL histogram.
- 18:01
So at, you know, lower input sequence lengths and higher output sequence lengths, you might want to do aggregated with a higher tensor parallelism. In the middle of the range, you may wanna use-- In the middle of the input sequence range, you may, may wanna use disaggregated.
- 18:13
And in the, you know, long context, uh, co- You know, its regime, you may wanna use disaggregated with context parallelism. Now, uh, again, this differs model to model, uh, and this is just an exemplary graph.
- 18:26
But, um, generally, if you specialize workers, you can also increase, uh, increase your speed, decrease your cost while keeping quality the same. Because again, you're not actually touching the execu- what the model is executing.
- 18:39
You're not touching the math it's doing. Um, one last thing about dynamism is load balance is quite important. As I mentioned earlier, doing... looking at the amount of P and D workers is really important to determine whether or not your disaggregated deployment is going to be successful.
- 18:56
So for example, if you have a histogram that you initially create your configuration based off of, for example, if you have app A and app B that have, like, these two input sequence length and output sequence lengths, you may end up with a scenario where a change in user distribution causes significant issues with your deployment.
- 19:13
Your de- in this case, by when you increase your s- s- input sequence length and output sequence length by a little bit, but more your input sequence length, you may create more demand for prefill workers than you do for decode workers, so your balance will change over time.
- 19:27
And this has been empirically proven by a wide variety of people that publish data. Um, and you actually have to do auto-scaling across these two, you know, types of instances in real time to account for changes in user usage distribution of your platform.
- 19:45
Um, so in this case, dynamic load balancing, uh, increases your speed and keeps your costs low. But mostly, it's, it's really just essential to ensuring that disaggre- disaggre- disaggregation actually works to maximum potential.
- 19:59
Um, okay, last things. Uh, here is the, uh, Dynamo repo. It's right here. Um, uh, it's [REDACTED:url]. Um, we also have a Dynamo meetup that is being hosted tomorrow, Thursday from 5:00 to 8:00 PM here in San Francisco.
- 20:13
Uh, please come. We're gonna be talking a lot more about how we actually implement these things at the event. [outro music]