KV Cache-Aware Routing and P/D Disaggregation on Kubernetes — Yuchen Fama & Ashish Kamra, Red Hat

A steady-state inference benchmark can tell you how fast a server handles a controlled stream of requests. An agent session brings a different problem: the same context returns across many turns, its length changes, and sub-agents suddenly add concurrent work. Red Hat’s Ashish Kamra, senior manager of performance engineering, and Yuchen Fama, product manager working with vLLM and llm-d, begin with that mismatch. Their two main tools are KV cache-aware routing and prefill/decode disaggregation.

Red Hat’s surrounding inference stack includes tools for benchmarking, model quantization, and speculative decoding. This session focuses on coordinating the model engine with distributed serving once a client controls the structure and timing of its context.

The agentic workloads examined in the talk have several properties that change capacity planning:

• Long sessions: Interactions range from a few turns to 3,000 turns. Each request belongs to a continuing sequence.
• Repeated input: Reused system prompts and tool definitions produce cache hit rates often exceeding 90%. Much of the incoming context has already been processed.
• Input-heavy work: Input-to-output token ratios often exceed 100:1. Counting output tokens alone hides most of the context the server handles.
• Uneven demand: Context lengths vary, and sub-agent fan-out complicates scheduling. Distributions and P90 values matter when sizing capacity; an average smooths away the larger requests and bursts.

Trace replay brings those pressures into evaluation. The collaboration with Google and IBM adds a replay tool to inference-perf to study the request patterns that agent sessions create. Repeated context and sudden concurrency occur together, so testing either in isolation misses part of the scheduling problem.

The KV cache holds state computed while processing a prompt. Reusing that state avoids repeating work on an unchanged prefix. But the client decides when context grows, changes, or returns. Frequent evictions and rewrites make cache management volatile, so engine-level caching needs help from scheduling and routing.

That shifts the performance target toward interactive latency and makes cached throughput worth measuring separately. The Anthropic pricing example presented in the talk has a 10× difference between cached and uncached input-token costs. It illustrates the economic value of reuse under that pricing scheme; it does not measure a 10× reduction in the cost of running an llm-d cluster.

The llm-d router’s endpoint picker plugin, or EPP, scores candidate pods using both load and cache information. It probes running and waiting requests, KV cache utilization, and prefix-cache availability. Placement balances the chance of a cache hit against the work already waiting on a pod. A cached prefix offers less recomputation, while a shorter queue offers less waiting; the picker needs both signals to choose an endpoint.

Routing and retention address different parts of the problem:

• Cache locality: The endpoint picker sends a request toward a pod that can reuse its prefix while accounting for load.
• Offloading tiers: The described development work explores NVMe/SSD, filesystems such as Ceph, and KV-oriented stores such as Mooncake to retain hot, warm, and cold session state.
• Session-aware eviction: Priority and session pinning aim to keep important context available when a session needs it again.

A router can only find reusable state that the cache system has retained. Additional tiers create room to preserve it; eviction policy decides which state remains available.

The routing demo follows one concrete change: whether the system prompt stays the same. The first request takes roughly three seconds and populates the cache. There is no hit because this is the opening turn. The second turn keeps the system prompt, returns to the same pod address, and reuses the cache. Its displayed duration falls to about one second.

The third turn changes the system prompt. It has no cache hit, lands on a different pod, and takes roughly three seconds again. The following turn changes only the user prompt while retaining this new system prompt. Reuse returns, and the duration falls to about one second. The causal steps are an unchanged prefix, available cached state, placement that reaches that state, and less repeated prompt processing. The pod address makes placement visible; the reported cache hit establishes reuse.

What changes across these four turns? The comparison below puts prompt continuity beside cache behavior and duration. It shows two cold-to-warm transitions. Changing the system prompt creates a new starting point, while changing the user prompt can leave the shared prefix reusable.

Prefix-aware routing primarily helps the wait before generation begins—time to first token, or TTFT—and can improve throughput by avoiding repeated work. Once output starts streaming, another measure matters: inter-token latency, or ITL, the time between generated tokens. Faster startup does not by itself prevent a stream from stalling.

llm-d supplies the distributed control plane around model execution. Alongside the router, workload APIs such as LeaderWorkerSet and disaggregated sets organize multi-node execution. Autoscalers respond to capacity and traffic so pods can scale independently. This coordination becomes especially useful when prefill and decode run in separate pools.

Aggregated serving asks one pod to optimize both startup and streaming. The two execution phases have different needs:

• Prefill: Processes the initial prompt and builds its KV cache. It is compute-intensive, bursty, and benefits from large-batch parallelism.
• Decode: Generates one token at a time. It needs memory bandwidth, resident KV state, and predictable latency.

When both share a GPU, a long incoming prompt can interrupt ongoing token generation. This phase interference appears to a user as pauses and jitter in a stream that had already started.

In the P/D request flow, the gateway router evaluates cluster state through the endpoint picker and selects a prefill worker and a decode worker. It coordinates prompt processing with the prefill worker, which builds the initial cache and produces KV transfer metadata. The selected decoder uses that metadata to pull the computed cache across the network. Prefill and decode now run on independently scalable inference pods.

Where does the state move when execution is split? The flow below separates placement decisions from KV movement. The decoder pulls the result of prompt processing instead of rebuilding it. Separating the GPU work can protect streaming, but moving its state introduces a network requirement that aggregated serving avoids.

The first latency comparison reports P99 ITL near 900 milliseconds for aggregated serving and about 100 milliseconds for disaggregated serving. P99 describes the slower tail of token intervals. The disaggregated curve is also smoother, so the improvement concerns both the length and variability of streaming pauses.

The following internal Red Hat experiment uses GPT-OSS 120B on 16 H100s. Aggregated serving has four replicas with tensor parallelism four; disaggregated serving has two prefill and two decode workers, each also using tensor parallelism four. The highly multi-turn workload has a 10,000-token prefix and 128 tokens per turn.

Its comparison separates three choices: aggregated serving with default Kubernetes scheduling, aggregated serving with llm-d cache-aware routing, and disaggregated serving. Routing alone produces gains. P/D is similar to aggregated serving at low and high concurrency, with its strongest advantage in the middle band. That band has no numeric threshold stated here; finding the favorable range requires a load curve for the deployment being sized.

A separate prefill-heavy test uses GPT-OSS 120B on 64 H100s: eight aggregated replicas at tensor parallelism eight, compared with three prefill and five decode workers at the same parallelism. Average input length is 5,000 tokens and output length is 500 tokens. Here, the presented P/D Pareto curve dominates the aggregated curve across the tested interactivity range. A Pareto comparison asks which configuration offers a better combination of throughput and responsiveness. The different result reinforces that workload and pool sizing determine the benefit.

P/D makes a phase-separation tradeoff. Its appeal grows when long contexts, high input-to-output ratios, and strict streaming requirements make prompt processing interfere with generation. Large models with useful model-parallel execution options and the favorable middle concurrency regime are further reasons to consider it.

The network belongs in that decision. The talk recommends a high-speed fabric such as RDMA or RoCE to support prefill-to-decode KV transfer. Without a suitable fabric, stay aggregated. Aggregated serving is also a reasonable choice for short or moderate contexts, low concurrency, or workloads whose main requirement is TTFT, which can be tuned within an aggregated deployment.

Once P/D is deployed, the prefill-to-decode ratio becomes another variable to manage. A static starting ratio needs to evolve as traffic changes. The scheduler considers service-level objectives, queue depths, cache locality, pool ratios, and network topology together to choose placement. Independent autoscaling and adjustments to tensor and data parallelism let the system add capacity where work is accumulating.

This connects the two techniques: routing reduces unnecessary prompt processing, while disaggregation lets necessary prompt processing run apart from active generation. Their combination still needs rate matching between the pools. The amount of prefill and decode capacity should follow the changing traffic mix and the latency objectives.

The closing case study starts with a hardware constraint: customers have H200s, even when impressive published results use B200s. Serving GLM-5.2 well on that existing hardware requires combining cache-aware routing, P/D separation, and parallelism. The companion technical article develops the workload and deployment pattern in more detail.

The design described in the recording supports up to three prefill workers optimized for throughput and one dedicated decode worker optimized for low latency. Each worker uses a LeaderWorkerSet group with tensor parallelism one, data parallelism eight, and expert parallelism eight. NIXL, identified in the companion article, handles KV movement between the pools. More prefill workers can be added without reconfiguring the decoder, so additional input-processing capacity does not require rebuilding both sides.

One tentative finding complicates the cache-format choice: Yuchen reports that BF16 KV cache was faster than another cache format for longer prefill. The comparison format is not established clearly enough to name, and the team was still investigating the result. For this case study, cache precision remained a performance choice to explore rather than a settled prescription.

In the closing case study, Yuchen reports preliminary results of 4× faster TTFT and 60% more requests with two prefill workers and one decode worker on an agentic dataset with a 45:1 input-to-output ratio, where prefill is the constraint. These are the speaker’s reported figures, not an established controlled P/D-versus-aggregate improvement: the comparison baseline is unspecified. The architecture supports up to three prefill workers, a design limit separate from the two-prefill, one-decode configuration reported for these results. The companion article’s separate experiments cannot supply the missing comparison baseline for these recording results.

The next steps follow the remaining bottleneck: tune the upper-layer scheduler to reduce TTFT and add more prefill replicas. The broader development direction includes session-graph orchestration, program-aware scheduling, state-reuse lifecycle management, and agentic benchmarks. These continuing efforts extend scheduling toward a connected agent program, whose turns and sub-agents share state and dependencies.

The closing invitation is to build that work upstream through llm-d and its special-interest groups, alongside collaborators including CoreWeave, Google, IBM, and Nvidia. The H200 serving guide provides a practical deployment path with configurable routing, offloading, and pool layouts. It gives the closing question something concrete to work with: how much reusable context and prefill capacity does the workload need before its next stream can begin?