KV Cache-Aware Routing and P/D Disaggregation on Kubernetes — Yuchen Fama & Ashish Kamra, Red Hat

All right. Um, welcome everyone to yet

another inference talk. I hope you have

had a good conference so far. And u, so

in this session, I mean I'm sure you

people who have been in the room uh must

have heard these terms many times by

now. So we're going to do a little bit

more deep dive into the challenges of

LLM deployments for agentic workloads

and uh in this session we'll focus

specifically on KV cache away routing

and uh PD disagregation

um and also you know when you when you

look at public inference uh benchmark

results you are typically looking at

very steady state isolated highly

sanitized numbers and what those

benchmarks actually don't show you u is

the chaotic reality of multi-turn

interactions, massive context

fluctuations which are very typical of

agentic workloads. So we'll also try to

pull the curtain back on some of those

complexities. Um by by way of

introduction uh my name is Ashish Kamra.

I'm a senior manager of performance

engineering at Red Hat. And with me

>> hi I'm Yuch Chen. I'm the product

manager at Red Hat Inference working

closely with VLM and AMD core

maintainers. also a contributor myself.

>> So here is the agenda for the next 20

minutes or so. Um Euchen will start with

an analysis of inference behavior in the

agentic era and some of the core

characteristics and challenges. Uh next

we next you will walk us through the KV

cache um utilization and management

strategies.

I will break down the mechanics of

pre-fill decode disagregation and walk

you through some some results and then

Euchen will again bring it all back

together with our ongoing case study on

our favorite open coding model GLM 5.2.

Um and just a couple of uh sources from

our side if you are more interested in

learning more about open source

inference we have a free course free

course on deep learning.ai AI uh by

Cedric and with Andrew Ning. Um and the

other is a series of blogs on the Red

Hat developer portal on distributed

inference concepts uh troubleshooting

and deployment patterns.

Uh and for those who may not be aware

since Red Hat is better known as the

Linux company for enterprise Linux and

uh the Kubernetes company for Open Shift

uh but more recently we are also a major

player in open source AI inference with

uh us being the top contributor in VLM

LLMD and the case of projects and also

uh having incubated guide LLM for

benchmarking LLM compressor for model

quantization and speculators for uh

speculative uh decoding models and we

also bring it bring all of that together

in a optimized model hub on hugging

phase under the Red Hat AI arc.

Um and we are also building the platform

for the next wave of agentic inference

workloads and with that I will hand over

to you to uh walk you through more of

it.

So we are currently um at this

inflection point moving from the era of

classic inference to the agentic era. So

when we look at the real world agentic

work workloads such as uh sweet bench

and also watrices from real world cloud

code sessions they fundamentally break

many assumptions we made with classic LM

serving. uh as you heard actually many

times in previous sessions for example

multi-turns and new standard we found

from a few turns all the way to 3,000

turns and also because agent frequently

reuse the uh system prompt and the total

definitions we usually see super high

cash hit rate um oftentimes well

exceeding 90%. Uh another thing is input

output ratio is uh is massive oftentimes

over a 100 ratio and even higher and in

many cases and on top of that the

context management is is incredibly

complex due to this high variance

because we can't just simply take the

average and oftentimes we need to look

at the distributions and the P90 numbers

especially when you do uh capacity

planning and also we observe really

interesting patterns like sub Asian

panel which is which further complex uh

complicates scheduling. So to help

communities study um this patterns we

collaborate with Google thank you and

also IBM our parent company to add uh a

a trace replay tool in the inference

perf you heard from earlier sessions u

from Ashoken and Jason. Um so yeah feel

free to check it out and the link is

here.

Uh next slide. Oh, so transition from

the class uh the characteristics um we

just saw for agentic workloads. We're no

longer chasing this um this this raw

throughput in a steady state. We often

need to optimize uh for example

interactive latency and they're very um

highly volatile and client-driven

context because user and you know client

define the prompt structure. So this

introduced several critical challenges.

First of all, KV cache management

becomes super volatile because the

context is client determined as I said.

So oftentimes we face this like you know

frequent evictions and rewrites and

secondly we also need to tune um the

engine like VM with upper layer uh

scheduling and routing.

It needs that coordination such as

prefix routing especially when latency

becomes a primary uh scheduling matrix

rather than like a secondary or

afterthought. And thirdly, we also need

to rethink our metrics. For example, we

need to measure cats throughput

separately. Why? Because on the right,

it's really clear that economic stakes

is very high. So, this is the uh

anthropic API pricing. You also heard

from earlier sessions. There's 10x cost

difference between cash and non-cash

tokens. So, 10x difference on your um

token balance sheet is is pretty serious

impact on your business.

So next let's let's look at how the KV

cache is um both utilized and managed in

LMD. So LMD router has this really

flexible um endpoint picker plugins we

call the EP that can route the request

to the optimal pods and that meet the KV

cache locality and also the load

criteria. So the EP continue probe each

pods like VM pod matrix to score each

pod on like the running for example

running and waiting request and then the

KV cache utilization also prefix uh

cache availability and so we can

schedule requests to the optimal pod

with the lowest load and also highest

possibility to um to of a cache hit. So

um going down from to the KV cache

management layer actually you also heard

from earlier session right before this.

So for agentic sessions when you have u

hot warm and cold cache our current

effort focus on for example um more

offloading tiers like NVME SSD and also

uh file system XF along with KV ccentric

store um like uh moon cake and also

implementing smarter and session a wire

eviction policies such as priority and

also session pinning to uh ensure this

uh really important you know the the

context persists exactly when and where

it's needed.

So, I'm gonna play this um video really

quick. Uh it's a it's a short demo.

>> Stand here so you can look at it.

>> Okay.

So,

okay. So, this is a example of a KV

cache bar routing. As you see, when we

send the very first request and it

populate the KV cache, it takes roughly

3 seconds. And when we actually look at

where it's you know the KV cache uh is

going there's no KV cache hit because

it's the very first turn. And then when

we have the second turn the request

actually reuse a KV cache because as you

see the system prompt is the same and

this time takes about one seconds. And

then when you actually look at the uh

pod address exactly the same because we

define the KV cache. Now going to the

third turn a new request with different

system prompt. Now it takes about three

uh seconds and as you see you know right

now and we don't find any KV cache here

because you can tell it's different pod

address and then if you just change the

user prompt and keep the same system

prompt and the next turn you you reuse

the KB cache and in this in this time it

takes roughly about uh one second. Yeah.

So it's a pretty intuitive demo and um

I'll turn it to Ashish to talk about the

next side but before that what does

problem does it solve? So often times

the prefix routing KB cache routing

helps you solve the TTFD problem and of

course you'll improve your lat uh your

your throughput but oftentimes for

agentic workload is not just a TTFT your

throughput is about your inter token

latency how do we solve that so preview

decode disagregation is a really uh

powerful technique but there are times

there work at times it doesn't work so

I'll turn it to Ashish to give you a

preview of um of the PD uh disregation

So before we dive into PD, let's just uh

look at what LLMD is. So LLMD is a high

performance Kubernetes native and

actually now works on non-cubernetes

environments as well. Distributed LM LLM

inference framework hosted under the

CNCF umbrella. LLMD provides a unified

intelligent control plane designed

specifically for agentic era of

inference workloads. Well, Euchin

already talked about the router and the

EP at the top of the slide. Um, the

other aspects are workload APIs such as

leader worker set and disagregated set

that orchestrates complex multi-

multi-node model execution and then

autoscalers that monitors capacity

bounds and real-time traffic mixes to

independently scale up and scale down uh

your pods depending on the system load.

So now look now let's look at uh prefill

decode disagregation in detail. Um uh

okay so why does PD exist in the first

place? So one of the most powerful

patterns implemented by LLMD is prefill

decode disagregation and you must have

heard from some of the previous talks as

well. So what happens is in in a nonPD

situation in aggregated serving one pod

is responsible for optimizing both your

time to first token and your inter token

latencies. Uh but in PD prefill and

decode become independently scalable

inference pods. But to understand why we

actually need this we have to look at

the physics of LLM execution.

colloccating uh both prefill and decode

tasks on the same GPU creates something

called as phase interference. Prefill

phase is the phase that creates the KV

caches for your initial prompt. It wants

high compute. It's highly bursty uh

utilizes GPUs at uh high flops and and

thrives on large batch parallelism to

process the prompts and builds the

initial KV cache. The decode phase on

the other hand is generating one token

at a time and it's more me memory

bandwidth hungry. It's highly latency

sensitive and requires high heavy cache

residency. So in a in a in a traditional

aggregated pod if you if there's a

sudden influx of a long prefilled palm,

it will completely stall the ongoing

decode token generation process causing

massive problems and jitter in user

streaming latency.

So, so how does PD actually work in

practice in LMD? So, LNMD uses um uh you

know like okay, we'll start with step

one. A incoming request hits the gateway

router which dynamically evaluates

cluster states using something known as

the endpoint picker you talked about and

schedules the request to use PD

disagregation selecting the optimal

prefill and decode workers. The router

then coordinates the transaction

directly with the designated pre-fill

worker. The pre-fill worker processes

the prompt, construct the initial KV

cache of the prompt and outputs the

standard KV transfer metadata. Um, and

the target decode worker actually pulls

the computed KV caches um, uh, across

the network fab fabric utilizing uh, the

KV transfer metadata that the uh, uh,

prefill pod had generated. Um okay so

with that yes that's kind of how uh PD

is implemented in practice in LMD and

next I would like to show you some uh

experimental results on where PD

actually shines. So in this graph you

can see that um

uh in in in the standard aggregated

deployment which is the top red line uh

the P99 ITL uh hovers roughly around 900

milliseconds and you can you can see

some fluctuations um up and down and but

the the bottom blue line is the P99 uh

inter token latency on a PD deployment

and you can see that it's drastically

almost nine times better at 100

millconds and it's also much smoother uh

than the aggregated serving

and uh this is some of our own internal

results at Red Hat. So for a GPOSS 12B

model uh 16 H100s

uh the aggregated config is four

replicas tensor parallelism 4 and the

disagregated is two prefilled 2D code

all with tensor parallelism 4. It's a

highly multi-turn workload with a 10,000

token prefix and 128 tokens for every

turn every turn. So, so this is a great

chart like you can see at the bottom

most line is a standard aggregated

config that's uh is doing the default

Kubernetes scheduling and uh and it's

aggregated. So that's kind of our

baseline and then the middle blue line

is still aggregated but with the LLMD uh

KV cache aware routing and you can

almost see the gains just just based on

the routing and the red line is actually

the PD uh the pre-fill decode config

with two pre-fill and two decode workers

and you can actually see that like it's

very similar to the aggregated config at

the lower concurrency regimes and uh

even and and very similar at the higher

concurrency regimes but it's actually

the middle part of the concurrency

regime that PD actually shines

and and these are some of the the

classic parita curves that we see when

you actually do PD and uh aggregated

side by side. So these results are again

from the GPTOSS 12B model 64 H100s

aggregated is eight replicas TP8 and

this a is uh three prefilled 5D code

again TP8 and a pre-filled heavy

workload with like 5,000 average input

sequence length and 500 output sequence

length and you can actually see the blue

line is the the PD curve and the red

line is the aggregated curve and the PD

curve kind of dominates um uh the

aggregate curve across the entire

interactivity spectrum.

Okay, but I don't want to leave you guys

that PD is the answer to everything and

it's a magic bullet. But um it's uh it's

essentially a separation phase

separation trade-off and not a magic

bullet. So we created this uh matrix to

help you decide when PD might be uh good

for you. So if you're managing long

context uh with high ISL OSL ratios and

you if you have a large model that

you're serving that can that you can

apply rich model parallelism techniques

um you're facing that middle concurrency

regime uh that I I showed you in the

previous graphs and and the very

important part is that if you want uh

strict ITL streaming requirements like

you want the you want the token

generation to be uh much more smooth um

then you want to consider PD but we also

saw that it requires transfer of KV

caches from your pre-filled workers to

your decode workers. So you must pro

process an advanced uh high-sp speeded

network fabric like uh RDMMA or rocky to

support that KV cache transfer. And if

you do not have such requirements, short

moderate context, any model size, low

concurrency regimes or uh if you have

strict TTF requirements because you can

actually tune them on an aggregate

serving um and you the biggest point is

like if you don't have the network

fabric to support those KV cache

transfers. So you might actually just

want to stick with aggregated.

So here is my key takeaway from all of

this. So architecting this complex

platform requires balancing a lot of u

knobs and a highly multi-dimensional

design space all of which is supported

in LLMD. As you saw the scheduler must

support or constantly evaluate SLO

targets uh QEPs KV cache locality

metrics PD ratios and network topologies

to be able to route the request to the

optimal FOD. While in while the PD

design space you you need dynamic PD

rate matching to adapt to PD ratios

because you know you can start with a

static PD ratio but it needs to evolve

with the autoscaler as the traffic

changes um and you need uh yeah

autoscaling to scale PD pools

independently

um and constantly tweaking model

parallelism techniques like tensor

parallelism data parallelism uh to meet

your SLOs's.

So um I think with these uh I will hand

it over to Euchen to anchor some of the

concepts that we showed with the real

world case study of serving the GLM 5.2

model uh which is uh still ongoing as we

speak.

>> Yeah, still ongoing. You probably have

seen tons of uh impressive numbers of

GLM 5.2 on B200 when we talk to our

customers and they usually don't have

you know the luxury of B200. They have a

lot of H200. So we have to figure out

how to like put all the knobs together

and make GM 5.2 work really well for

cluster of of H200. So uh we let's

anchor all the concept together. Um we

went through for example the uh KV cache

routing PD disagregation. We kind of

call them a wildl path in LMD and also

we combine with different parallelism

strategies to so we can uh independently

uh scale prefuel paths because for

agentic workload is super uh long you

know like heavy prefill. So uh in this

case we designed the prefuel pool using

up to three workers optimized for uh

high throughput uh with deep and then

for decoup we use uh one dedicated

worker and um that's optimized for for

low latency. So we use Nixo for

efficient KV transfer between the pools

and also with the each worker we have

the leader worker set group uh with TP1

DP8 and also uh EP8 uh expert

parallelism 8. So the architecture is

just highly modular because you can uh

actually scale the throughput by simply

adding uh preview workers without

reconfiguring and um the decoup. So uh

this highlights how AMD effectly

effectively managed the complexity of

combining like PB and DB and EPI scale.

And also we found some interesting fun

fact actually a couple days ago. Um B B

B B B B B B B B B B B B B B B B B B B B

B B B B B B B B B B B B B B B B B B Bf6

uh BF16 KV cache actually is faster than

using like FPA uh KV cache for longer

preview. Um this is also like we

continue to explore and found like more

interesting patterns, but more

importantly uh we want to kind of just

show the result really quick. So um for

this uh data set agentic workload data

set the ISO OSL ratio is pretty high 45

to1 ratio preview is uh is really the

constraint you can tell um with 2P even

1D we have um 4x passer TDFT and also 60

uh% more requests and this is continuous

like work in progress so the next step

is we need to also put the upper layer

lower TTFT and also adding more more

preview replicas so um I know we're

running out of time really quick. Uh we

um the fundamental shift for agentic

workload we're continuing to uh have

this um uh agentic north uh northstar uh

with session graph orchestration program

award scheduling uh state reuse life

cycle and also the uh agentic benchmark

um we're working on. So you can find

them uh in AMD upstream AMD and also you

know feel free to join the SIG group and

uh and contribute and um this is the

very last slide. So distri distributed

inference is not challenge uh every

single comp a single company can solve

along. We're proud to be uh building

this uh future in the open alongside our

incredible ecosystem collaborators uh

core wave Google IBM Nvidia growing list

of launch partners and industry

adopters. So if you're passionate about

the future of opensource inference, we

invite you to join us. We do have a

booth downstairs. Feel free to stop by,

ask us any questions. And uh thank you

so much for your time.

[applause]

>> [music]