[
  {
    "startMs": 12800,
    "text": "All right. Um, welcome everyone to yet"
  },
  {
    "startMs": 15839,
    "text": "another inference talk. I hope you have"
  },
  {
    "startMs": 18560,
    "text": "had a good conference so far. And u, so"
  },
  {
    "startMs": 22720,
    "text": "in this session, I mean I'm sure you"
  },
  {
    "startMs": 24640,
    "text": "people who have been in the room uh must"
  },
  {
    "startMs": 26800,
    "text": "have heard these terms many times by"
  },
  {
    "startMs": 28320,
    "text": "now. So we're going to do a little bit"
  },
  {
    "startMs": 30080,
    "text": "more deep dive into the challenges of"
  },
  {
    "startMs": 32239,
    "text": "LLM deployments for agentic workloads"
  },
  {
    "startMs": 35440,
    "text": "and uh in this session we'll focus"
  },
  {
    "startMs": 37840,
    "text": "specifically on KV cache away routing"
  },
  {
    "startMs": 39760,
    "text": "and uh PD disagregation"
  },
  {
    "startMs": 43120,
    "text": "um and also you know when you when you"
  },
  {
    "startMs": 45600,
    "text": "look at public inference uh benchmark"
  },
  {
    "startMs": 48079,
    "text": "results you are typically looking at"
  },
  {
    "startMs": 49600,
    "text": "very steady state isolated highly"
  },
  {
    "startMs": 51920,
    "text": "sanitized numbers and what those"
  },
  {
    "startMs": 53760,
    "text": "benchmarks actually don't show you u is"
  },
  {
    "startMs": 56239,
    "text": "the chaotic reality of multi-turn"
  },
  {
    "startMs": 59039,
    "text": "interactions, massive context"
  },
  {
    "startMs": 60960,
    "text": "fluctuations which are very typical of"
  },
  {
    "startMs": 63440,
    "text": "agentic workloads. So we'll also try to"
  },
  {
    "startMs": 66479,
    "text": "pull the curtain back on some of those"
  },
  {
    "startMs": 68000,
    "text": "complexities. Um by by way of"
  },
  {
    "startMs": 70960,
    "text": "introduction uh my name is Ashish Kamra."
  },
  {
    "startMs": 73520,
    "text": "I'm a senior manager of performance"
  },
  {
    "startMs": 75600,
    "text": "engineering at Red Hat. And with me"
  },
  {
    "startMs": 78320,
    "text": ">> hi I'm Yuch Chen. I'm the product"
  },
  {
    "startMs": 80080,
    "text": "manager at Red Hat Inference working"
  },
  {
    "startMs": 81920,
    "text": "closely with VLM and AMD core"
  },
  {
    "startMs": 84400,
    "text": "maintainers. also a contributor myself."
  },
  {
    "startMs": 88880,
    "text": ">> So here is the agenda for the next 20"
  },
  {
    "startMs": 90960,
    "text": "minutes or so. Um Euchen will start with"
  },
  {
    "startMs": 93600,
    "text": "an analysis of inference behavior in the"
  },
  {
    "startMs": 96560,
    "text": "agentic era and some of the core"
  },
  {
    "startMs": 98880,
    "text": "characteristics and challenges. Uh next"
  },
  {
    "startMs": 101840,
    "text": "we next you will walk us through the KV"
  },
  {
    "startMs": 104159,
    "text": "cache um utilization and management"
  },
  {
    "startMs": 107040,
    "text": "strategies."
  },
  {
    "startMs": 108720,
    "text": "I will break down the mechanics of"
  },
  {
    "startMs": 110640,
    "text": "pre-fill decode disagregation and walk"
  },
  {
    "startMs": 112880,
    "text": "you through some some results and then"
  },
  {
    "startMs": 115360,
    "text": "Euchen will again bring it all back"
  },
  {
    "startMs": 117040,
    "text": "together with our ongoing case study on"
  },
  {
    "startMs": 119920,
    "text": "our favorite open coding model GLM 5.2."
  },
  {
    "startMs": 123840,
    "text": "Um and just a couple of uh sources from"
  },
  {
    "startMs": 126640,
    "text": "our side if you are more interested in"
  },
  {
    "startMs": 128800,
    "text": "learning more about open source"
  },
  {
    "startMs": 130399,
    "text": "inference we have a free course free"
  },
  {
    "startMs": 132560,
    "text": "course on deep learning.ai AI uh by"
  },
  {
    "startMs": 135120,
    "text": "Cedric and with Andrew Ning. Um and the"
  },
  {
    "startMs": 137840,
    "text": "other is a series of blogs on the Red"
  },
  {
    "startMs": 139920,
    "text": "Hat developer portal on distributed"
  },
  {
    "startMs": 142000,
    "text": "inference concepts uh troubleshooting"
  },
  {
    "startMs": 144319,
    "text": "and deployment patterns."
  },
  {
    "startMs": 147680,
    "text": "Uh and for those who may not be aware"
  },
  {
    "startMs": 150480,
    "text": "since Red Hat is better known as the"
  },
  {
    "startMs": 152480,
    "text": "Linux company for enterprise Linux and"
  },
  {
    "startMs": 156000,
    "text": "uh the Kubernetes company for Open Shift"
  },
  {
    "startMs": 159120,
    "text": "uh but more recently we are also a major"
  },
  {
    "startMs": 161040,
    "text": "player in open source AI inference with"
  },
  {
    "startMs": 165040,
    "text": "uh us being the top contributor in VLM"
  },
  {
    "startMs": 167840,
    "text": "LLMD and the case of projects and also"
  },
  {
    "startMs": 171120,
    "text": "uh having incubated guide LLM for"
  },
  {
    "startMs": 174000,
    "text": "benchmarking LLM compressor for model"
  },
  {
    "startMs": 176720,
    "text": "quantization and speculators for uh"
  },
  {
    "startMs": 180480,
    "text": "speculative uh decoding models and we"
  },
  {
    "startMs": 184080,
    "text": "also bring it bring all of that together"
  },
  {
    "startMs": 185920,
    "text": "in a optimized model hub on hugging"
  },
  {
    "startMs": 189440,
    "text": "phase under the Red Hat AI arc."
  },
  {
    "startMs": 192800,
    "text": "Um and we are also building the platform"
  },
  {
    "startMs": 194879,
    "text": "for the next wave of agentic inference"
  },
  {
    "startMs": 196640,
    "text": "workloads and with that I will hand over"
  },
  {
    "startMs": 198319,
    "text": "to you to uh walk you through more of"
  },
  {
    "startMs": 201599,
    "text": "it."
  },
  {
    "startMs": 205040,
    "text": "So we are currently um at this"
  },
  {
    "startMs": 207360,
    "text": "inflection point moving from the era of"
  },
  {
    "startMs": 209920,
    "text": "classic inference to the agentic era. So"
  },
  {
    "startMs": 213760,
    "text": "when we look at the real world agentic"
  },
  {
    "startMs": 216080,
    "text": "work workloads such as uh sweet bench"
  },
  {
    "startMs": 218720,
    "text": "and also watrices from real world cloud"
  },
  {
    "startMs": 221280,
    "text": "code sessions they fundamentally break"
  },
  {
    "startMs": 224239,
    "text": "many assumptions we made with classic LM"
  },
  {
    "startMs": 226799,
    "text": "serving. uh as you heard actually many"
  },
  {
    "startMs": 228959,
    "text": "times in previous sessions for example"
  },
  {
    "startMs": 230640,
    "text": "multi-turns and new standard we found"
  },
  {
    "startMs": 232799,
    "text": "from a few turns all the way to 3,000"
  },
  {
    "startMs": 235040,
    "text": "turns and also because agent frequently"
  },
  {
    "startMs": 237680,
    "text": "reuse the uh system prompt and the total"
  },
  {
    "startMs": 240720,
    "text": "definitions we usually see super high"
  },
  {
    "startMs": 242959,
    "text": "cash hit rate um oftentimes well"
  },
  {
    "startMs": 245519,
    "text": "exceeding 90%. Uh another thing is input"
  },
  {
    "startMs": 248799,
    "text": "output ratio is uh is massive oftentimes"
  },
  {
    "startMs": 252319,
    "text": "over a 100 ratio and even higher and in"
  },
  {
    "startMs": 255519,
    "text": "many cases and on top of that the"
  },
  {
    "startMs": 257840,
    "text": "context management is is incredibly"
  },
  {
    "startMs": 260400,
    "text": "complex due to this high variance"
  },
  {
    "startMs": 262639,
    "text": "because we can't just simply take the"
  },
  {
    "startMs": 264800,
    "text": "average and oftentimes we need to look"
  },
  {
    "startMs": 267120,
    "text": "at the distributions and the P90 numbers"
  },
  {
    "startMs": 269680,
    "text": "especially when you do uh capacity"
  },
  {
    "startMs": 271360,
    "text": "planning and also we observe really"
  },
  {
    "startMs": 274639,
    "text": "interesting patterns like sub Asian"
  },
  {
    "startMs": 276479,
    "text": "panel which is which further complex uh"
  },
  {
    "startMs": 279520,
    "text": "complicates scheduling. So to help"
  },
  {
    "startMs": 281759,
    "text": "communities study um this patterns we"
  },
  {
    "startMs": 284080,
    "text": "collaborate with Google thank you and"
  },
  {
    "startMs": 286800,
    "text": "also IBM our parent company to add uh a"
  },
  {
    "startMs": 289600,
    "text": "a trace replay tool in the inference"
  },
  {
    "startMs": 291440,
    "text": "perf you heard from earlier sessions u"
  },
  {
    "startMs": 294000,
    "text": "from Ashoken and Jason. Um so yeah feel"
  },
  {
    "startMs": 296800,
    "text": "free to check it out and the link is"
  },
  {
    "startMs": 298400,
    "text": "here."
  },
  {
    "startMs": 301680,
    "text": "Uh next slide. Oh, so transition from"
  },
  {
    "startMs": 304240,
    "text": "the class uh the characteristics um we"
  },
  {
    "startMs": 306560,
    "text": "just saw for agentic workloads. We're no"
  },
  {
    "startMs": 309440,
    "text": "longer chasing this um this this raw"
  },
  {
    "startMs": 312400,
    "text": "throughput in a steady state. We often"
  },
  {
    "startMs": 314400,
    "text": "need to optimize uh for example"
  },
  {
    "startMs": 316800,
    "text": "interactive latency and they're very um"
  },
  {
    "startMs": 319759,
    "text": "highly volatile and client-driven"
  },
  {
    "startMs": 321680,
    "text": "context because user and you know client"
  },
  {
    "startMs": 324320,
    "text": "define the prompt structure. So this"
  },
  {
    "startMs": 326639,
    "text": "introduced several critical challenges."
  },
  {
    "startMs": 328479,
    "text": "First of all, KV cache management"
  },
  {
    "startMs": 330240,
    "text": "becomes super volatile because the"
  },
  {
    "startMs": 332160,
    "text": "context is client determined as I said."
  },
  {
    "startMs": 334800,
    "text": "So oftentimes we face this like you know"
  },
  {
    "startMs": 337520,
    "text": "frequent evictions and rewrites and"
  },
  {
    "startMs": 340320,
    "text": "secondly we also need to tune um the"
  },
  {
    "startMs": 343199,
    "text": "engine like VM with upper layer uh"
  },
  {
    "startMs": 345360,
    "text": "scheduling and routing."
  },
  {
    "startMs": 347600,
    "text": "It needs that coordination such as"
  },
  {
    "startMs": 349280,
    "text": "prefix routing especially when latency"
  },
  {
    "startMs": 352160,
    "text": "becomes a primary uh scheduling matrix"
  },
  {
    "startMs": 354479,
    "text": "rather than like a secondary or"
  },
  {
    "startMs": 356479,
    "text": "afterthought. And thirdly, we also need"
  },
  {
    "startMs": 358960,
    "text": "to rethink our metrics. For example, we"
  },
  {
    "startMs": 360960,
    "text": "need to measure cats throughput"
  },
  {
    "startMs": 362560,
    "text": "separately. Why? Because on the right,"
  },
  {
    "startMs": 364960,
    "text": "it's really clear that economic stakes"
  },
  {
    "startMs": 367199,
    "text": "is very high. So, this is the uh"
  },
  {
    "startMs": 369039,
    "text": "anthropic API pricing. You also heard"
  },
  {
    "startMs": 371199,
    "text": "from earlier sessions. There's 10x cost"
  },
  {
    "startMs": 373840,
    "text": "difference between cash and non-cash"
  },
  {
    "startMs": 375600,
    "text": "tokens. So, 10x difference on your um"
  },
  {
    "startMs": 378479,
    "text": "token balance sheet is is pretty serious"
  },
  {
    "startMs": 380880,
    "text": "impact on your business."
  },
  {
    "startMs": 383440,
    "text": "So next let's let's look at how the KV"
  },
  {
    "startMs": 385440,
    "text": "cache is um both utilized and managed in"
  },
  {
    "startMs": 388240,
    "text": "LMD. So LMD router has this really"
  },
  {
    "startMs": 391440,
    "text": "flexible um endpoint picker plugins we"
  },
  {
    "startMs": 394319,
    "text": "call the EP that can route the request"
  },
  {
    "startMs": 396560,
    "text": "to the optimal pods and that meet the KV"
  },
  {
    "startMs": 398960,
    "text": "cache locality and also the load"
  },
  {
    "startMs": 401680,
    "text": "criteria. So the EP continue probe each"
  },
  {
    "startMs": 404400,
    "text": "pods like VM pod matrix to score each"
  },
  {
    "startMs": 407120,
    "text": "pod on like the running for example"
  },
  {
    "startMs": 409440,
    "text": "running and waiting request and then the"
  },
  {
    "startMs": 411440,
    "text": "KV cache utilization also prefix uh"
  },
  {
    "startMs": 414240,
    "text": "cache availability and so we can"
  },
  {
    "startMs": 416319,
    "text": "schedule requests to the optimal pod"
  },
  {
    "startMs": 418080,
    "text": "with the lowest load and also highest"
  },
  {
    "startMs": 420000,
    "text": "possibility to um to of a cache hit. So"
  },
  {
    "startMs": 423919,
    "text": "um going down from to the KV cache"
  },
  {
    "startMs": 426400,
    "text": "management layer actually you also heard"
  },
  {
    "startMs": 427919,
    "text": "from earlier session right before this."
  },
  {
    "startMs": 430400,
    "text": "So for agentic sessions when you have u"
  },
  {
    "startMs": 432720,
    "text": "hot warm and cold cache our current"
  },
  {
    "startMs": 435120,
    "text": "effort focus on for example um more"
  },
  {
    "startMs": 437280,
    "text": "offloading tiers like NVME SSD and also"
  },
  {
    "startMs": 440720,
    "text": "uh file system XF along with KV ccentric"
  },
  {
    "startMs": 443520,
    "text": "store um like uh moon cake and also"
  },
  {
    "startMs": 446240,
    "text": "implementing smarter and session a wire"
  },
  {
    "startMs": 448479,
    "text": "eviction policies such as priority and"
  },
  {
    "startMs": 450880,
    "text": "also session pinning to uh ensure this"
  },
  {
    "startMs": 453759,
    "text": "uh really important you know the the"
  },
  {
    "startMs": 455759,
    "text": "context persists exactly when and where"
  },
  {
    "startMs": 458479,
    "text": "it's needed."
  },
  {
    "startMs": 461520,
    "text": "So, I'm gonna play this um video really"
  },
  {
    "startMs": 464400,
    "text": "quick. Uh it's a it's a short demo."
  },
  {
    "startMs": 466880,
    "text": ">> Stand here so you can look at it."
  },
  {
    "startMs": 468400,
    "text": ">> Okay."
  },
  {
    "startMs": 472000,
    "text": "So,"
  },
  {
    "startMs": 474479,
    "text": "okay. So, this is a example of a KV"
  },
  {
    "startMs": 476960,
    "text": "cache bar routing. As you see, when we"
  },
  {
    "startMs": 478720,
    "text": "send the very first request and it"
  },
  {
    "startMs": 480639,
    "text": "populate the KV cache, it takes roughly"
  },
  {
    "startMs": 482879,
    "text": "3 seconds. And when we actually look at"
  },
  {
    "startMs": 486000,
    "text": "where it's you know the KV cache uh is"
  },
  {
    "startMs": 488160,
    "text": "going there's no KV cache hit because"
  },
  {
    "startMs": 490000,
    "text": "it's the very first turn. And then when"
  },
  {
    "startMs": 492160,
    "text": "we have the second turn the request"
  },
  {
    "startMs": 493919,
    "text": "actually reuse a KV cache because as you"
  },
  {
    "startMs": 496000,
    "text": "see the system prompt is the same and"
  },
  {
    "startMs": 498160,
    "text": "this time takes about one seconds. And"
  },
  {
    "startMs": 500319,
    "text": "then when you actually look at the uh"
  },
  {
    "startMs": 501919,
    "text": "pod address exactly the same because we"
  },
  {
    "startMs": 504240,
    "text": "define the KV cache. Now going to the"
  },
  {
    "startMs": 506639,
    "text": "third turn a new request with different"
  },
  {
    "startMs": 508639,
    "text": "system prompt. Now it takes about three"
  },
  {
    "startMs": 510720,
    "text": "uh seconds and as you see you know right"
  },
  {
    "startMs": 514000,
    "text": "now and we don't find any KV cache here"
  },
  {
    "startMs": 516959,
    "text": "because you can tell it's different pod"
  },
  {
    "startMs": 518880,
    "text": "address and then if you just change the"
  },
  {
    "startMs": 521599,
    "text": "user prompt and keep the same system"
  },
  {
    "startMs": 523360,
    "text": "prompt and the next turn you you reuse"
  },
  {
    "startMs": 526320,
    "text": "the KB cache and in this in this time it"
  },
  {
    "startMs": 528640,
    "text": "takes roughly about uh one second. Yeah."
  },
  {
    "startMs": 531120,
    "text": "So it's a pretty intuitive demo and um"
  },
  {
    "startMs": 533680,
    "text": "I'll turn it to Ashish to talk about the"
  },
  {
    "startMs": 536080,
    "text": "next side but before that what does"
  },
  {
    "startMs": 537839,
    "text": "problem does it solve? So often times"
  },
  {
    "startMs": 539680,
    "text": "the prefix routing KB cache routing"
  },
  {
    "startMs": 542160,
    "text": "helps you solve the TTFD problem and of"
  },
  {
    "startMs": 544880,
    "text": "course you'll improve your lat uh your"
  },
  {
    "startMs": 546480,
    "text": "your throughput but oftentimes for"
  },
  {
    "startMs": 548560,
    "text": "agentic workload is not just a TTFT your"
  },
  {
    "startMs": 550959,
    "text": "throughput is about your inter token"
  },
  {
    "startMs": 552640,
    "text": "latency how do we solve that so preview"
  },
  {
    "startMs": 555360,
    "text": "decode disagregation is a really uh"
  },
  {
    "startMs": 557600,
    "text": "powerful technique but there are times"
  },
  {
    "startMs": 559120,
    "text": "there work at times it doesn't work so"
  },
  {
    "startMs": 561360,
    "text": "I'll turn it to Ashish to give you a"
  },
  {
    "startMs": 563600,
    "text": "preview of um of the PD uh disregation"
  },
  {
    "startMs": 568959,
    "text": "So before we dive into PD, let's just uh"
  },
  {
    "startMs": 571920,
    "text": "look at what LLMD is. So LLMD is a high"
  },
  {
    "startMs": 574800,
    "text": "performance Kubernetes native and"
  },
  {
    "startMs": 576640,
    "text": "actually now works on non-cubernetes"
  },
  {
    "startMs": 578480,
    "text": "environments as well. Distributed LM LLM"
  },
  {
    "startMs": 581839,
    "text": "inference framework hosted under the"
  },
  {
    "startMs": 583600,
    "text": "CNCF umbrella. LLMD provides a unified"
  },
  {
    "startMs": 587360,
    "text": "intelligent control plane designed"
  },
  {
    "startMs": 589120,
    "text": "specifically for agentic era of"
  },
  {
    "startMs": 590959,
    "text": "inference workloads. Well, Euchin"
  },
  {
    "startMs": 592959,
    "text": "already talked about the router and the"
  },
  {
    "startMs": 594800,
    "text": "EP at the top of the slide. Um, the"
  },
  {
    "startMs": 598000,
    "text": "other aspects are workload APIs such as"
  },
  {
    "startMs": 600320,
    "text": "leader worker set and disagregated set"
  },
  {
    "startMs": 602399,
    "text": "that orchestrates complex multi-"
  },
  {
    "startMs": 605760,
    "text": "multi-node model execution and then"
  },
  {
    "startMs": 608480,
    "text": "autoscalers that monitors capacity"
  },
  {
    "startMs": 610800,
    "text": "bounds and real-time traffic mixes to"
  },
  {
    "startMs": 613600,
    "text": "independently scale up and scale down uh"
  },
  {
    "startMs": 616480,
    "text": "your pods depending on the system load."
  },
  {
    "startMs": 619040,
    "text": "So now look now let's look at uh prefill"
  },
  {
    "startMs": 621600,
    "text": "decode disagregation in detail. Um uh"
  },
  {
    "startMs": 625680,
    "text": "okay so why does PD exist in the first"
  },
  {
    "startMs": 629360,
    "text": "place? So one of the most powerful"
  },
  {
    "startMs": 630880,
    "text": "patterns implemented by LLMD is prefill"
  },
  {
    "startMs": 633519,
    "text": "decode disagregation and you must have"
  },
  {
    "startMs": 635440,
    "text": "heard from some of the previous talks as"
  },
  {
    "startMs": 637120,
    "text": "well. So what happens is in in a nonPD"
  },
  {
    "startMs": 640640,
    "text": "situation in aggregated serving one pod"
  },
  {
    "startMs": 643519,
    "text": "is responsible for optimizing both your"
  },
  {
    "startMs": 646160,
    "text": "time to first token and your inter token"
  },
  {
    "startMs": 648079,
    "text": "latencies. Uh but in PD prefill and"
  },
  {
    "startMs": 651120,
    "text": "decode become independently scalable"
  },
  {
    "startMs": 653279,
    "text": "inference pods. But to understand why we"
  },
  {
    "startMs": 655680,
    "text": "actually need this we have to look at"
  },
  {
    "startMs": 658000,
    "text": "the physics of LLM execution."
  },
  {
    "startMs": 660480,
    "text": "colloccating uh both prefill and decode"
  },
  {
    "startMs": 663839,
    "text": "tasks on the same GPU creates something"
  },
  {
    "startMs": 665920,
    "text": "called as phase interference. Prefill"
  },
  {
    "startMs": 668240,
    "text": "phase is the phase that creates the KV"
  },
  {
    "startMs": 670880,
    "text": "caches for your initial prompt. It wants"
  },
  {
    "startMs": 673600,
    "text": "high compute. It's highly bursty uh"
  },
  {
    "startMs": 677200,
    "text": "utilizes GPUs at uh high flops and and"
  },
  {
    "startMs": 681279,
    "text": "thrives on large batch parallelism to"
  },
  {
    "startMs": 684000,
    "text": "process the prompts and builds the"
  },
  {
    "startMs": 685760,
    "text": "initial KV cache. The decode phase on"
  },
  {
    "startMs": 688160,
    "text": "the other hand is generating one token"
  },
  {
    "startMs": 690320,
    "text": "at a time and it's more me memory"
  },
  {
    "startMs": 692560,
    "text": "bandwidth hungry. It's highly latency"
  },
  {
    "startMs": 694800,
    "text": "sensitive and requires high heavy cache"
  },
  {
    "startMs": 697519,
    "text": "residency. So in a in a in a traditional"
  },
  {
    "startMs": 700880,
    "text": "aggregated pod if you if there's a"
  },
  {
    "startMs": 703360,
    "text": "sudden influx of a long prefilled palm,"
  },
  {
    "startMs": 706079,
    "text": "it will completely stall the ongoing"
  },
  {
    "startMs": 708320,
    "text": "decode token generation process causing"
  },
  {
    "startMs": 710800,
    "text": "massive problems and jitter in user"
  },
  {
    "startMs": 713600,
    "text": "streaming latency."
  },
  {
    "startMs": 716560,
    "text": "So, so how does PD actually work in"
  },
  {
    "startMs": 718959,
    "text": "practice in LMD? So, LNMD uses um uh you"
  },
  {
    "startMs": 723760,
    "text": "know like okay, we'll start with step"
  },
  {
    "startMs": 725279,
    "text": "one. A incoming request hits the gateway"
  },
  {
    "startMs": 727600,
    "text": "router which dynamically evaluates"
  },
  {
    "startMs": 730240,
    "text": "cluster states using something known as"
  },
  {
    "startMs": 732480,
    "text": "the endpoint picker you talked about and"
  },
  {
    "startMs": 735360,
    "text": "schedules the request to use PD"
  },
  {
    "startMs": 737200,
    "text": "disagregation selecting the optimal"
  },
  {
    "startMs": 739360,
    "text": "prefill and decode workers. The router"
  },
  {
    "startMs": 742079,
    "text": "then coordinates the transaction"
  },
  {
    "startMs": 743519,
    "text": "directly with the designated pre-fill"
  },
  {
    "startMs": 745360,
    "text": "worker. The pre-fill worker processes"
  },
  {
    "startMs": 747360,
    "text": "the prompt, construct the initial KV"
  },
  {
    "startMs": 749200,
    "text": "cache of the prompt and outputs the"
  },
  {
    "startMs": 752000,
    "text": "standard KV transfer metadata. Um, and"
  },
  {
    "startMs": 755760,
    "text": "the target decode worker actually pulls"
  },
  {
    "startMs": 757760,
    "text": "the computed KV caches um, uh, across"
  },
  {
    "startMs": 761839,
    "text": "the network fab fabric utilizing uh, the"
  },
  {
    "startMs": 764720,
    "text": "KV transfer metadata that the uh, uh,"
  },
  {
    "startMs": 767360,
    "text": "prefill pod had generated. Um okay so"
  },
  {
    "startMs": 772399,
    "text": "with that yes that's kind of how uh PD"
  },
  {
    "startMs": 775200,
    "text": "is implemented in practice in LMD and"
  },
  {
    "startMs": 777120,
    "text": "next I would like to show you some uh"
  },
  {
    "startMs": 779279,
    "text": "experimental results on where PD"
  },
  {
    "startMs": 781680,
    "text": "actually shines. So in this graph you"
  },
  {
    "startMs": 783600,
    "text": "can see that um"
  },
  {
    "startMs": 786240,
    "text": "uh in in in the standard aggregated"
  },
  {
    "startMs": 788800,
    "text": "deployment which is the top red line uh"
  },
  {
    "startMs": 791839,
    "text": "the P99 ITL uh hovers roughly around 900"
  },
  {
    "startMs": 795920,
    "text": "milliseconds and you can you can see"
  },
  {
    "startMs": 797760,
    "text": "some fluctuations um up and down and but"
  },
  {
    "startMs": 802639,
    "text": "the the bottom blue line is the P99 uh"
  },
  {
    "startMs": 806000,
    "text": "inter token latency on a PD deployment"
  },
  {
    "startMs": 808240,
    "text": "and you can see that it's drastically"
  },
  {
    "startMs": 810399,
    "text": "almost nine times better at 100"
  },
  {
    "startMs": 812399,
    "text": "millconds and it's also much smoother uh"
  },
  {
    "startMs": 815279,
    "text": "than the aggregated serving"
  },
  {
    "startMs": 820079,
    "text": "and uh this is some of our own internal"
  },
  {
    "startMs": 823040,
    "text": "results at Red Hat. So for a GPOSS 12B"
  },
  {
    "startMs": 826160,
    "text": "model uh 16 H100s"
  },
  {
    "startMs": 829360,
    "text": "uh the aggregated config is four"
  },
  {
    "startMs": 832000,
    "text": "replicas tensor parallelism 4 and the"
  },
  {
    "startMs": 834480,
    "text": "disagregated is two prefilled 2D code"
  },
  {
    "startMs": 836560,
    "text": "all with tensor parallelism 4. It's a"
  },
  {
    "startMs": 838399,
    "text": "highly multi-turn workload with a 10,000"
  },
  {
    "startMs": 841360,
    "text": "token prefix and 128 tokens for every"
  },
  {
    "startMs": 845120,
    "text": "turn every turn. So, so this is a great"
  },
  {
    "startMs": 848240,
    "text": "chart like you can see at the bottom"
  },
  {
    "startMs": 849600,
    "text": "most line is a standard aggregated"
  },
  {
    "startMs": 851680,
    "text": "config that's uh is doing the default"
  },
  {
    "startMs": 854480,
    "text": "Kubernetes scheduling and uh and it's"
  },
  {
    "startMs": 857360,
    "text": "aggregated. So that's kind of our"
  },
  {
    "startMs": 858639,
    "text": "baseline and then the middle blue line"
  },
  {
    "startMs": 861120,
    "text": "is still aggregated but with the LLMD uh"
  },
  {
    "startMs": 865199,
    "text": "KV cache aware routing and you can"
  },
  {
    "startMs": 867120,
    "text": "almost see the gains just just based on"
  },
  {
    "startMs": 869839,
    "text": "the routing and the red line is actually"
  },
  {
    "startMs": 871680,
    "text": "the PD uh the pre-fill decode config"
  },
  {
    "startMs": 875120,
    "text": "with two pre-fill and two decode workers"
  },
  {
    "startMs": 877199,
    "text": "and you can actually see that like it's"
  },
  {
    "startMs": 879040,
    "text": "very similar to the aggregated config at"
  },
  {
    "startMs": 881519,
    "text": "the lower concurrency regimes and uh"
  },
  {
    "startMs": 884079,
    "text": "even and and very similar at the higher"
  },
  {
    "startMs": 885920,
    "text": "concurrency regimes but it's actually"
  },
  {
    "startMs": 887760,
    "text": "the middle part of the concurrency"
  },
  {
    "startMs": 889920,
    "text": "regime that PD actually shines"
  },
  {
    "startMs": 893920,
    "text": "and and these are some of the the"
  },
  {
    "startMs": 896320,
    "text": "classic parita curves that we see when"
  },
  {
    "startMs": 898399,
    "text": "you actually do PD and uh aggregated"
  },
  {
    "startMs": 901440,
    "text": "side by side. So these results are again"
  },
  {
    "startMs": 903440,
    "text": "from the GPTOSS 12B model 64 H100s"
  },
  {
    "startMs": 907279,
    "text": "aggregated is eight replicas TP8 and"
  },
  {
    "startMs": 910399,
    "text": "this a is uh three prefilled 5D code"
  },
  {
    "startMs": 913199,
    "text": "again TP8 and a pre-filled heavy"
  },
  {
    "startMs": 915440,
    "text": "workload with like 5,000 average input"
  },
  {
    "startMs": 918560,
    "text": "sequence length and 500 output sequence"
  },
  {
    "startMs": 920480,
    "text": "length and you can actually see the blue"
  },
  {
    "startMs": 922399,
    "text": "line is the the PD curve and the red"
  },
  {
    "startMs": 924880,
    "text": "line is the aggregated curve and the PD"
  },
  {
    "startMs": 926959,
    "text": "curve kind of dominates um uh the"
  },
  {
    "startMs": 930000,
    "text": "aggregate curve across the entire"
  },
  {
    "startMs": 932079,
    "text": "interactivity spectrum."
  },
  {
    "startMs": 937040,
    "text": "Okay, but I don't want to leave you guys"
  },
  {
    "startMs": 939120,
    "text": "that PD is the answer to everything and"
  },
  {
    "startMs": 940959,
    "text": "it's a magic bullet. But um it's uh it's"
  },
  {
    "startMs": 943760,
    "text": "essentially a separation phase"
  },
  {
    "startMs": 945199,
    "text": "separation trade-off and not a magic"
  },
  {
    "startMs": 946959,
    "text": "bullet. So we created this uh matrix to"
  },
  {
    "startMs": 949920,
    "text": "help you decide when PD might be uh good"
  },
  {
    "startMs": 953040,
    "text": "for you. So if you're managing long"
  },
  {
    "startMs": 955920,
    "text": "context uh with high ISL OSL ratios and"
  },
  {
    "startMs": 960160,
    "text": "you if you have a large model that"
  },
  {
    "startMs": 961839,
    "text": "you're serving that can that you can"
  },
  {
    "startMs": 964240,
    "text": "apply rich model parallelism techniques"
  },
  {
    "startMs": 967360,
    "text": "um you're facing that middle concurrency"
  },
  {
    "startMs": 969360,
    "text": "regime uh that I I showed you in the"
  },
  {
    "startMs": 971920,
    "text": "previous graphs and and the very"
  },
  {
    "startMs": 974079,
    "text": "important part is that if you want uh"
  },
  {
    "startMs": 976240,
    "text": "strict ITL streaming requirements like"
  },
  {
    "startMs": 978079,
    "text": "you want the you want the token"
  },
  {
    "startMs": 979920,
    "text": "generation to be uh much more smooth um"
  },
  {
    "startMs": 982800,
    "text": "then you want to consider PD but we also"
  },
  {
    "startMs": 985360,
    "text": "saw that it requires transfer of KV"
  },
  {
    "startMs": 987440,
    "text": "caches from your pre-filled workers to"
  },
  {
    "startMs": 989279,
    "text": "your decode workers. So you must pro"
  },
  {
    "startMs": 991519,
    "text": "process an advanced uh high-sp speeded"
  },
  {
    "startMs": 993360,
    "text": "network fabric like uh RDMMA or rocky to"
  },
  {
    "startMs": 997040,
    "text": "support that KV cache transfer. And if"
  },
  {
    "startMs": 999279,
    "text": "you do not have such requirements, short"
  },
  {
    "startMs": 1001440,
    "text": "moderate context, any model size, low"
  },
  {
    "startMs": 1006160,
    "text": "concurrency regimes or uh if you have"
  },
  {
    "startMs": 1008560,
    "text": "strict TTF requirements because you can"
  },
  {
    "startMs": 1010560,
    "text": "actually tune them on an aggregate"
  },
  {
    "startMs": 1012000,
    "text": "serving um and you the biggest point is"
  },
  {
    "startMs": 1015120,
    "text": "like if you don't have the network"
  },
  {
    "startMs": 1016240,
    "text": "fabric to support those KV cache"
  },
  {
    "startMs": 1017920,
    "text": "transfers. So you might actually just"
  },
  {
    "startMs": 1019199,
    "text": "want to stick with aggregated."
  },
  {
    "startMs": 1022480,
    "text": "So here is my key takeaway from all of"
  },
  {
    "startMs": 1024240,
    "text": "this. So architecting this complex"
  },
  {
    "startMs": 1026400,
    "text": "platform requires balancing a lot of u"
  },
  {
    "startMs": 1029679,
    "text": "knobs and a highly multi-dimensional"
  },
  {
    "startMs": 1031760,
    "text": "design space all of which is supported"
  },
  {
    "startMs": 1033679,
    "text": "in LLMD. As you saw the scheduler must"
  },
  {
    "startMs": 1036559,
    "text": "support or constantly evaluate SLO"
  },
  {
    "startMs": 1038880,
    "text": "targets uh QEPs KV cache locality"
  },
  {
    "startMs": 1041520,
    "text": "metrics PD ratios and network topologies"
  },
  {
    "startMs": 1044640,
    "text": "to be able to route the request to the"
  },
  {
    "startMs": 1046480,
    "text": "optimal FOD. While in while the PD"
  },
  {
    "startMs": 1049679,
    "text": "design space you you need dynamic PD"
  },
  {
    "startMs": 1052480,
    "text": "rate matching to adapt to PD ratios"
  },
  {
    "startMs": 1055280,
    "text": "because you know you can start with a"
  },
  {
    "startMs": 1056720,
    "text": "static PD ratio but it needs to evolve"
  },
  {
    "startMs": 1058960,
    "text": "with the autoscaler as the traffic"
  },
  {
    "startMs": 1060960,
    "text": "changes um and you need uh yeah"
  },
  {
    "startMs": 1064240,
    "text": "autoscaling to scale PD pools"
  },
  {
    "startMs": 1066320,
    "text": "independently"
  },
  {
    "startMs": 1067840,
    "text": "um and constantly tweaking model"
  },
  {
    "startMs": 1070160,
    "text": "parallelism techniques like tensor"
  },
  {
    "startMs": 1071840,
    "text": "parallelism data parallelism uh to meet"
  },
  {
    "startMs": 1074080,
    "text": "your SLOs's."
  },
  {
    "startMs": 1076080,
    "text": "So um I think with these uh I will hand"
  },
  {
    "startMs": 1079520,
    "text": "it over to Euchen to anchor some of the"
  },
  {
    "startMs": 1081679,
    "text": "concepts that we showed with the real"
  },
  {
    "startMs": 1084320,
    "text": "world case study of serving the GLM 5.2"
  },
  {
    "startMs": 1087120,
    "text": "model uh which is uh still ongoing as we"
  },
  {
    "startMs": 1090640,
    "text": "speak."
  },
  {
    "startMs": 1091520,
    "text": ">> Yeah, still ongoing. You probably have"
  },
  {
    "startMs": 1093280,
    "text": "seen tons of uh impressive numbers of"
  },
  {
    "startMs": 1096080,
    "text": "GLM 5.2 on B200 when we talk to our"
  },
  {
    "startMs": 1099520,
    "text": "customers and they usually don't have"
  },
  {
    "startMs": 1101520,
    "text": "you know the luxury of B200. They have a"
  },
  {
    "startMs": 1103280,
    "text": "lot of H200. So we have to figure out"
  },
  {
    "startMs": 1105440,
    "text": "how to like put all the knobs together"
  },
  {
    "startMs": 1107280,
    "text": "and make GM 5.2 work really well for"
  },
  {
    "startMs": 1110480,
    "text": "cluster of of H200. So uh we let's"
  },
  {
    "startMs": 1113600,
    "text": "anchor all the concept together. Um we"
  },
  {
    "startMs": 1115600,
    "text": "went through for example the uh KV cache"
  },
  {
    "startMs": 1118160,
    "text": "routing PD disagregation. We kind of"
  },
  {
    "startMs": 1120960,
    "text": "call them a wildl path in LMD and also"
  },
  {
    "startMs": 1123840,
    "text": "we combine with different parallelism"
  },
  {
    "startMs": 1125520,
    "text": "strategies to so we can uh independently"
  },
  {
    "startMs": 1128160,
    "text": "uh scale prefuel paths because for"
  },
  {
    "startMs": 1130080,
    "text": "agentic workload is super uh long you"
  },
  {
    "startMs": 1132480,
    "text": "know like heavy prefill. So uh in this"
  },
  {
    "startMs": 1135120,
    "text": "case we designed the prefuel pool using"
  },
  {
    "startMs": 1137360,
    "text": "up to three workers optimized for uh"
  },
  {
    "startMs": 1139919,
    "text": "high throughput uh with deep and then"
  },
  {
    "startMs": 1142400,
    "text": "for decoup we use uh one dedicated"
  },
  {
    "startMs": 1144400,
    "text": "worker and um that's optimized for for"
  },
  {
    "startMs": 1146720,
    "text": "low latency. So we use Nixo for"
  },
  {
    "startMs": 1148799,
    "text": "efficient KV transfer between the pools"
  },
  {
    "startMs": 1151440,
    "text": "and also with the each worker we have"
  },
  {
    "startMs": 1153520,
    "text": "the leader worker set group uh with TP1"
  },
  {
    "startMs": 1155840,
    "text": "DP8 and also uh EP8 uh expert"
  },
  {
    "startMs": 1158960,
    "text": "parallelism 8. So the architecture is"
  },
  {
    "startMs": 1161120,
    "text": "just highly modular because you can uh"
  },
  {
    "startMs": 1163120,
    "text": "actually scale the throughput by simply"
  },
  {
    "startMs": 1164960,
    "text": "adding uh preview workers without"
  },
  {
    "startMs": 1166960,
    "text": "reconfiguring and um the decoup. So uh"
  },
  {
    "startMs": 1170559,
    "text": "this highlights how AMD effectly"
  },
  {
    "startMs": 1172720,
    "text": "effectively managed the complexity of"
  },
  {
    "startMs": 1174640,
    "text": "combining like PB and DB and EPI scale."
  },
  {
    "startMs": 1178080,
    "text": "And also we found some interesting fun"
  },
  {
    "startMs": 1179840,
    "text": "fact actually a couple days ago. Um B B"
  },
  {
    "startMs": 1182559,
    "text": "B B B B B B B B B B B B B B B B B B B B"
  },
  {
    "startMs": 1182640,
    "text": "B B B B B B B B B B B B B B B B B B Bf6"
  },
  {
    "startMs": 1183280,
    "text": "uh BF16 KV cache actually is faster than"
  },
  {
    "startMs": 1186160,
    "text": "using like FPA uh KV cache for longer"
  },
  {
    "startMs": 1189200,
    "text": "preview. Um this is also like we"
  },
  {
    "startMs": 1191679,
    "text": "continue to explore and found like more"
  },
  {
    "startMs": 1193760,
    "text": "interesting patterns, but more"
  },
  {
    "startMs": 1195440,
    "text": "importantly uh we want to kind of just"
  },
  {
    "startMs": 1197360,
    "text": "show the result really quick. So um for"
  },
  {
    "startMs": 1199520,
    "text": "this uh data set agentic workload data"
  },
  {
    "startMs": 1201520,
    "text": "set the ISO OSL ratio is pretty high 45"
  },
  {
    "startMs": 1204000,
    "text": "to1 ratio preview is uh is really the"
  },
  {
    "startMs": 1207039,
    "text": "constraint you can tell um with 2P even"
  },
  {
    "startMs": 1209440,
    "text": "1D we have um 4x passer TDFT and also 60"
  },
  {
    "startMs": 1214559,
    "text": "uh% more requests and this is continuous"
  },
  {
    "startMs": 1217600,
    "text": "like work in progress so the next step"
  },
  {
    "startMs": 1219280,
    "text": "is we need to also put the upper layer"
  },
  {
    "startMs": 1222160,
    "text": "lower TTFT and also adding more more"
  },
  {
    "startMs": 1224400,
    "text": "preview replicas so um I know we're"
  },
  {
    "startMs": 1227600,
    "text": "running out of time really quick. Uh we"
  },
  {
    "startMs": 1230080,
    "text": "um the fundamental shift for agentic"
  },
  {
    "startMs": 1231919,
    "text": "workload we're continuing to uh have"
  },
  {
    "startMs": 1234480,
    "text": "this um uh agentic north uh northstar uh"
  },
  {
    "startMs": 1237760,
    "text": "with session graph orchestration program"
  },
  {
    "startMs": 1239760,
    "text": "award scheduling uh state reuse life"
  },
  {
    "startMs": 1242080,
    "text": "cycle and also the uh agentic benchmark"
  },
  {
    "startMs": 1245520,
    "text": "um we're working on. So you can find"
  },
  {
    "startMs": 1247600,
    "text": "them uh in AMD upstream AMD and also you"
  },
  {
    "startMs": 1251360,
    "text": "know feel free to join the SIG group and"
  },
  {
    "startMs": 1254000,
    "text": "uh and contribute and um this is the"
  },
  {
    "startMs": 1257200,
    "text": "very last slide. So distri distributed"
  },
  {
    "startMs": 1259120,
    "text": "inference is not challenge uh every"
  },
  {
    "startMs": 1260880,
    "text": "single comp a single company can solve"
  },
  {
    "startMs": 1262720,
    "text": "along. We're proud to be uh building"
  },
  {
    "startMs": 1265360,
    "text": "this uh future in the open alongside our"
  },
  {
    "startMs": 1267840,
    "text": "incredible ecosystem collaborators uh"
  },
  {
    "startMs": 1270400,
    "text": "core wave Google IBM Nvidia growing list"
  },
  {
    "startMs": 1273280,
    "text": "of launch partners and industry"
  },
  {
    "startMs": 1275120,
    "text": "adopters. So if you're passionate about"
  },
  {
    "startMs": 1277120,
    "text": "the future of opensource inference, we"
  },
  {
    "startMs": 1279520,
    "text": "invite you to join us. We do have a"
  },
  {
    "startMs": 1281200,
    "text": "booth downstairs. Feel free to stop by,"
  },
  {
    "startMs": 1283120,
    "text": "ask us any questions. And uh thank you"
  },
  {
    "startMs": 1285280,
    "text": "so much for your time."
  },
  {
    "startMs": 1287744,
    "text": "[applause]"
  },
  {
    "startMs": 1304774,
    "text": ">> [music]"
  }
]