[
  {
    "startMs": 5839,
    "text": "Good morning everyone. My name is"
  },
  {
    "startMs": 7510,
    "text": "Good morning everyone. My name is"
  },
  {
    "startMs": 7520,
    "text": "Good morning everyone. My name is Hershel Jan and I have here with me"
  },
  {
    "startMs": 10230,
    "text": "Hershel Jan and I have here with me"
  },
  {
    "startMs": 10240,
    "text": "Hershel Jan and I have here with me Pearl. Today we are going to talk about"
  },
  {
    "startMs": 13749,
    "text": "Pearl. Today we are going to talk about"
  },
  {
    "startMs": 13759,
    "text": "Pearl. Today we are going to talk about saying no. It's called as admission"
  },
  {
    "startMs": 16790,
    "text": "saying no. It's called as admission"
  },
  {
    "startMs": 16800,
    "text": "saying no. It's called as admission control. How our system decides which"
  },
  {
    "startMs": 20550,
    "text": "control. How our system decides which"
  },
  {
    "startMs": 20560,
    "text": "control. How our system decides which request to get in when your coding"
  },
  {
    "startMs": 22390,
    "text": "request to get in when your coding"
  },
  {
    "startMs": 22400,
    "text": "request to get in when your coding agents overwhelm the LLMs you host"
  },
  {
    "startMs": 25750,
    "text": "agents overwhelm the LLMs you host"
  },
  {
    "startMs": 25760,
    "text": "agents overwhelm the LLMs you host yourself."
  },
  {
    "startMs": 27269,
    "text": "yourself."
  },
  {
    "startMs": 27279,
    "text": "yourself. Uh before we move on a quick intro about"
  },
  {
    "startMs": 30470,
    "text": "Uh before we move on a quick intro about"
  },
  {
    "startMs": 30480,
    "text": "Uh before we move on a quick intro about us."
  },
  {
    "startMs": 31990,
    "text": "us."
  },
  {
    "startMs": 32000,
    "text": "us. So I am a senior software engineer at"
  },
  {
    "startMs": 34389,
    "text": "So I am a senior software engineer at"
  },
  {
    "startMs": 34399,
    "text": "So I am a senior software engineer at Audible where I lead some of the MLAI"
  },
  {
    "startMs": 37190,
    "text": "Audible where I lead some of the MLAI"
  },
  {
    "startMs": 37200,
    "text": "Audible where I lead some of the MLAI infrastructure."
  },
  {
    "startMs": 38790,
    "text": "infrastructure."
  },
  {
    "startMs": 38800,
    "text": "infrastructure. Uh I am also writing an open-source"
  },
  {
    "startMs": 40869,
    "text": "Uh I am also writing an open-source"
  },
  {
    "startMs": 40879,
    "text": "Uh I am also writing an open-source handbook called as LLM inference at"
  },
  {
    "startMs": 43350,
    "text": "handbook called as LLM inference at"
  },
  {
    "startMs": 43360,
    "text": "handbook called as LLM inference at scale. It has received over 300 plus"
  },
  {
    "startMs": 45590,
    "text": "scale. It has received over 300 plus"
  },
  {
    "startMs": 45600,
    "text": "scale. It has received over 300 plus stars by now."
  },
  {
    "startMs": 47510,
    "text": "stars by now."
  },
  {
    "startMs": 47520,
    "text": "stars by now. I am also maintaining an open-source"
  },
  {
    "startMs": 49830,
    "text": "I am also maintaining an open-source"
  },
  {
    "startMs": 49840,
    "text": "I am also maintaining an open-source framework called agentship which helps"
  },
  {
    "startMs": 51830,
    "text": "framework called agentship which helps"
  },
  {
    "startMs": 51840,
    "text": "framework called agentship which helps you ship reliable and secure agentic"
  },
  {
    "startMs": 55110,
    "text": "you ship reliable and secure agentic"
  },
  {
    "startMs": 55120,
    "text": "you ship reliable and secure agentic harnesses in an ultra fast speed. Uh"
  },
  {
    "startMs": 59029,
    "text": "harnesses in an ultra fast speed. Uh"
  },
  {
    "startMs": 59039,
    "text": "harnesses in an ultra fast speed. Uh perv would you like to introduce"
  },
  {
    "startMs": 60790,
    "text": "perv would you like to introduce"
  },
  {
    "startMs": 60800,
    "text": "perv would you like to introduce yourself?"
  },
  {
    "startMs": 61830,
    "text": "yourself?"
  },
  {
    "startMs": 61840,
    "text": "yourself? Sure. Hi everyone. I'm Perv. I'm a"
  },
  {
    "startMs": 63990,
    "text": "Sure. Hi everyone. I'm Perv. I'm a"
  },
  {
    "startMs": 64000,
    "text": "Sure. Hi everyone. I'm Perv. I'm a senior software developer at SAP. I work"
  },
  {
    "startMs": 66390,
    "text": "senior software developer at SAP. I work"
  },
  {
    "startMs": 66400,
    "text": "senior software developer at SAP. I work on like cloud application and LLM"
  },
  {
    "startMs": 69510,
    "text": "on like cloud application and LLM"
  },
  {
    "startMs": 69520,
    "text": "on like cloud application and LLM infrastructure. I wrote an LLM fine-tune"
  },
  {
    "startMs": 72630,
    "text": "infrastructure. I wrote an LLM fine-tune"
  },
  {
    "startMs": 72640,
    "text": "infrastructure. I wrote an LLM fine-tune guide. I contribute to agentship and I'm"
  },
  {
    "startMs": 75350,
    "text": "guide. I contribute to agentship and I'm"
  },
  {
    "startMs": 75360,
    "text": "guide. I contribute to agentship and I'm like building a task tracker which is a"
  },
  {
    "startMs": 78070,
    "text": "like building a task tracker which is a"
  },
  {
    "startMs": 78080,
    "text": "like building a task tracker which is a local post task board for you and your"
  },
  {
    "startMs": 80950,
    "text": "local post task board for you and your"
  },
  {
    "startMs": 80960,
    "text": "local post task board for you and your agent."
  },
  {
    "startMs": 82630,
    "text": "agent."
  },
  {
    "startMs": 82640,
    "text": "agent. Uh apart from this if you would like to"
  },
  {
    "startMs": 84789,
    "text": "Uh apart from this if you would like to"
  },
  {
    "startMs": 84799,
    "text": "Uh apart from this if you would like to connect with us our LinkedIn QR codes"
  },
  {
    "startMs": 87190,
    "text": "connect with us our LinkedIn QR codes"
  },
  {
    "startMs": 87200,
    "text": "connect with us our LinkedIn QR codes are right there on this slide. Now back"
  },
  {
    "startMs": 89749,
    "text": "are right there on this slide. Now back"
  },
  {
    "startMs": 89759,
    "text": "are right there on this slide. Now back to you Hershel."
  },
  {
    "startMs": 92230,
    "text": "to you Hershel."
  },
  {
    "startMs": 92240,
    "text": "to you Hershel. Great. So"
  },
  {
    "startMs": 95510,
    "text": "Great. So"
  },
  {
    "startMs": 95520,
    "text": "Great. So here is the plan. We are going to start"
  },
  {
    "startMs": 97830,
    "text": "here is the plan. We are going to start"
  },
  {
    "startMs": 97840,
    "text": "here is the plan. We are going to start with understanding the motivation why"
  },
  {
    "startMs": 101830,
    "text": "with understanding the motivation why"
  },
  {
    "startMs": 101840,
    "text": "with understanding the motivation why any serving stack can admit so much of a"
  },
  {
    "startMs": 105030,
    "text": "any serving stack can admit so much of a"
  },
  {
    "startMs": 105040,
    "text": "any serving stack can admit so much of a traffic."
  },
  {
    "startMs": 106550,
    "text": "traffic."
  },
  {
    "startMs": 106560,
    "text": "traffic. Second, we try to dive into admission"
  },
  {
    "startMs": 108789,
    "text": "Second, we try to dive into admission"
  },
  {
    "startMs": 108799,
    "text": "Second, we try to dive into admission control where the decision to say no"
  },
  {
    "startMs": 111670,
    "text": "control where the decision to say no"
  },
  {
    "startMs": 111680,
    "text": "control where the decision to say no lives and what it needs to see. Third,"
  },
  {
    "startMs": 115109,
    "text": "lives and what it needs to see. Third,"
  },
  {
    "startMs": 115119,
    "text": "lives and what it needs to see. Third, we will go through some of the proven"
  },
  {
    "startMs": 117270,
    "text": "we will go through some of the proven"
  },
  {
    "startMs": 117280,
    "text": "we will go through some of the proven strategies, the different ways a system"
  },
  {
    "startMs": 120069,
    "text": "strategies, the different ways a system"
  },
  {
    "startMs": 120079,
    "text": "strategies, the different ways a system can actually make the admission"
  },
  {
    "startMs": 121749,
    "text": "can actually make the admission"
  },
  {
    "startMs": 121759,
    "text": "can actually make the admission decision. And fourth, we will try to"
  },
  {
    "startMs": 125270,
    "text": "decision. And fourth, we will try to"
  },
  {
    "startMs": 125280,
    "text": "decision. And fourth, we will try to dive into the benchmarking where you"
  },
  {
    "startMs": 127749,
    "text": "dive into the benchmarking where you"
  },
  {
    "startMs": 127759,
    "text": "dive into the benchmarking where you will learn how to prove that your policy"
  },
  {
    "startMs": 131029,
    "text": "will learn how to prove that your policy"
  },
  {
    "startMs": 131039,
    "text": "will learn how to prove that your policy for admission is working or it's not"
  },
  {
    "startMs": 133670,
    "text": "for admission is working or it's not"
  },
  {
    "startMs": 133680,
    "text": "for admission is working or it's not working."
  },
  {
    "startMs": 136390,
    "text": "working."
  },
  {
    "startMs": 136400,
    "text": "working. So let's get started with the"
  },
  {
    "startMs": 138229,
    "text": "So let's get started with the"
  },
  {
    "startMs": 138239,
    "text": "So let's get started with the motivation. Um"
  },
  {
    "startMs": 141750,
    "text": "motivation. Um"
  },
  {
    "startMs": 141760,
    "text": "motivation. Um and for this"
  },
  {
    "startMs": 146640,
    "text": "everything today uh let's try to"
  },
  {
    "startMs": 148550,
    "text": "everything today uh let's try to"
  },
  {
    "startMs": 148560,
    "text": "everything today uh let's try to understand what is inference. Everything"
  },
  {
    "startMs": 151030,
    "text": "understand what is inference. Everything"
  },
  {
    "startMs": 151040,
    "text": "understand what is inference. Everything today happens during this inference"
  },
  {
    "startMs": 154710,
    "text": "today happens during this inference"
  },
  {
    "startMs": 154720,
    "text": "today happens during this inference itself. So inference is simply a moment"
  },
  {
    "startMs": 157190,
    "text": "itself. So inference is simply a moment"
  },
  {
    "startMs": 157200,
    "text": "itself. So inference is simply a moment when someone sends a prompt to the model"
  },
  {
    "startMs": 160150,
    "text": "when someone sends a prompt to the model"
  },
  {
    "startMs": 160160,
    "text": "when someone sends a prompt to the model and gets back an answer."
  },
  {
    "startMs": 163270,
    "text": "and gets back an answer."
  },
  {
    "startMs": 163280,
    "text": "and gets back an answer. So the step one is like a user prompt"
  },
  {
    "startMs": 165830,
    "text": "So the step one is like a user prompt"
  },
  {
    "startMs": 165840,
    "text": "So the step one is like a user prompt comes in. It's a sequence of words. Uh"
  },
  {
    "startMs": 168790,
    "text": "comes in. It's a sequence of words. Uh"
  },
  {
    "startMs": 168800,
    "text": "comes in. It's a sequence of words. Uh for example, it could be like failing a"
  },
  {
    "startMs": 171589,
    "text": "for example, it could be like failing a"
  },
  {
    "startMs": 171599,
    "text": "for example, it could be like failing a fix a failing test. The model which is"
  },
  {
    "startMs": 175030,
    "text": "fix a failing test. The model which is"
  },
  {
    "startMs": 175040,
    "text": "fix a failing test. The model which is running on your GPUs writes the answer"
  },
  {
    "startMs": 177830,
    "text": "running on your GPUs writes the answer"
  },
  {
    "startMs": 177840,
    "text": "running on your GPUs writes the answer one token at a time. A token is roughly"
  },
  {
    "startMs": 180630,
    "text": "one token at a time. A token is roughly"
  },
  {
    "startMs": 180640,
    "text": "one token at a time. A token is roughly like a word as I mentioned. Uh it writes"
  },
  {
    "startMs": 183589,
    "text": "like a word as I mentioned. Uh it writes"
  },
  {
    "startMs": 183599,
    "text": "like a word as I mentioned. Uh it writes the first word then uses that word to"
  },
  {
    "startMs": 186710,
    "text": "the first word then uses that word to"
  },
  {
    "startMs": 186720,
    "text": "the first word then uses that word to pick the second uh then it uses both to"
  },
  {
    "startMs": 190550,
    "text": "pick the second uh then it uses both to"
  },
  {
    "startMs": 190560,
    "text": "pick the second uh then it uses both to pick the third and it keeps on"
  },
  {
    "startMs": 192309,
    "text": "pick the third and it keeps on"
  },
  {
    "startMs": 192319,
    "text": "pick the third and it keeps on continuing like that and it cannot write"
  },
  {
    "startMs": 196070,
    "text": "continuing like that and it cannot write"
  },
  {
    "startMs": 196080,
    "text": "continuing like that and it cannot write all of them at once because every word"
  },
  {
    "startMs": 198390,
    "text": "all of them at once because every word"
  },
  {
    "startMs": 198400,
    "text": "all of them at once because every word depends on the previous one. So those"
  },
  {
    "startMs": 202229,
    "text": "depends on the previous one. So those"
  },
  {
    "startMs": 202239,
    "text": "depends on the previous one. So those once all the words are generated they"
  },
  {
    "startMs": 204470,
    "text": "once all the words are generated they"
  },
  {
    "startMs": 204480,
    "text": "once all the words are generated they stream back to the user uh who ask for"
  },
  {
    "startMs": 208470,
    "text": "stream back to the user uh who ask for"
  },
  {
    "startMs": 208480,
    "text": "stream back to the user uh who ask for the result and that volt prompt and the"
  },
  {
    "startMs": 211990,
    "text": "the result and that volt prompt and the"
  },
  {
    "startMs": 212000,
    "text": "the result and that volt prompt and the answer out is like that one request. Now"
  },
  {
    "startMs": 214949,
    "text": "answer out is like that one request. Now"
  },
  {
    "startMs": 214959,
    "text": "answer out is like that one request. Now I want you to hold on to one small"
  },
  {
    "startMs": 217670,
    "text": "I want you to hold on to one small"
  },
  {
    "startMs": 217680,
    "text": "I want you to hold on to one small detail like model has no idea how long"
  },
  {
    "startMs": 221430,
    "text": "detail like model has no idea how long"
  },
  {
    "startMs": 221440,
    "text": "detail like model has no idea how long this answer is going to be. It only"
  },
  {
    "startMs": 224070,
    "text": "this answer is going to be. It only"
  },
  {
    "startMs": 224080,
    "text": "this answer is going to be. It only finds out whether it writes when it only"
  },
  {
    "startMs": 229110,
    "text": "finds out whether it writes when it only"
  },
  {
    "startMs": 229120,
    "text": "finds out whether it writes when it only finds out when it writes the first last"
  },
  {
    "startMs": 231350,
    "text": "finds out when it writes the first last"
  },
  {
    "startMs": 231360,
    "text": "finds out when it writes the first last token."
  },
  {
    "startMs": 233270,
    "text": "token."
  },
  {
    "startMs": 233280,
    "text": "token. Keep that in mind. It will matter in the"
  },
  {
    "startMs": 235670,
    "text": "Keep that in mind. It will matter in the"
  },
  {
    "startMs": 235680,
    "text": "Keep that in mind. It will matter in the few minutes."
  },
  {
    "startMs": 241920,
    "text": "So a model never sits on the internet"
  },
  {
    "startMs": 245350,
    "text": "So a model never sits on the internet"
  },
  {
    "startMs": 245360,
    "text": "So a model never sits on the internet itself. Every request travels through a"
  },
  {
    "startMs": 248710,
    "text": "itself. Every request travels through a"
  },
  {
    "startMs": 248720,
    "text": "itself. Every request travels through a short pipeline. It starts in your app or"
  },
  {
    "startMs": 252869,
    "text": "short pipeline. It starts in your app or"
  },
  {
    "startMs": 252879,
    "text": "short pipeline. It starts in your app or your agent. Then it reaches your API"
  },
  {
    "startMs": 255110,
    "text": "your agent. Then it reaches your API"
  },
  {
    "startMs": 255120,
    "text": "your agent. Then it reaches your API gateway"
  },
  {
    "startMs": 256629,
    "text": "gateway"
  },
  {
    "startMs": 256639,
    "text": "gateway which checks your API key counts how"
  },
  {
    "startMs": 258789,
    "text": "which checks your API key counts how"
  },
  {
    "startMs": 258799,
    "text": "which checks your API key counts how many requests you are sending. Then a"
  },
  {
    "startMs": 261110,
    "text": "many requests you are sending. Then a"
  },
  {
    "startMs": 261120,
    "text": "many requests you are sending. Then a router picks a replica which which is"
  },
  {
    "startMs": 263670,
    "text": "router picks a replica which which is"
  },
  {
    "startMs": 263680,
    "text": "router picks a replica which which is just one running copy of the model and"
  },
  {
    "startMs": 266310,
    "text": "just one running copy of the model and"
  },
  {
    "startMs": 266320,
    "text": "just one running copy of the model and finally the requests land on the"
  },
  {
    "startMs": 268469,
    "text": "finally the requests land on the"
  },
  {
    "startMs": 268479,
    "text": "finally the requests land on the inference engine. The common some of the"
  },
  {
    "startMs": 271350,
    "text": "inference engine. The common some of the"
  },
  {
    "startMs": 271360,
    "text": "inference engine. The common some of the common inference engines are the VLMs,"
  },
  {
    "startMs": 273990,
    "text": "common inference engines are the VLMs,"
  },
  {
    "startMs": 274000,
    "text": "common inference engines are the VLMs, SGLAN, Tensor RT etc. engine is what"
  },
  {
    "startMs": 279270,
    "text": "SGLAN, Tensor RT etc. engine is what"
  },
  {
    "startMs": 279280,
    "text": "SGLAN, Tensor RT etc. engine is what actually hosts your model."
  },
  {
    "startMs": 282390,
    "text": "actually hosts your model."
  },
  {
    "startMs": 282400,
    "text": "actually hosts your model. It actually serves your model uh and it"
  },
  {
    "startMs": 285189,
    "text": "It actually serves your model uh and it"
  },
  {
    "startMs": 285199,
    "text": "It actually serves your model uh and it loads the model weights into your GPUs."
  },
  {
    "startMs": 288070,
    "text": "loads the model weights into your GPUs."
  },
  {
    "startMs": 288080,
    "text": "loads the model weights into your GPUs. It groups many requests into one batch"
  },
  {
    "startMs": 290310,
    "text": "It groups many requests into one batch"
  },
  {
    "startMs": 290320,
    "text": "It groups many requests into one batch so the GPU stays busy and for every"
  },
  {
    "startMs": 293590,
    "text": "so the GPU stays busy and for every"
  },
  {
    "startMs": 293600,
    "text": "so the GPU stays busy and for every request it keeps something called as a"
  },
  {
    "startMs": 295990,
    "text": "request it keeps something called as a"
  },
  {
    "startMs": 296000,
    "text": "request it keeps something called as a KV cache. In plain words, the KV cache"
  },
  {
    "startMs": 298790,
    "text": "KV cache. In plain words, the KV cache"
  },
  {
    "startMs": 298800,
    "text": "KV cache. In plain words, the KV cache is the memory the request builds up when"
  },
  {
    "startMs": 302230,
    "text": "is the memory the request builds up when"
  },
  {
    "startMs": 302240,
    "text": "is the memory the request builds up when it try when its answer is being written."
  },
  {
    "startMs": 305030,
    "text": "it try when its answer is being written."
  },
  {
    "startMs": 305040,
    "text": "it try when its answer is being written. Longer the answer, bigger this memory"
  },
  {
    "startMs": 307110,
    "text": "Longer the answer, bigger this memory"
  },
  {
    "startMs": 307120,
    "text": "Longer the answer, bigger this memory gets."
  },
  {
    "startMs": 308710,
    "text": "gets."
  },
  {
    "startMs": 308720,
    "text": "gets. Keep this picture in your head. Every"
  },
  {
    "startMs": 310629,
    "text": "Keep this picture in your head. Every"
  },
  {
    "startMs": 310639,
    "text": "Keep this picture in your head. Every box here makes some decision but one"
  },
  {
    "startMs": 313749,
    "text": "box here makes some decision but one"
  },
  {
    "startMs": 313759,
    "text": "box here makes some decision but one decision is missing and we will find it"
  },
  {
    "startMs": 316230,
    "text": "decision is missing and we will find it"
  },
  {
    "startMs": 316240,
    "text": "decision is missing and we will find it out in the few minutes."
  },
  {
    "startMs": 319590,
    "text": "out in the few minutes."
  },
  {
    "startMs": 319600,
    "text": "out in the few minutes. First, how much an engine can actually"
  },
  {
    "startMs": 322550,
    "text": "First, how much an engine can actually"
  },
  {
    "startMs": 322560,
    "text": "First, how much an engine can actually hold."
  },
  {
    "startMs": 326320,
    "text": "Every deployment has a ceiling and it"
  },
  {
    "startMs": 328790,
    "text": "Every deployment has a ceiling and it"
  },
  {
    "startMs": 328800,
    "text": "Every deployment has a ceiling and it comes from the three targets that pull"
  },
  {
    "startMs": 330629,
    "text": "comes from the three targets that pull"
  },
  {
    "startMs": 330639,
    "text": "comes from the three targets that pull against each other. Quality is how big"
  },
  {
    "startMs": 333430,
    "text": "against each other. Quality is how big"
  },
  {
    "startMs": 333440,
    "text": "against each other. Quality is how big your and precise your model is. Latency"
  },
  {
    "startMs": 337350,
    "text": "your and precise your model is. Latency"
  },
  {
    "startMs": 337360,
    "text": "your and precise your model is. Latency is how fast the first token arrives and"
  },
  {
    "startMs": 340310,
    "text": "is how fast the first token arrives and"
  },
  {
    "startMs": 340320,
    "text": "is how fast the first token arrives and every token after it. Throughput is how"
  },
  {
    "startMs": 343029,
    "text": "every token after it. Throughput is how"
  },
  {
    "startMs": 343039,
    "text": "every token after it. Throughput is how many requests your GPUs can serve at a"
  },
  {
    "startMs": 345590,
    "text": "many requests your GPUs can serve at a"
  },
  {
    "startMs": 345600,
    "text": "many requests your GPUs can serve at a same time. You know the old saying good,"
  },
  {
    "startMs": 349270,
    "text": "same time. You know the old saying good,"
  },
  {
    "startMs": 349280,
    "text": "same time. You know the old saying good, fast, cheap, but you have to pick two."
  },
  {
    "startMs": 351749,
    "text": "fast, cheap, but you have to pick two."
  },
  {
    "startMs": 351759,
    "text": "fast, cheap, but you have to pick two. It's the same here. You cannot pick all"
  },
  {
    "startMs": 353830,
    "text": "It's the same here. You cannot pick all"
  },
  {
    "startMs": 353840,
    "text": "It's the same here. You cannot pick all the three. You cannot prioritize all the"
  },
  {
    "startMs": 355590,
    "text": "the three. You cannot prioritize all the"
  },
  {
    "startMs": 355600,
    "text": "the three. You cannot prioritize all the three. So you can't So every product has"
  },
  {
    "startMs": 359430,
    "text": "three. So you can't So every product has"
  },
  {
    "startMs": 359440,
    "text": "three. So you can't So every product has to pick two. For an example like an"
  },
  {
    "startMs": 361909,
    "text": "to pick two. For an example like an"
  },
  {
    "startMs": 361919,
    "text": "to pick two. For an example like an interactive chat, you would want to go"
  },
  {
    "startMs": 364870,
    "text": "interactive chat, you would want to go"
  },
  {
    "startMs": 364880,
    "text": "interactive chat, you would want to go with the quality and the latency because"
  },
  {
    "startMs": 366950,
    "text": "with the quality and the latency because"
  },
  {
    "startMs": 366960,
    "text": "with the quality and the latency because users are more sensitive to that. It's"
  },
  {
    "startMs": 369110,
    "text": "users are more sensitive to that. It's"
  },
  {
    "startMs": 369120,
    "text": "users are more sensitive to that. It's quite obvious to the users if the"
  },
  {
    "startMs": 371110,
    "text": "quite obvious to the users if the"
  },
  {
    "startMs": 371120,
    "text": "quite obvious to the users if the quality is down or the latency is high."
  },
  {
    "startMs": 373990,
    "text": "quality is down or the latency is high."
  },
  {
    "startMs": 374000,
    "text": "quality is down or the latency is high. For something like an async agents, you"
  },
  {
    "startMs": 376150,
    "text": "For something like an async agents, you"
  },
  {
    "startMs": 376160,
    "text": "For something like an async agents, you would want to go with a higher"
  },
  {
    "startMs": 377590,
    "text": "would want to go with a higher"
  },
  {
    "startMs": 377600,
    "text": "would want to go with a higher throughput as well as quality. If"
  },
  {
    "startMs": 380550,
    "text": "throughput as well as quality. If"
  },
  {
    "startMs": 380560,
    "text": "throughput as well as quality. If quality is less, you will have to redo"
  },
  {
    "startMs": 383110,
    "text": "quality is less, you will have to redo"
  },
  {
    "startMs": 383120,
    "text": "quality is less, you will have to redo the task and redoing is very very"
  },
  {
    "startMs": 385270,
    "text": "the task and redoing is very very"
  },
  {
    "startMs": 385280,
    "text": "the task and redoing is very very expensive because these are the"
  },
  {
    "startMs": 387430,
    "text": "expensive because these are the"
  },
  {
    "startMs": 387440,
    "text": "expensive because these are the background and the asynchronous tasks."
  },
  {
    "startMs": 390390,
    "text": "background and the asynchronous tasks."
  },
  {
    "startMs": 390400,
    "text": "background and the asynchronous tasks. And the high volume real-time systems"
  },
  {
    "startMs": 392390,
    "text": "And the high volume real-time systems"
  },
  {
    "startMs": 392400,
    "text": "And the high volume real-time systems such as the autocompletes they would try"
  },
  {
    "startMs": 395510,
    "text": "such as the autocompletes they would try"
  },
  {
    "startMs": 395520,
    "text": "such as the autocompletes they would try to favor latency as well as and the"
  },
  {
    "startMs": 397670,
    "text": "to favor latency as well as and the"
  },
  {
    "startMs": 397680,
    "text": "to favor latency as well as and the throughput in favor of the quality"
  },
  {
    "startMs": 399670,
    "text": "throughput in favor of the quality"
  },
  {
    "startMs": 399680,
    "text": "throughput in favor of the quality because you don't need an expensive"
  },
  {
    "startMs": 401270,
    "text": "because you don't need an expensive"
  },
  {
    "startMs": 401280,
    "text": "because you don't need an expensive model for doing the autocompletes for"
  },
  {
    "startMs": 403270,
    "text": "model for doing the autocompletes for"
  },
  {
    "startMs": 403280,
    "text": "model for doing the autocompletes for you. But whichever corner you give up"
  },
  {
    "startMs": 406710,
    "text": "you. But whichever corner you give up"
  },
  {
    "startMs": 406720,
    "text": "you. But whichever corner you give up there is still a maximum load you can"
  },
  {
    "startMs": 409430,
    "text": "there is still a maximum load you can"
  },
  {
    "startMs": 409440,
    "text": "there is still a maximum load you can carry while hitting your SLO"
  },
  {
    "startMs": 412309,
    "text": "carry while hitting your SLO"
  },
  {
    "startMs": 412319,
    "text": "carry while hitting your SLO your service level objective. Let's say"
  },
  {
    "startMs": 414710,
    "text": "your service level objective. Let's say"
  },
  {
    "startMs": 414720,
    "text": "your service level objective. Let's say that on this deployment it's eight"
  },
  {
    "startMs": 417830,
    "text": "that on this deployment it's eight"
  },
  {
    "startMs": 417840,
    "text": "that on this deployment it's eight requests at once. In a normal minute,"
  },
  {
    "startMs": 420710,
    "text": "requests at once. In a normal minute,"
  },
  {
    "startMs": 420720,
    "text": "requests at once. In a normal minute, you never notice that limit. Now,"
  },
  {
    "startMs": 423029,
    "text": "you never notice that limit. Now,"
  },
  {
    "startMs": 423039,
    "text": "you never notice that limit. Now, picture the 40 requests arriving in the"
  },
  {
    "startMs": 425189,
    "text": "picture the 40 requests arriving in the"
  },
  {
    "startMs": 425199,
    "text": "picture the 40 requests arriving in the same two seconds."
  },
  {
    "startMs": 428960,
    "text": "It's such a high load."
  },
  {
    "startMs": 435440,
    "text": "And this is not like hypothetical. It"
  },
  {
    "startMs": 437589,
    "text": "And this is not like hypothetical. It"
  },
  {
    "startMs": 437599,
    "text": "And this is not like hypothetical. It has happened to like biggest providers"
  },
  {
    "startMs": 439510,
    "text": "has happened to like biggest providers"
  },
  {
    "startMs": 439520,
    "text": "has happened to like biggest providers in the world. Here are the two public"
  },
  {
    "startMs": 441589,
    "text": "in the world. Here are the two public"
  },
  {
    "startMs": 441599,
    "text": "in the world. Here are the two public postmortms two years apart."
  },
  {
    "startMs": 444629,
    "text": "postmortms two years apart."
  },
  {
    "startMs": 444639,
    "text": "postmortms two years apart. On November 8, 2023, OpenAI after a day"
  },
  {
    "startMs": 447830,
    "text": "On November 8, 2023, OpenAI after a day"
  },
  {
    "startMs": 447840,
    "text": "On November 8, 2023, OpenAI after a day of unusual heavy traffic, maybe not 40"
  },
  {
    "startMs": 451270,
    "text": "of unusual heavy traffic, maybe not 40"
  },
  {
    "startMs": 451280,
    "text": "of unusual heavy traffic, maybe not 40 requests, it would be some million"
  },
  {
    "startMs": 453350,
    "text": "requests, it would be some million"
  },
  {
    "startMs": 453360,
    "text": "requests, it would be some million requests, their routing servers ran out"
  },
  {
    "startMs": 456150,
    "text": "requests, their routing servers ran out"
  },
  {
    "startMs": 456160,
    "text": "requests, their routing servers ran out of the memory. They started failing"
  },
  {
    "startMs": 457909,
    "text": "of the memory. They started failing"
  },
  {
    "startMs": 457919,
    "text": "of the memory. They started failing their health checks. They added more"
  },
  {
    "startMs": 460550,
    "text": "their health checks. They added more"
  },
  {
    "startMs": 460560,
    "text": "their health checks. They added more replicas, then had to roll them back."
  },
  {
    "startMs": 462629,
    "text": "replicas, then had to roll them back."
  },
  {
    "startMs": 462639,
    "text": "replicas, then had to roll them back. The way out was to cut incoming traffic"
  },
  {
    "startMs": 465350,
    "text": "The way out was to cut incoming traffic"
  },
  {
    "startMs": 465360,
    "text": "The way out was to cut incoming traffic and ramp it up slowly. In January 21"
  },
  {
    "startMs": 470309,
    "text": "and ramp it up slowly. In January 21"
  },
  {
    "startMs": 470319,
    "text": "and ramp it up slowly. In January 21 2026 enthropic the sonnet 4.5 tire gear"
  },
  {
    "startMs": 475589,
    "text": "2026 enthropic the sonnet 4.5 tire gear"
  },
  {
    "startMs": 475599,
    "text": "2026 enthropic the sonnet 4.5 tire gear got slow first then it started returning"
  },
  {
    "startMs": 479110,
    "text": "got slow first then it started returning"
  },
  {
    "startMs": 479120,
    "text": "got slow first then it started returning errors over about 90 minutes. The cause"
  },
  {
    "startMs": 482309,
    "text": "errors over about 90 minutes. The cause"
  },
  {
    "startMs": 482319,
    "text": "errors over about 90 minutes. The cause they gave was simple not enough compute"
  },
  {
    "startMs": 484550,
    "text": "they gave was simple not enough compute"
  },
  {
    "startMs": 484560,
    "text": "they gave was simple not enough compute for the demand."
  },
  {
    "startMs": 487189,
    "text": "for the demand."
  },
  {
    "startMs": 487199,
    "text": "for the demand. And now let's look at what each team"
  },
  {
    "startMs": 490230,
    "text": "And now let's look at what each team"
  },
  {
    "startMs": 490240,
    "text": "And now let's look at what each team decided as their next set of the action"
  },
  {
    "startMs": 492950,
    "text": "decided as their next set of the action"
  },
  {
    "startMs": 492960,
    "text": "decided as their next set of the action item. Neither of them says we need a"
  },
  {
    "startMs": 495430,
    "text": "item. Neither of them says we need a"
  },
  {
    "startMs": 495440,
    "text": "item. Neither of them says we need a faster engine."
  },
  {
    "startMs": 497670,
    "text": "faster engine."
  },
  {
    "startMs": 497680,
    "text": "faster engine. Open AI's followup is graceful load"
  },
  {
    "startMs": 500710,
    "text": "Open AI's followup is graceful load"
  },
  {
    "startMs": 500720,
    "text": "Open AI's followup is graceful load shedding. Enthropics is detecting"
  },
  {
    "startMs": 503990,
    "text": "shedding. Enthropics is detecting"
  },
  {
    "startMs": 504000,
    "text": "shedding. Enthropics is detecting capacity limits earlier."
  },
  {
    "startMs": 506869,
    "text": "capacity limits earlier."
  },
  {
    "startMs": 506879,
    "text": "capacity limits earlier. It's the same idea in the different"
  },
  {
    "startMs": 509029,
    "text": "It's the same idea in the different"
  },
  {
    "startMs": 509039,
    "text": "It's the same idea in the different words which is decide on your request"
  },
  {
    "startMs": 512469,
    "text": "words which is decide on your request"
  },
  {
    "startMs": 512479,
    "text": "words which is decide on your request sooner"
  },
  {
    "startMs": 514310,
    "text": "sooner"
  },
  {
    "startMs": 514320,
    "text": "sooner to make sure which request gets in. And"
  },
  {
    "startMs": 518230,
    "text": "to make sure which request gets in. And"
  },
  {
    "startMs": 518240,
    "text": "to make sure which request gets in. And there are more incidents which we have"
  },
  {
    "startMs": 521430,
    "text": "there are more incidents which we have"
  },
  {
    "startMs": 521440,
    "text": "there are more incidents which we have linked in the appendix. If you are"
  },
  {
    "startMs": 523029,
    "text": "linked in the appendix. If you are"
  },
  {
    "startMs": 523039,
    "text": "linked in the appendix. If you are interested definitely go and read"
  },
  {
    "startMs": 524870,
    "text": "interested definitely go and read"
  },
  {
    "startMs": 524880,
    "text": "interested definitely go and read through them."
  },
  {
    "startMs": 529760,
    "text": "So"
  },
  {
    "startMs": 532150,
    "text": "So"
  },
  {
    "startMs": 532160,
    "text": "So if you walk backward from those"
  },
  {
    "startMs": 534070,
    "text": "if you walk backward from those"
  },
  {
    "startMs": 534080,
    "text": "if you walk backward from those follow-ups,"
  },
  {
    "startMs": 535590,
    "text": "follow-ups,"
  },
  {
    "startMs": 535600,
    "text": "follow-ups, they both point at the same place, the"
  },
  {
    "startMs": 537670,
    "text": "they both point at the same place, the"
  },
  {
    "startMs": 537680,
    "text": "they both point at the same place, the front door. Here is the pipeline again"
  },
  {
    "startMs": 541590,
    "text": "front door. Here is the pipeline again"
  },
  {
    "startMs": 541600,
    "text": "front door. Here is the pipeline again with the missing step filled in."
  },
  {
    "startMs": 545269,
    "text": "with the missing step filled in."
  },
  {
    "startMs": 545279,
    "text": "with the missing step filled in. We have a client, we have an admit"
  },
  {
    "startMs": 548310,
    "text": "We have a client, we have an admit"
  },
  {
    "startMs": 548320,
    "text": "We have a client, we have an admit layer, the Q router, and then the"
  },
  {
    "startMs": 550630,
    "text": "layer, the Q router, and then the"
  },
  {
    "startMs": 550640,
    "text": "layer, the Q router, and then the engine."
  },
  {
    "startMs": 552470,
    "text": "engine."
  },
  {
    "startMs": 552480,
    "text": "engine. It's never the client talking directly"
  },
  {
    "startMs": 554630,
    "text": "It's never the client talking directly"
  },
  {
    "startMs": 554640,
    "text": "It's never the client talking directly to your engine or client talking to your"
  },
  {
    "startMs": 557430,
    "text": "to your engine or client talking to your"
  },
  {
    "startMs": 557440,
    "text": "to your engine or client talking to your queue directly. There is an admission"
  },
  {
    "startMs": 560150,
    "text": "queue directly. There is an admission"
  },
  {
    "startMs": 560160,
    "text": "queue directly. There is an admission phase in between."
  },
  {
    "startMs": 562230,
    "text": "phase in between."
  },
  {
    "startMs": 562240,
    "text": "phase in between. Admission is a decision made before a"
  },
  {
    "startMs": 565590,
    "text": "Admission is a decision made before a"
  },
  {
    "startMs": 565600,
    "text": "Admission is a decision made before a request takes any GPU memory and it only"
  },
  {
    "startMs": 568790,
    "text": "request takes any GPU memory and it only"
  },
  {
    "startMs": 568800,
    "text": "request takes any GPU memory and it only has three possible answers."
  },
  {
    "startMs": 573120,
    "text": "Admission decides"
  },
  {
    "startMs": 576150,
    "text": "Admission decides"
  },
  {
    "startMs": 576160,
    "text": "Admission decides whether to start now or you want to"
  },
  {
    "startMs": 581990,
    "text": "whether to start now or you want to"
  },
  {
    "startMs": 582000,
    "text": "whether to start now or you want to hold the request. You want to accept it"
  },
  {
    "startMs": 584310,
    "text": "hold the request. You want to accept it"
  },
  {
    "startMs": 584320,
    "text": "hold the request. You want to accept it but make it wait. Third is the reject."
  },
  {
    "startMs": 587829,
    "text": "but make it wait. Third is the reject."
  },
  {
    "startMs": 587839,
    "text": "but make it wait. Third is the reject. That means not right now. It can throw"
  },
  {
    "startMs": 590630,
    "text": "That means not right now. It can throw"
  },
  {
    "startMs": 590640,
    "text": "That means not right now. It can throw different amount of errors here such as"
  },
  {
    "startMs": 592790,
    "text": "different amount of errors here such as"
  },
  {
    "startMs": 592800,
    "text": "different amount of errors here such as 429s and we will dive into this bit more"
  },
  {
    "startMs": 596710,
    "text": "429s and we will dive into this bit more"
  },
  {
    "startMs": 596720,
    "text": "429s and we will dive into this bit more deeper in the second part."
  },
  {
    "startMs": 600070,
    "text": "deeper in the second part."
  },
  {
    "startMs": 600080,
    "text": "deeper in the second part. this that is the decision that was"
  },
  {
    "startMs": 601910,
    "text": "this that is the decision that was"
  },
  {
    "startMs": 601920,
    "text": "this that is the decision that was missing from our pipeline."
  },
  {
    "startMs": 606880,
    "text": "So why does it have to happen before the"
  },
  {
    "startMs": 609509,
    "text": "So why does it have to happen before the"
  },
  {
    "startMs": 609519,
    "text": "So why does it have to happen before the engine"
  },
  {
    "startMs": 611110,
    "text": "engine"
  },
  {
    "startMs": 611120,
    "text": "engine and what would happen if you will"
  },
  {
    "startMs": 613910,
    "text": "and what would happen if you will"
  },
  {
    "startMs": 613920,
    "text": "and what would happen if you will actually accept the request what would"
  },
  {
    "startMs": 616150,
    "text": "actually accept the request what would"
  },
  {
    "startMs": 616160,
    "text": "actually accept the request what would be the cost of that yes"
  },
  {
    "startMs": 619030,
    "text": "be the cost of that yes"
  },
  {
    "startMs": 619040,
    "text": "be the cost of that yes so let's try to take a look at this like"
  },
  {
    "startMs": 622310,
    "text": "so let's try to take a look at this like"
  },
  {
    "startMs": 622320,
    "text": "so let's try to take a look at this like two phases of the inference like prefill"
  },
  {
    "startMs": 624470,
    "text": "two phases of the inference like prefill"
  },
  {
    "startMs": 624480,
    "text": "two phases of the inference like prefill and the decode. So first comes the"
  },
  {
    "startMs": 626710,
    "text": "and the decode. So first comes the"
  },
  {
    "startMs": 626720,
    "text": "and the decode. So first comes the prefill. That's the engine reading the W"
  },
  {
    "startMs": 629030,
    "text": "prefill. That's the engine reading the W"
  },
  {
    "startMs": 629040,
    "text": "prefill. That's the engine reading the W prompt, building the request KV cache."
  },
  {
    "startMs": 631670,
    "text": "prompt, building the request KV cache."
  },
  {
    "startMs": 631680,
    "text": "prompt, building the request KV cache. And then comes the decode. That's the"
  },
  {
    "startMs": 633590,
    "text": "And then comes the decode. That's the"
  },
  {
    "startMs": 633600,
    "text": "And then comes the decode. That's the engine writing the answer one token at a"
  },
  {
    "startMs": 635829,
    "text": "engine writing the answer one token at a"
  },
  {
    "startMs": 635839,
    "text": "engine writing the answer one token at a time. With every token, the cash grows a"
  },
  {
    "startMs": 638710,
    "text": "time. With every token, the cash grows a"
  },
  {
    "startMs": 638720,
    "text": "time. With every token, the cash grows a little more. All of that memory stays"
  },
  {
    "startMs": 640630,
    "text": "little more. All of that memory stays"
  },
  {
    "startMs": 640640,
    "text": "little more. All of that memory stays reserved until the request finishes."
  },
  {
    "startMs": 645030,
    "text": "reserved until the request finishes."
  },
  {
    "startMs": 645040,
    "text": "reserved until the request finishes. So a yes is not a moment. It's a"
  },
  {
    "startMs": 648069,
    "text": "So a yes is not a moment. It's a"
  },
  {
    "startMs": 648079,
    "text": "So a yes is not a moment. It's a commitment that lasts the request whole"
  },
  {
    "startMs": 650069,
    "text": "commitment that lasts the request whole"
  },
  {
    "startMs": 650079,
    "text": "commitment that lasts the request whole life. And here you can look at the"
  },
  {
    "startMs": 653430,
    "text": "life. And here you can look at the"
  },
  {
    "startMs": 653440,
    "text": "life. And here you can look at the example."
  },
  {
    "startMs": 655430,
    "text": "example."
  },
  {
    "startMs": 655440,
    "text": "example. There is a request A, there is a request"
  },
  {
    "startMs": 657590,
    "text": "There is a request A, there is a request"
  },
  {
    "startMs": 657600,
    "text": "There is a request A, there is a request B. Once you admit a request A which is a"
  },
  {
    "startMs": 660550,
    "text": "B. Once you admit a request A which is a"
  },
  {
    "startMs": 660560,
    "text": "B. Once you admit a request A which is a short answer request and then you admit"
  },
  {
    "startMs": 662470,
    "text": "short answer request and then you admit"
  },
  {
    "startMs": 662480,
    "text": "short answer request and then you admit a request B which is a long answer"
  },
  {
    "startMs": 664310,
    "text": "a request B which is a long answer"
  },
  {
    "startMs": 664320,
    "text": "a request B which is a long answer request. Now when you admit a request C"
  },
  {
    "startMs": 666790,
    "text": "request. Now when you admit a request C"
  },
  {
    "startMs": 666800,
    "text": "request. Now when you admit a request C which is again a short answer request,"
  },
  {
    "startMs": 669350,
    "text": "which is again a short answer request,"
  },
  {
    "startMs": 669360,
    "text": "which is again a short answer request, we don't know how longer it would take"
  },
  {
    "startMs": 671350,
    "text": "we don't know how longer it would take"
  },
  {
    "startMs": 671360,
    "text": "we don't know how longer it would take because there is a longer request which"
  },
  {
    "startMs": 673350,
    "text": "because there is a longer request which"
  },
  {
    "startMs": 673360,
    "text": "because there is a longer request which was already in the memory. Should we"
  },
  {
    "startMs": 676230,
    "text": "was already in the memory. Should we"
  },
  {
    "startMs": 676240,
    "text": "was already in the memory. Should we even take that request C or should we"
  },
  {
    "startMs": 678310,
    "text": "even take that request C or should we"
  },
  {
    "startMs": 678320,
    "text": "even take that request C or should we not even take the request C. So there is"
  },
  {
    "startMs": 680949,
    "text": "not even take the request C. So there is"
  },
  {
    "startMs": 680959,
    "text": "not even take the request C. So there is a cost associated of like admitting"
  },
  {
    "startMs": 683350,
    "text": "a cost associated of like admitting"
  },
  {
    "startMs": 683360,
    "text": "a cost associated of like admitting every request because admission could"
  },
  {
    "startMs": 686630,
    "text": "every request because admission could"
  },
  {
    "startMs": 686640,
    "text": "every request because admission could still result in missing out your SLOs's."
  },
  {
    "startMs": 694640,
    "text": "Another impact apart from the SLOs's is"
  },
  {
    "startMs": 697670,
    "text": "Another impact apart from the SLOs's is"
  },
  {
    "startMs": 697680,
    "text": "Another impact apart from the SLOs's is your memory."
  },
  {
    "startMs": 700550,
    "text": "your memory."
  },
  {
    "startMs": 700560,
    "text": "your memory. Here is what it does to your GPU. So"
  },
  {
    "startMs": 703350,
    "text": "Here is what it does to your GPU. So"
  },
  {
    "startMs": 703360,
    "text": "Here is what it does to your GPU. So every running request it keeps growing"
  },
  {
    "startMs": 705350,
    "text": "every running request it keeps growing"
  },
  {
    "startMs": 705360,
    "text": "every running request it keeps growing your cache as it writes. So your total"
  },
  {
    "startMs": 707509,
    "text": "your cache as it writes. So your total"
  },
  {
    "startMs": 707519,
    "text": "your cache as it writes. So your total memory keeps climbing. When the memory"
  },
  {
    "startMs": 709509,
    "text": "memory keeps climbing. When the memory"
  },
  {
    "startMs": 709519,
    "text": "memory keeps climbing. When the memory gets to the full engine has three ways"
  },
  {
    "startMs": 711509,
    "text": "gets to the full engine has three ways"
  },
  {
    "startMs": 711519,
    "text": "gets to the full engine has three ways to cut. Either it will aait"
  },
  {
    "startMs": 714949,
    "text": "to cut. Either it will aait"
  },
  {
    "startMs": 714959,
    "text": "to cut. Either it will aait or like throw away your request. Either"
  },
  {
    "startMs": 717269,
    "text": "or like throw away your request. Either"
  },
  {
    "startMs": 717279,
    "text": "or like throw away your request. Either it will put it to your waiting state"
  },
  {
    "startMs": 719190,
    "text": "it will put it to your waiting state"
  },
  {
    "startMs": 719200,
    "text": "it will put it to your waiting state that means it will swap or it will just"
  },
  {
    "startMs": 723110,
    "text": "that means it will swap or it will just"
  },
  {
    "startMs": 723120,
    "text": "that means it will swap or it will just say okay I won't be able to serve and I"
  },
  {
    "startMs": 725430,
    "text": "say okay I won't be able to serve and I"
  },
  {
    "startMs": 725440,
    "text": "say okay I won't be able to serve and I I will probably return you an error."
  },
  {
    "startMs": 729350,
    "text": "I will probably return you an error."
  },
  {
    "startMs": 729360,
    "text": "I will probably return you an error. Every one of those throws I mean every"
  },
  {
    "startMs": 732069,
    "text": "Every one of those throws I mean every"
  },
  {
    "startMs": 732079,
    "text": "Every one of those throws I mean every one of those causes a certain delay in"
  },
  {
    "startMs": 736069,
    "text": "one of those causes a certain delay in"
  },
  {
    "startMs": 736079,
    "text": "one of those causes a certain delay in processing of the request. and missing"
  },
  {
    "startMs": 739030,
    "text": "processing of the request. and missing"
  },
  {
    "startMs": 739040,
    "text": "processing of the request. and missing out your SLOs's again while also wasting"
  },
  {
    "startMs": 741910,
    "text": "out your SLOs's again while also wasting"
  },
  {
    "startMs": 741920,
    "text": "out your SLOs's again while also wasting your memory."
  },
  {
    "startMs": 750000,
    "text": "So why is it like our problem like we"
  },
  {
    "startMs": 753509,
    "text": "So why is it like our problem like we"
  },
  {
    "startMs": 753519,
    "text": "So why is it like our problem like we have con servers from open AI anthropic"
  },
  {
    "startMs": 757670,
    "text": "have con servers from open AI anthropic"
  },
  {
    "startMs": 757680,
    "text": "have con servers from open AI anthropic bold teams which are serving their own"
  },
  {
    "startMs": 759829,
    "text": "bold teams which are serving their own"
  },
  {
    "startMs": 759839,
    "text": "bold teams which are serving their own infrastructure"
  },
  {
    "startMs": 761430,
    "text": "infrastructure"
  },
  {
    "startMs": 761440,
    "text": "infrastructure for the inference."
  },
  {
    "startMs": 763990,
    "text": "for the inference."
  },
  {
    "startMs": 764000,
    "text": "for the inference. So it's a problem for us because"
  },
  {
    "startMs": 768790,
    "text": "So it's a problem for us because"
  },
  {
    "startMs": 768800,
    "text": "So it's a problem for us because there are teams who want privacy. They"
  },
  {
    "startMs": 771670,
    "text": "there are teams who want privacy. They"
  },
  {
    "startMs": 771680,
    "text": "there are teams who want privacy. They want to have their own models hosted on"
  },
  {
    "startMs": 774389,
    "text": "want to have their own models hosted on"
  },
  {
    "startMs": 774399,
    "text": "want to have their own models hosted on their own GPUs"
  },
  {
    "startMs": 777670,
    "text": "their own GPUs"
  },
  {
    "startMs": 777680,
    "text": "their own GPUs and they want it to be hosted on their"
  },
  {
    "startMs": 779829,
    "text": "and they want it to be hosted on their"
  },
  {
    "startMs": 779839,
    "text": "and they want it to be hosted on their own GPUs due to the cost perspectives"
  },
  {
    "startMs": 782790,
    "text": "own GPUs due to the cost perspectives"
  },
  {
    "startMs": 782800,
    "text": "own GPUs due to the cost perspectives but they have to also protect"
  },
  {
    "startMs": 785990,
    "text": "but they have to also protect"
  },
  {
    "startMs": 786000,
    "text": "but they have to also protect the burndown on those GPUs."
  },
  {
    "startMs": 788949,
    "text": "the burndown on those GPUs."
  },
  {
    "startMs": 788959,
    "text": "the burndown on those GPUs. So they need to effectively decide which"
  },
  {
    "startMs": 791190,
    "text": "So they need to effectively decide which"
  },
  {
    "startMs": 791200,
    "text": "So they need to effectively decide which request to serve across their inference"
  },
  {
    "startMs": 794230,
    "text": "request to serve across their inference"
  },
  {
    "startMs": 794240,
    "text": "request to serve across their inference ecosystem."
  },
  {
    "startMs": 802079,
    "text": "On top of"
  },
  {
    "startMs": 804069,
    "text": "On top of"
  },
  {
    "startMs": 804079,
    "text": "On top of and on top of that like the traffic"
  },
  {
    "startMs": 806870,
    "text": "and on top of that like the traffic"
  },
  {
    "startMs": 806880,
    "text": "and on top of that like the traffic arriving at the door has also changed."
  },
  {
    "startMs": 810069,
    "text": "arriving at the door has also changed."
  },
  {
    "startMs": 810079,
    "text": "arriving at the door has also changed. Earlier there used to be a person that"
  },
  {
    "startMs": 812069,
    "text": "Earlier there used to be a person that"
  },
  {
    "startMs": 812079,
    "text": "Earlier there used to be a person that was chatting. He would send one request"
  },
  {
    "startMs": 814470,
    "text": "was chatting. He would send one request"
  },
  {
    "startMs": 814480,
    "text": "was chatting. He would send one request at a time. There would be a seconds of"
  },
  {
    "startMs": 816389,
    "text": "at a time. There would be a seconds of"
  },
  {
    "startMs": 816399,
    "text": "at a time. There would be a seconds of reading in between and then there would"
  },
  {
    "startMs": 818069,
    "text": "reading in between and then there would"
  },
  {
    "startMs": 818079,
    "text": "reading in between and then there would be a slow answer. But now coding agents"
  },
  {
    "startMs": 820629,
    "text": "be a slow answer. But now coding agents"
  },
  {
    "startMs": 820639,
    "text": "be a slow answer. But now coding agents they have taken over. There are tons of"
  },
  {
    "startMs": 822790,
    "text": "they have taken over. There are tons of"
  },
  {
    "startMs": 822800,
    "text": "they have taken over. There are tons of calls that these coding agents they send"
  },
  {
    "startMs": 825910,
    "text": "calls that these coding agents they send"
  },
  {
    "startMs": 825920,
    "text": "calls that these coding agents they send along with the retries, pauses,"
  },
  {
    "startMs": 829430,
    "text": "along with the retries, pauses,"
  },
  {
    "startMs": 829440,
    "text": "along with the retries, pauses, um"
  },
  {
    "startMs": 831110,
    "text": "um"
  },
  {
    "startMs": 831120,
    "text": "um like the undo calls as well and it there"
  },
  {
    "startMs": 834790,
    "text": "like the undo calls as well and it there"
  },
  {
    "startMs": 834800,
    "text": "like the undo calls as well and it there is such a overwhelming traffic that your"
  },
  {
    "startMs": 837350,
    "text": "is such a overwhelming traffic that your"
  },
  {
    "startMs": 837360,
    "text": "is such a overwhelming traffic that your cluster has to handle and what that"
  },
  {
    "startMs": 840310,
    "text": "cluster has to handle and what that"
  },
  {
    "startMs": 840320,
    "text": "cluster has to handle and what that means is cluster has a responsibility to"
  },
  {
    "startMs": 843670,
    "text": "means is cluster has a responsibility to"
  },
  {
    "startMs": 843680,
    "text": "means is cluster has a responsibility to take a wise decision of which requests"
  },
  {
    "startMs": 845750,
    "text": "take a wise decision of which requests"
  },
  {
    "startMs": 845760,
    "text": "take a wise decision of which requests to accept and which uh requests to"
  },
  {
    "startMs": 848230,
    "text": "to accept and which uh requests to"
  },
  {
    "startMs": 848240,
    "text": "to accept and which uh requests to reject."
  },
  {
    "startMs": 855279,
    "text": "And this is exactly that visualization."
  },
  {
    "startMs": 857590,
    "text": "And this is exactly that visualization."
  },
  {
    "startMs": 857600,
    "text": "And this is exactly that visualization. You ask your agent that fix an o test."
  },
  {
    "startMs": 861590,
    "text": "You ask your agent that fix an o test."
  },
  {
    "startMs": 861600,
    "text": "You ask your agent that fix an o test. It runs all your it tries to run"
  },
  {
    "startMs": 864150,
    "text": "It runs all your it tries to run"
  },
  {
    "startMs": 864160,
    "text": "It runs all your it tries to run multiple tool calls that is reading of"
  },
  {
    "startMs": 867030,
    "text": "multiple tool calls that is reading of"
  },
  {
    "startMs": 867040,
    "text": "multiple tool calls that is reading of the files like around there could be"
  },
  {
    "startMs": 869910,
    "text": "the files like around there could be"
  },
  {
    "startMs": 869920,
    "text": "the files like around there could be like 14 files. There could be like"
  },
  {
    "startMs": 872389,
    "text": "like 14 files. There could be like"
  },
  {
    "startMs": 872399,
    "text": "like 14 files. There could be like around 10 tests that it has to run and"
  },
  {
    "startMs": 875670,
    "text": "around 10 tests that it has to run and"
  },
  {
    "startMs": 875680,
    "text": "around 10 tests that it has to run and it would combine together to make 40 50"
  },
  {
    "startMs": 878389,
    "text": "it would combine together to make 40 50"
  },
  {
    "startMs": 878399,
    "text": "it would combine together to make 40 50 calls for just one request and all these"
  },
  {
    "startMs": 881990,
    "text": "calls for just one request and all these"
  },
  {
    "startMs": 882000,
    "text": "calls for just one request and all these 40 requests they require like reaching"
  },
  {
    "startMs": 884310,
    "text": "40 requests they require like reaching"
  },
  {
    "startMs": 884320,
    "text": "40 requests they require like reaching out to the LLM getting back the"
  },
  {
    "startMs": 886310,
    "text": "out to the LLM getting back the"
  },
  {
    "startMs": 886320,
    "text": "out to the LLM getting back the response. Uh and in between some of"
  },
  {
    "startMs": 890870,
    "text": "response. Uh and in between some of"
  },
  {
    "startMs": 890880,
    "text": "response. Uh and in between some of these requests would be long running"
  },
  {
    "startMs": 892629,
    "text": "these requests would be long running"
  },
  {
    "startMs": 892639,
    "text": "these requests would be long running some of the requests would be short"
  },
  {
    "startMs": 895509,
    "text": "some of the requests would be short"
  },
  {
    "startMs": 895519,
    "text": "some of the requests would be short prompt request. So we need to decide"
  },
  {
    "startMs": 897910,
    "text": "prompt request. So we need to decide"
  },
  {
    "startMs": 897920,
    "text": "prompt request. So we need to decide effectively where which one to pick and"
  },
  {
    "startMs": 900629,
    "text": "effectively where which one to pick and"
  },
  {
    "startMs": 900639,
    "text": "effectively where which one to pick and which one not to pick."
  },
  {
    "startMs": 909040,
    "text": "Yeah. And that was the like reason why"
  },
  {
    "startMs": 912629,
    "text": "Yeah. And that was the like reason why"
  },
  {
    "startMs": 912639,
    "text": "Yeah. And that was the like reason why we need to worry about"
  },
  {
    "startMs": 915829,
    "text": "we need to worry about"
  },
  {
    "startMs": 915839,
    "text": "we need to worry about admission control. Why we need to worry"
  },
  {
    "startMs": 918550,
    "text": "admission control. Why we need to worry"
  },
  {
    "startMs": 918560,
    "text": "admission control. Why we need to worry about"
  },
  {
    "startMs": 921430,
    "text": "about"
  },
  {
    "startMs": 921440,
    "text": "about handling the traffic right way across"
  },
  {
    "startMs": 923590,
    "text": "handling the traffic right way across"
  },
  {
    "startMs": 923600,
    "text": "handling the traffic right way across our infancy ecosystem."
  },
  {
    "startMs": 932959,
    "text": "Okay. So this brings us to the admission"
  },
  {
    "startMs": 936310,
    "text": "Okay. So this brings us to the admission"
  },
  {
    "startMs": 936320,
    "text": "Okay. So this brings us to the admission control."
  },
  {
    "startMs": 937829,
    "text": "control."
  },
  {
    "startMs": 937839,
    "text": "control. Uh now moving on to the next slide. Uh"
  },
  {
    "startMs": 942069,
    "text": "Uh now moving on to the next slide. Uh"
  },
  {
    "startMs": 942079,
    "text": "Uh now moving on to the next slide. Uh that is slide number 18. So in this"
  },
  {
    "startMs": 945670,
    "text": "that is slide number 18. So in this"
  },
  {
    "startMs": 945680,
    "text": "that is slide number 18. So in this slide we will be covering three"
  },
  {
    "startMs": 947350,
    "text": "slide we will be covering three"
  },
  {
    "startMs": 947360,
    "text": "slide we will be covering three questions. Where does this decision"
  },
  {
    "startMs": 949990,
    "text": "questions. Where does this decision"
  },
  {
    "startMs": 950000,
    "text": "questions. Where does this decision from? which request to admit"
  },
  {
    "startMs": 952870,
    "text": "from? which request to admit"
  },
  {
    "startMs": 952880,
    "text": "from? which request to admit and which to reject sits back at your"
  },
  {
    "startMs": 955749,
    "text": "and which to reject sits back at your"
  },
  {
    "startMs": 955759,
    "text": "and which to reject sits back at your stack"
  },
  {
    "startMs": 957910,
    "text": "stack"
  },
  {
    "startMs": 957920,
    "text": "stack and the another question is what does it"
  },
  {
    "startMs": 960790,
    "text": "and the another question is what does it"
  },
  {
    "startMs": 960800,
    "text": "and the another question is what does it need to see to make a good call or"
  },
  {
    "startMs": 963829,
    "text": "need to see to make a good call or"
  },
  {
    "startMs": 963839,
    "text": "need to see to make a good call or judgment on the request and what does it"
  },
  {
    "startMs": 967749,
    "text": "judgment on the request and what does it"
  },
  {
    "startMs": 967759,
    "text": "judgment on the request and what does it returns back to the client. Moving on to"
  },
  {
    "startMs": 970870,
    "text": "returns back to the client. Moving on to"
  },
  {
    "startMs": 970880,
    "text": "returns back to the client. Moving on to the next slide. So here uh we see the"
  },
  {
    "startMs": 975269,
    "text": "the next slide. So here uh we see the"
  },
  {
    "startMs": 975279,
    "text": "the next slide. So here uh we see the pipeline again which each layer's jobs"
  },
  {
    "startMs": 977749,
    "text": "pipeline again which each layer's jobs"
  },
  {
    "startMs": 977759,
    "text": "pipeline again which each layer's jobs is written out. The agent sees its own"
  },
  {
    "startMs": 981749,
    "text": "is written out. The agent sees its own"
  },
  {
    "startMs": 981759,
    "text": "is written out. The agent sees its own call and decides when to retry."
  },
  {
    "startMs": 985509,
    "text": "call and decides when to retry."
  },
  {
    "startMs": 985519,
    "text": "call and decides when to retry. The gateway sees API key and request"
  },
  {
    "startMs": 988310,
    "text": "The gateway sees API key and request"
  },
  {
    "startMs": 988320,
    "text": "The gateway sees API key and request count and enforces a rate limit rate"
  },
  {
    "startMs": 991509,
    "text": "count and enforces a rate limit rate"
  },
  {
    "startMs": 991519,
    "text": "count and enforces a rate limit rate limiter for each and the router which"
  },
  {
    "startMs": 995269,
    "text": "limiter for each and the router which"
  },
  {
    "startMs": 995279,
    "text": "limiter for each and the router which sees replicas or healthy and which one"
  },
  {
    "startMs": 997509,
    "text": "sees replicas or healthy and which one"
  },
  {
    "startMs": 997519,
    "text": "sees replicas or healthy and which one to pick. Now the engine sees its KV"
  },
  {
    "startMs": 1001509,
    "text": "to pick. Now the engine sees its KV"
  },
  {
    "startMs": 1001519,
    "text": "to pick. Now the engine sees its KV cache and the current batch and decides"
  },
  {
    "startMs": 1004470,
    "text": "cache and the current batch and decides"
  },
  {
    "startMs": 1004480,
    "text": "cache and the current batch and decides the order of request to run and what to"
  },
  {
    "startMs": 1007430,
    "text": "the order of request to run and what to"
  },
  {
    "startMs": 1007440,
    "text": "the order of request to run and what to evict and print."
  },
  {
    "startMs": 1010389,
    "text": "evict and print."
  },
  {
    "startMs": 1010399,
    "text": "evict and print. So every layer decides something. Now"
  },
  {
    "startMs": 1014150,
    "text": "So every layer decides something. Now"
  },
  {
    "startMs": 1014160,
    "text": "So every layer decides something. Now ask each of them one question. Is the"
  },
  {
    "startMs": 1017110,
    "text": "ask each of them one question. Is the"
  },
  {
    "startMs": 1017120,
    "text": "ask each of them one question. Is the room for this request?"
  },
  {
    "startMs": 1020629,
    "text": "room for this request?"
  },
  {
    "startMs": 1020639,
    "text": "room for this request? The agent says not my job. The gateway"
  },
  {
    "startMs": 1023430,
    "text": "The agent says not my job. The gateway"
  },
  {
    "startMs": 1023440,
    "text": "The agent says not my job. The gateway says I cannot see it. And the router say"
  },
  {
    "startMs": 1027029,
    "text": "says I cannot see it. And the router say"
  },
  {
    "startMs": 1027039,
    "text": "says I cannot see it. And the router say not my job and the engine says I only"
  },
  {
    "startMs": 1029909,
    "text": "not my job and the engine says I only"
  },
  {
    "startMs": 1029919,
    "text": "not my job and the engine says I only find out when I have already said yes."
  },
  {
    "startMs": 1033829,
    "text": "find out when I have already said yes."
  },
  {
    "startMs": 1033839,
    "text": "find out when I have already said yes. The admission box is missing in in"
  },
  {
    "startMs": 1037510,
    "text": "The admission box is missing in in"
  },
  {
    "startMs": 1037520,
    "text": "The admission box is missing in in between."
  },
  {
    "startMs": 1039990,
    "text": "between."
  },
  {
    "startMs": 1040000,
    "text": "between. So now moving on to the next slide. Um"
  },
  {
    "startMs": 1043750,
    "text": "So now moving on to the next slide. Um"
  },
  {
    "startMs": 1043760,
    "text": "So now moving on to the next slide. Um so where to put this decision change"
  },
  {
    "startMs": 1048150,
    "text": "so where to put this decision change"
  },
  {
    "startMs": 1048160,
    "text": "so where to put this decision change what it can see and what it cost you to"
  },
  {
    "startMs": 1051750,
    "text": "what it can see and what it cost you to"
  },
  {
    "startMs": 1051760,
    "text": "what it can see and what it cost you to run. So there are three possible"
  },
  {
    "startMs": 1054950,
    "text": "run. So there are three possible"
  },
  {
    "startMs": 1054960,
    "text": "run. So there are three possible moments. The first is an invoice right"
  },
  {
    "startMs": 1058549,
    "text": "moments. The first is an invoice right"
  },
  {
    "startMs": 1058559,
    "text": "moments. The first is an invoice right in front of the engine. Here the policy"
  },
  {
    "startMs": 1061190,
    "text": "in front of the engine. Here the policy"
  },
  {
    "startMs": 1061200,
    "text": "in front of the engine. Here the policy sees the request itself plus whatever"
  },
  {
    "startMs": 1063830,
    "text": "sees the request itself plus whatever"
  },
  {
    "startMs": 1063840,
    "text": "sees the request itself plus whatever metrics the engine exposes. All you need"
  },
  {
    "startMs": 1067110,
    "text": "metrics the engine exposes. All you need"
  },
  {
    "startMs": 1067120,
    "text": "metrics the engine exposes. All you need is proxy or gateway. The engine stays"
  },
  {
    "startMs": 1069990,
    "text": "is proxy or gateway. The engine stays"
  },
  {
    "startMs": 1070000,
    "text": "is proxy or gateway. The engine stays exactly as it is. The second one is the"
  },
  {
    "startMs": 1073909,
    "text": "exactly as it is. The second one is the"
  },
  {
    "startMs": 1073919,
    "text": "exactly as it is. The second one is the Q boundary. Now the decision happens"
  },
  {
    "startMs": 1076549,
    "text": "Q boundary. Now the decision happens"
  },
  {
    "startMs": 1076559,
    "text": "Q boundary. Now the decision happens inside the engineuler"
  },
  {
    "startMs": 1078950,
    "text": "inside the engineuler"
  },
  {
    "startMs": 1078960,
    "text": "inside the engineuler as it picks the next batch to run."
  },
  {
    "startMs": 1083190,
    "text": "as it picks the next batch to run."
  },
  {
    "startMs": 1083200,
    "text": "as it picks the next batch to run. The third is during decoding when the"
  },
  {
    "startMs": 1085990,
    "text": "The third is during decoding when the"
  },
  {
    "startMs": 1086000,
    "text": "The third is during decoding when the scheduleuler can pause a request or take"
  },
  {
    "startMs": 1088470,
    "text": "scheduleuler can pause a request or take"
  },
  {
    "startMs": 1088480,
    "text": "scheduleuler can pause a request or take back its cache halfway through an"
  },
  {
    "startMs": 1091110,
    "text": "back its cache halfway through an"
  },
  {
    "startMs": 1091120,
    "text": "back its cache halfway through an answer. Those later movement can see"
  },
  {
    "startMs": 1094390,
    "text": "answer. Those later movement can see"
  },
  {
    "startMs": 1094400,
    "text": "answer. Those later movement can see more but they mean changing the"
  },
  {
    "startMs": 1096950,
    "text": "more but they mean changing the"
  },
  {
    "startMs": 1096960,
    "text": "more but they mean changing the engineuler."
  },
  {
    "startMs": 1098789,
    "text": "engineuler."
  },
  {
    "startMs": 1098799,
    "text": "engineuler. That means keeping your own modified"
  },
  {
    "startMs": 1100789,
    "text": "That means keeping your own modified"
  },
  {
    "startMs": 1100799,
    "text": "That means keeping your own modified copy of BLLM or SG lang and updating it"
  },
  {
    "startMs": 1104789,
    "text": "copy of BLLM or SG lang and updating it"
  },
  {
    "startMs": 1104799,
    "text": "copy of BLLM or SG lang and updating it every time original changes forever. So"
  },
  {
    "startMs": 1108789,
    "text": "every time original changes forever. So"
  },
  {
    "startMs": 1108799,
    "text": "every time original changes forever. So an ingress is the only movement that"
  },
  {
    "startMs": 1110870,
    "text": "an ingress is the only movement that"
  },
  {
    "startMs": 1110880,
    "text": "an ingress is the only movement that works with the engine exactly as it"
  },
  {
    "startMs": 1113029,
    "text": "works with the engine exactly as it"
  },
  {
    "startMs": 1113039,
    "text": "works with the engine exactly as it ships."
  },
  {
    "startMs": 1114630,
    "text": "ships."
  },
  {
    "startMs": 1114640,
    "text": "ships. So that's where we will stay for the"
  },
  {
    "startMs": 1116630,
    "text": "So that's where we will stay for the"
  },
  {
    "startMs": 1116640,
    "text": "So that's where we will stay for the rest of the talk. So now moving on to"
  },
  {
    "startMs": 1119669,
    "text": "rest of the talk. So now moving on to"
  },
  {
    "startMs": 1119679,
    "text": "rest of the talk. So now moving on to the next slide."
  },
  {
    "startMs": 1121990,
    "text": "the next slide."
  },
  {
    "startMs": 1122000,
    "text": "the next slide. Uh so what actually sits in ingress,"
  },
  {
    "startMs": 1126070,
    "text": "Uh so what actually sits in ingress,"
  },
  {
    "startMs": 1126080,
    "text": "Uh so what actually sits in ingress, right? Uh so you you you can think of it"
  },
  {
    "startMs": 1129110,
    "text": "right? Uh so you you you can think of it"
  },
  {
    "startMs": 1129120,
    "text": "right? Uh so you you you can think of it as a function. It can take in what it"
  },
  {
    "startMs": 1132549,
    "text": "as a function. It can take in what it"
  },
  {
    "startMs": 1132559,
    "text": "as a function. It can take in what it knows about the request and current"
  },
  {
    "startMs": 1134710,
    "text": "knows about the request and current"
  },
  {
    "startMs": 1134720,
    "text": "knows about the request and current state of the system and it returns one"
  },
  {
    "startMs": 1138710,
    "text": "state of the system and it returns one"
  },
  {
    "startMs": 1138720,
    "text": "state of the system and it returns one of the three answers"
  },
  {
    "startMs": 1141270,
    "text": "of the three answers"
  },
  {
    "startMs": 1141280,
    "text": "of the three answers like if we have to admit the request. So"
  },
  {
    "startMs": 1144630,
    "text": "like if we have to admit the request. So"
  },
  {
    "startMs": 1144640,
    "text": "like if we have to admit the request. So the admit is the request enters the"
  },
  {
    "startMs": 1147110,
    "text": "the admit is the request enters the"
  },
  {
    "startMs": 1147120,
    "text": "the admit is the request enters the engine now and memory is committed for"
  },
  {
    "startMs": 1149110,
    "text": "engine now and memory is committed for"
  },
  {
    "startMs": 1149120,
    "text": "engine now and memory is committed for this moment. If it's deferred with a"
  },
  {
    "startMs": 1153190,
    "text": "this moment. If it's deferred with a"
  },
  {
    "startMs": 1153200,
    "text": "this moment. If it's deferred with a time we can call toao hold on like uh"
  },
  {
    "startMs": 1156630,
    "text": "time we can call toao hold on like uh"
  },
  {
    "startMs": 1156640,
    "text": "time we can call toao hold on like uh hold on and come back after some time"
  },
  {
    "startMs": 1159510,
    "text": "hold on and come back after some time"
  },
  {
    "startMs": 1159520,
    "text": "hold on and come back after some time and there is another decision which is"
  },
  {
    "startMs": 1161990,
    "text": "and there is another decision which is"
  },
  {
    "startMs": 1162000,
    "text": "and there is another decision which is with uh reject with a code that no here"
  },
  {
    "startMs": 1166150,
    "text": "with uh reject with a code that no here"
  },
  {
    "startMs": 1166160,
    "text": "with uh reject with a code that no here is the reason why and until the policy"
  },
  {
    "startMs": 1169270,
    "text": "is the reason why and until the policy"
  },
  {
    "startMs": 1169280,
    "text": "is the reason why and until the policy says admit the request cost you nothing."
  },
  {
    "startMs": 1172549,
    "text": "says admit the request cost you nothing."
  },
  {
    "startMs": 1172559,
    "text": "says admit the request cost you nothing. It holds no GPU memory at all."
  },
  {
    "startMs": 1177029,
    "text": "It holds no GPU memory at all."
  },
  {
    "startMs": 1177039,
    "text": "It holds no GPU memory at all. Now moving on to the next slide."
  },
  {
    "startMs": 1183840,
    "text": "uh to make the call well the policy"
  },
  {
    "startMs": 1186710,
    "text": "uh to make the call well the policy"
  },
  {
    "startMs": 1186720,
    "text": "uh to make the call well the policy needs two views at the same time from"
  },
  {
    "startMs": 1189909,
    "text": "needs two views at the same time from"
  },
  {
    "startMs": 1189919,
    "text": "needs two views at the same time from the request and from the engine. So from"
  },
  {
    "startMs": 1193029,
    "text": "the request and from the engine. So from"
  },
  {
    "startMs": 1193039,
    "text": "the request and from the engine. So from the request how long the input is, which"
  },
  {
    "startMs": 1196710,
    "text": "the request how long the input is, which"
  },
  {
    "startMs": 1196720,
    "text": "the request how long the input is, which client or tenant sends it, which agent"
  },
  {
    "startMs": 1199830,
    "text": "client or tenant sends it, which agent"
  },
  {
    "startMs": 1199840,
    "text": "client or tenant sends it, which agent session it belongs to, and a deadline."
  },
  {
    "startMs": 1203750,
    "text": "session it belongs to, and a deadline."
  },
  {
    "startMs": 1203760,
    "text": "session it belongs to, and a deadline. And from the engine point of view, how"
  },
  {
    "startMs": 1206390,
    "text": "And from the engine point of view, how"
  },
  {
    "startMs": 1206400,
    "text": "And from the engine point of view, how long uh KV cache is, how long the oldest"
  },
  {
    "startMs": 1210230,
    "text": "long uh KV cache is, how long the oldest"
  },
  {
    "startMs": 1210240,
    "text": "long uh KV cache is, how long the oldest request has been waiting in the queue,"
  },
  {
    "startMs": 1212870,
    "text": "request has been waiting in the queue,"
  },
  {
    "startMs": 1212880,
    "text": "request has been waiting in the queue, how often the prompt is already cached,"
  },
  {
    "startMs": 1215110,
    "text": "how often the prompt is already cached,"
  },
  {
    "startMs": 1215120,
    "text": "how often the prompt is already cached, and how how loaded each pool of replica"
  },
  {
    "startMs": 1219110,
    "text": "and how how loaded each pool of replica"
  },
  {
    "startMs": 1219120,
    "text": "and how how loaded each pool of replica is. Now look back at the ST. The gateway"
  },
  {
    "startMs": 1222950,
    "text": "is. Now look back at the ST. The gateway"
  },
  {
    "startMs": 1222960,
    "text": "is. Now look back at the ST. The gateway has left side and the engine has the"
  },
  {
    "startMs": 1226630,
    "text": "has left side and the engine has the"
  },
  {
    "startMs": 1226640,
    "text": "has left side and the engine has the right side. Nobody has the both."
  },
  {
    "startMs": 1230870,
    "text": "right side. Nobody has the both."
  },
  {
    "startMs": 1230880,
    "text": "right side. Nobody has the both. And that's really why admission has to"
  },
  {
    "startMs": 1233110,
    "text": "And that's really why admission has to"
  },
  {
    "startMs": 1233120,
    "text": "And that's really why admission has to be its own layer. Now moving on to the"
  },
  {
    "startMs": 1236149,
    "text": "be its own layer. Now moving on to the"
  },
  {
    "startMs": 1236159,
    "text": "be its own layer. Now moving on to the next slide."
  },
  {
    "startMs": 1239029,
    "text": "next slide."
  },
  {
    "startMs": 1239039,
    "text": "next slide. Here are those eight signals turned into"
  },
  {
    "startMs": 1241909,
    "text": "Here are those eight signals turned into"
  },
  {
    "startMs": 1241919,
    "text": "Here are those eight signals turned into simple rule. Each one is compared to"
  },
  {
    "startMs": 1244390,
    "text": "simple rule. Each one is compared to"
  },
  {
    "startMs": 1244400,
    "text": "simple rule. Each one is compared to threshold and when it crosses it pushes"
  },
  {
    "startMs": 1247510,
    "text": "threshold and when it crosses it pushes"
  },
  {
    "startMs": 1247520,
    "text": "threshold and when it crosses it pushes towards the outcome in the colored pale"
  },
  {
    "startMs": 1251110,
    "text": "towards the outcome in the colored pale"
  },
  {
    "startMs": 1251120,
    "text": "towards the outcome in the colored pale from the request side."
  },
  {
    "startMs": 1253909,
    "text": "from the request side."
  },
  {
    "startMs": 1253919,
    "text": "from the request side. If the input is too big to fit in the"
  },
  {
    "startMs": 1256630,
    "text": "If the input is too big to fit in the"
  },
  {
    "startMs": 1256640,
    "text": "If the input is too big to fit in the memory headroom just refer it. If a"
  },
  {
    "startMs": 1259909,
    "text": "memory headroom just refer it. If a"
  },
  {
    "startMs": 1259919,
    "text": "memory headroom just refer it. If a client or tenant is over"
  },
  {
    "startMs": 1262950,
    "text": "client or tenant is over"
  },
  {
    "startMs": 1262960,
    "text": "client or tenant is over it share reject it with 429."
  },
  {
    "startMs": 1266230,
    "text": "it share reject it with 429."
  },
  {
    "startMs": 1266240,
    "text": "it share reject it with 429. If one agent session has too many calls"
  },
  {
    "startMs": 1269029,
    "text": "If one agent session has too many calls"
  },
  {
    "startMs": 1269039,
    "text": "If one agent session has too many calls in flight defer with extras. If waiting"
  },
  {
    "startMs": 1273350,
    "text": "in flight defer with extras. If waiting"
  },
  {
    "startMs": 1273360,
    "text": "in flight defer with extras. If waiting would make a request miss its deadline"
  },
  {
    "startMs": 1275830,
    "text": "would make a request miss its deadline"
  },
  {
    "startMs": 1275840,
    "text": "would make a request miss its deadline anyway reject it from the engine side."
  },
  {
    "startMs": 1279669,
    "text": "anyway reject it from the engine side."
  },
  {
    "startMs": 1279679,
    "text": "anyway reject it from the engine side. Uh you can see at the bottom half of"
  },
  {
    "startMs": 1283750,
    "text": "Uh you can see at the bottom half of"
  },
  {
    "startMs": 1283760,
    "text": "Uh you can see at the bottom half of bottom half side. While KB cache is"
  },
  {
    "startMs": 1286630,
    "text": "bottom half side. While KB cache is"
  },
  {
    "startMs": 1286640,
    "text": "bottom half side. While KB cache is under high water mark admit it. When the"
  },
  {
    "startMs": 1291029,
    "text": "under high water mark admit it. When the"
  },
  {
    "startMs": 1291039,
    "text": "under high water mark admit it. When the oldest request has waited longer than"
  },
  {
    "startMs": 1293350,
    "text": "oldest request has waited longer than"
  },
  {
    "startMs": 1293360,
    "text": "oldest request has waited longer than your target defer it. If the prompt is"
  },
  {
    "startMs": 1296789,
    "text": "your target defer it. If the prompt is"
  },
  {
    "startMs": 1296799,
    "text": "your target defer it. If the prompt is already cached admit it first because"
  },
  {
    "startMs": 1299270,
    "text": "already cached admit it first because"
  },
  {
    "startMs": 1299280,
    "text": "already cached admit it first because it's cheap to serve. And when every pool"
  },
  {
    "startMs": 1301909,
    "text": "it's cheap to serve. And when every pool"
  },
  {
    "startMs": 1301919,
    "text": "it's cheap to serve. And when every pool is full, reject it with 529. One rule of"
  },
  {
    "startMs": 1305669,
    "text": "is full, reject it with 529. One rule of"
  },
  {
    "startMs": 1305679,
    "text": "is full, reject it with 529. One rule of thumb, pick the signal that changes in"
  },
  {
    "startMs": 1308950,
    "text": "thumb, pick the signal that changes in"
  },
  {
    "startMs": 1308960,
    "text": "thumb, pick the signal that changes in second, like a QH, not a one minute"
  },
  {
    "startMs": 1312310,
    "text": "second, like a QH, not a one minute"
  },
  {
    "startMs": 1312320,
    "text": "second, like a QH, not a one minute average. A coding agent sends 40 calls"
  },
  {
    "startMs": 1315510,
    "text": "average. A coding agent sends 40 calls"
  },
  {
    "startMs": 1315520,
    "text": "average. A coding agent sends 40 calls in 2 seconds."
  },
  {
    "startMs": 1317830,
    "text": "in 2 seconds."
  },
  {
    "startMs": 1317840,
    "text": "in 2 seconds. A minute average won't even notice. So"
  },
  {
    "startMs": 1320549,
    "text": "A minute average won't even notice. So"
  },
  {
    "startMs": 1320559,
    "text": "A minute average won't even notice. So that's what the policy reads. Now what"
  },
  {
    "startMs": 1323430,
    "text": "that's what the policy reads. Now what"
  },
  {
    "startMs": 1323440,
    "text": "that's what the policy reads. Now what does it say back? Now moving on to the"
  },
  {
    "startMs": 1326070,
    "text": "does it say back? Now moving on to the"
  },
  {
    "startMs": 1326080,
    "text": "does it say back? Now moving on to the next slide."
  },
  {
    "startMs": 1328470,
    "text": "next slide."
  },
  {
    "startMs": 1328480,
    "text": "next slide. We will take the three answers one at a"
  },
  {
    "startMs": 1331750,
    "text": "We will take the three answers one at a"
  },
  {
    "startMs": 1331760,
    "text": "We will take the three answers one at a time and look at each from both sides."
  },
  {
    "startMs": 1335350,
    "text": "time and look at each from both sides."
  },
  {
    "startMs": 1335360,
    "text": "time and look at each from both sides. What the server sends and what your"
  },
  {
    "startMs": 1337909,
    "text": "What the server sends and what your"
  },
  {
    "startMs": 1337919,
    "text": "What the server sends and what your agent should do with it. First, yes, the"
  },
  {
    "startMs": 1341270,
    "text": "agent should do with it. First, yes, the"
  },
  {
    "startMs": 1341280,
    "text": "agent should do with it. First, yes, the server sends back a 200 and the response"
  },
  {
    "startMs": 1344149,
    "text": "server sends back a 200 and the response"
  },
  {
    "startMs": 1344159,
    "text": "server sends back a 200 and the response starts streaming. At that moment, three"
  },
  {
    "startMs": 1346950,
    "text": "starts streaming. At that moment, three"
  },
  {
    "startMs": 1346960,
    "text": "starts streaming. At that moment, three things happens inside the engine. It"
  },
  {
    "startMs": 1349350,
    "text": "things happens inside the engine. It"
  },
  {
    "startMs": 1349360,
    "text": "things happens inside the engine. It commits KV cache from the whole prompt"
  },
  {
    "startMs": 1351990,
    "text": "commits KV cache from the whole prompt"
  },
  {
    "startMs": 1352000,
    "text": "commits KV cache from the whole prompt and the request joins the running batch"
  },
  {
    "startMs": 1354310,
    "text": "and the request joins the running batch"
  },
  {
    "startMs": 1354320,
    "text": "and the request joins the running batch and decode one token per step alongside"
  },
  {
    "startMs": 1357510,
    "text": "and decode one token per step alongside"
  },
  {
    "startMs": 1357520,
    "text": "and decode one token per step alongside everyone else"
  },
  {
    "startMs": 1360149,
    "text": "everyone else"
  },
  {
    "startMs": 1360159,
    "text": "everyone else and it holds that memory until it"
  },
  {
    "startMs": 1362470,
    "text": "and it holds that memory until it"
  },
  {
    "startMs": 1362480,
    "text": "and it holds that memory until it finishes or until the client"
  },
  {
    "startMs": 1364310,
    "text": "finishes or until the client"
  },
  {
    "startMs": 1364320,
    "text": "finishes or until the client disconnects."
  },
  {
    "startMs": 1366149,
    "text": "disconnects."
  },
  {
    "startMs": 1366159,
    "text": "disconnects. That's the commitment from part one just"
  },
  {
    "startMs": 1369510,
    "text": "That's the commitment from part one just"
  },
  {
    "startMs": 1369520,
    "text": "That's the commitment from part one just in the form of an HTTP response which is"
  },
  {
    "startMs": 1372230,
    "text": "in the form of an HTTP response which is"
  },
  {
    "startMs": 1372240,
    "text": "in the form of an HTTP response which is why the admission decision has to happen"
  },
  {
    "startMs": 1374870,
    "text": "why the admission decision has to happen"
  },
  {
    "startMs": 1374880,
    "text": "why the admission decision has to happen before 200 never after it."
  },
  {
    "startMs": 1379830,
    "text": "before 200 never after it."
  },
  {
    "startMs": 1379840,
    "text": "before 200 never after it. Now the client side if we are writing"
  },
  {
    "startMs": 1382549,
    "text": "Now the client side if we are writing"
  },
  {
    "startMs": 1382559,
    "text": "Now the client side if we are writing the agent here is how to handle 200."
  },
  {
    "startMs": 1386230,
    "text": "the agent here is how to handle 200."
  },
  {
    "startMs": 1386240,
    "text": "the agent here is how to handle 200. Watch the stream not the status code."
  },
  {
    "startMs": 1389990,
    "text": "Watch the stream not the status code."
  },
  {
    "startMs": 1390000,
    "text": "Watch the stream not the status code. First time is properly start your"
  },
  {
    "startMs": 1393430,
    "text": "First time is properly start your"
  },
  {
    "startMs": 1393440,
    "text": "First time is properly start your timeout clock when the first token"
  },
  {
    "startMs": 1395830,
    "text": "timeout clock when the first token"
  },
  {
    "startMs": 1395840,
    "text": "timeout clock when the first token arrives not when you send the request"
  },
  {
    "startMs": 1398710,
    "text": "arrives not when you send the request"
  },
  {
    "startMs": 1398720,
    "text": "arrives not when you send the request under the heavy load prefill alone can"
  },
  {
    "startMs": 1401029,
    "text": "under the heavy load prefill alone can"
  },
  {
    "startMs": 1401039,
    "text": "under the heavy load prefill alone can take few seconds. Second, if you decide"
  },
  {
    "startMs": 1404310,
    "text": "take few seconds. Second, if you decide"
  },
  {
    "startMs": 1404320,
    "text": "take few seconds. Second, if you decide to give up, actually close the"
  },
  {
    "startMs": 1406149,
    "text": "to give up, actually close the"
  },
  {
    "startMs": 1406159,
    "text": "to give up, actually close the connection. The server only frees the KV"
  },
  {
    "startMs": 1409110,
    "text": "connection. The server only frees the KV"
  },
  {
    "startMs": 1409120,
    "text": "connection. The server only frees the KV cache when the request finishes or the"
  },
  {
    "startMs": 1411510,
    "text": "cache when the request finishes or the"
  },
  {
    "startMs": 1411520,
    "text": "cache when the request finishes or the client disconnects. An agent that times"
  },
  {
    "startMs": 1414390,
    "text": "client disconnects. An agent that times"
  },
  {
    "startMs": 1414400,
    "text": "client disconnects. An agent that times out but leaves the connection open is"
  },
  {
    "startMs": 1416789,
    "text": "out but leaves the connection open is"
  },
  {
    "startMs": 1416799,
    "text": "out but leaves the connection open is still holding the GPU memory. And the"
  },
  {
    "startMs": 1419909,
    "text": "still holding the GPU memory. And the"
  },
  {
    "startMs": 1419919,
    "text": "still holding the GPU memory. And the third one, uh, a stream can fail halfway"
  },
  {
    "startMs": 1423110,
    "text": "third one, uh, a stream can fail halfway"
  },
  {
    "startMs": 1423120,
    "text": "third one, uh, a stream can fail halfway through, right? And that that's no"
  },
  {
    "startMs": 1426630,
    "text": "through, right? And that that's no"
  },
  {
    "startMs": 1426640,
    "text": "through, right? And that that's no status for that, right? and you're"
  },
  {
    "startMs": 1429350,
    "text": "status for that, right? and you're"
  },
  {
    "startMs": 1429360,
    "text": "status for that, right? and you're you're left with partial answer and you"
  },
  {
    "startMs": 1431590,
    "text": "you're left with partial answer and you"
  },
  {
    "startMs": 1431600,
    "text": "you're left with partial answer and you have to decide whether to retry or carry"
  },
  {
    "startMs": 1433750,
    "text": "have to decide whether to retry or carry"
  },
  {
    "startMs": 1433760,
    "text": "have to decide whether to retry or carry on. In short, 200 promise to start. It's"
  },
  {
    "startMs": 1438390,
    "text": "on. In short, 200 promise to start. It's"
  },
  {
    "startMs": 1438400,
    "text": "on. In short, 200 promise to start. It's not a promise to finish. Right?"
  },
  {
    "startMs": 1443029,
    "text": "not a promise to finish. Right?"
  },
  {
    "startMs": 1443039,
    "text": "not a promise to finish. Right? The second answer is dipper which"
  },
  {
    "startMs": 1445110,
    "text": "The second answer is dipper which"
  },
  {
    "startMs": 1445120,
    "text": "The second answer is dipper which basically means not yet. The server"
  },
  {
    "startMs": 1447350,
    "text": "basically means not yet. The server"
  },
  {
    "startMs": 1447360,
    "text": "basically means not yet. The server sends back 503 with retry header."
  },
  {
    "startMs": 1451190,
    "text": "sends back 503 with retry header."
  },
  {
    "startMs": 1451200,
    "text": "sends back 503 with retry header. The header tells the client how many"
  },
  {
    "startMs": 1453269,
    "text": "The header tells the client how many"
  },
  {
    "startMs": 1453279,
    "text": "The header tells the client how many seconds to wait. The policy picks that"
  },
  {
    "startMs": 1455750,
    "text": "seconds to wait. The policy picks that"
  },
  {
    "startMs": 1455760,
    "text": "seconds to wait. The policy picks that number from the engine. For example, how"
  },
  {
    "startMs": 1458149,
    "text": "number from the engine. For example, how"
  },
  {
    "startMs": 1458159,
    "text": "number from the engine. For example, how long until the running batch finishes"
  },
  {
    "startMs": 1459990,
    "text": "long until the running batch finishes"
  },
  {
    "startMs": 1460000,
    "text": "long until the running batch finishes and freeze the enough cash"
  },
  {
    "startMs": 1462789,
    "text": "and freeze the enough cash"
  },
  {
    "startMs": 1462799,
    "text": "and freeze the enough cash and you differ when capacity is coming"
  },
  {
    "startMs": 1465029,
    "text": "and you differ when capacity is coming"
  },
  {
    "startMs": 1465039,
    "text": "and you differ when capacity is coming back soon? It's a burst, not so not a"
  },
  {
    "startMs": 1467590,
    "text": "back soon? It's a burst, not so not a"
  },
  {
    "startMs": 1467600,
    "text": "back soon? It's a burst, not so not a trend, right? And for example, like if"
  },
  {
    "startMs": 1470470,
    "text": "trend, right? And for example, like if"
  },
  {
    "startMs": 1470480,
    "text": "trend, right? And for example, like if you think about it like Q age is rising"
  },
  {
    "startMs": 1474789,
    "text": "you think about it like Q age is rising"
  },
  {
    "startMs": 1474799,
    "text": "you think about it like Q age is rising but the system isn't saturated."
  },
  {
    "startMs": 1479190,
    "text": "but the system isn't saturated."
  },
  {
    "startMs": 1479200,
    "text": "but the system isn't saturated. If the capacity isn't coming back"
  },
  {
    "startMs": 1481269,
    "text": "If the capacity isn't coming back"
  },
  {
    "startMs": 1481279,
    "text": "If the capacity isn't coming back anytime soon, a dipper just kicks kicks"
  },
  {
    "startMs": 1484870,
    "text": "anytime soon, a dipper just kicks kicks"
  },
  {
    "startMs": 1484880,
    "text": "anytime soon, a dipper just kicks kicks the can down the road, right? And in"
  },
  {
    "startMs": 1488390,
    "text": "the can down the road, right? And in"
  },
  {
    "startMs": 1488400,
    "text": "the can down the road, right? And in that case, we can just reject it with"
  },
  {
    "startMs": 1490630,
    "text": "that case, we can just reject it with"
  },
  {
    "startMs": 1490640,
    "text": "that case, we can just reject it with the honest answer."
  },
  {
    "startMs": 1493909,
    "text": "the honest answer."
  },
  {
    "startMs": 1493919,
    "text": "the honest answer. Moving on to the next slide. On the"
  },
  {
    "startMs": 1495909,
    "text": "Moving on to the next slide. On the"
  },
  {
    "startMs": 1495919,
    "text": "Moving on to the next slide. On the agent side, handling a defer is simple."
  },
  {
    "startMs": 1499590,
    "text": "agent side, handling a defer is simple."
  },
  {
    "startMs": 1499600,
    "text": "agent side, handling a defer is simple. Sleep for retry after time and a little"
  },
  {
    "startMs": 1501990,
    "text": "Sleep for retry after time and a little"
  },
  {
    "startMs": 1502000,
    "text": "Sleep for retry after time and a little random jitter, then retry again. The"
  },
  {
    "startMs": 1505110,
    "text": "random jitter, then retry again. The"
  },
  {
    "startMs": 1505120,
    "text": "random jitter, then retry again. The jitter just means a small random extra"
  },
  {
    "startMs": 1507430,
    "text": "jitter just means a small random extra"
  },
  {
    "startMs": 1507440,
    "text": "jitter just means a small random extra delay. So client don't like don't call"
  },
  {
    "startMs": 1511669,
    "text": "delay. So client don't like don't call"
  },
  {
    "startMs": 1511679,
    "text": "delay. So client don't like don't call or like don't at all come back at the"
  },
  {
    "startMs": 1514310,
    "text": "or like don't at all come back at the"
  },
  {
    "startMs": 1514320,
    "text": "or like don't at all come back at the exact same moment. They will have to"
  },
  {
    "startMs": 1516630,
    "text": "exact same moment. They will have to"
  },
  {
    "startMs": 1516640,
    "text": "exact same moment. They will have to wait."
  },
  {
    "startMs": 1519110,
    "text": "wait."
  },
  {
    "startMs": 1519120,
    "text": "wait. Look what the server sees from 10"
  },
  {
    "startMs": 1521590,
    "text": "Look what the server sees from 10"
  },
  {
    "startMs": 1521600,
    "text": "Look what the server sees from 10 retries without retry after every agent"
  },
  {
    "startMs": 1525830,
    "text": "retries without retry after every agent"
  },
  {
    "startMs": 1525840,
    "text": "retries without retry after every agent picks its own delay. And that's usually"
  },
  {
    "startMs": 1529669,
    "text": "picks its own delay. And that's usually"
  },
  {
    "startMs": 1529679,
    "text": "picks its own delay. And that's usually that's usually zero. So they all come"
  },
  {
    "startMs": 1532390,
    "text": "that's usually zero. So they all come"
  },
  {
    "startMs": 1532400,
    "text": "that's usually zero. So they all come back at once. That's how 40 calls turns"
  },
  {
    "startMs": 1535590,
    "text": "back at once. That's how 40 calls turns"
  },
  {
    "startMs": 1535600,
    "text": "back at once. That's how 40 calls turns into 80."
  },
  {
    "startMs": 1538549,
    "text": "into 80."
  },
  {
    "startMs": 1538559,
    "text": "into 80. With retry after and jitter the retry"
  },
  {
    "startMs": 1541669,
    "text": "With retry after and jitter the retry"
  },
  {
    "startMs": 1541679,
    "text": "With retry after and jitter the retry spreads out across the window serves"
  },
  {
    "startMs": 1543750,
    "text": "spreads out across the window serves"
  },
  {
    "startMs": 1543760,
    "text": "spreads out across the window serves asked for."
  },
  {
    "startMs": 1550640,
    "text": "Okay. So if if we see the uh reject"
  },
  {
    "startMs": 1555909,
    "text": "Okay. So if if we see the uh reject"
  },
  {
    "startMs": 1555919,
    "text": "Okay. So if if we see the uh reject reject returns 429 or 529 the third"
  },
  {
    "startMs": 1559190,
    "text": "reject returns 429 or 529 the third"
  },
  {
    "startMs": 1559200,
    "text": "reject returns 429 or 529 the third answer is no."
  },
  {
    "startMs": 1561830,
    "text": "answer is no."
  },
  {
    "startMs": 1561840,
    "text": "answer is no. Uh and there there are two different"
  },
  {
    "startMs": 1563750,
    "text": "Uh and there there are two different"
  },
  {
    "startMs": 1563760,
    "text": "Uh and there there are two different codes for it because there are two"
  },
  {
    "startMs": 1565909,
    "text": "codes for it because there are two"
  },
  {
    "startMs": 1565919,
    "text": "codes for it because there are two different reasons. A 429s means too many"
  },
  {
    "startMs": 1568870,
    "text": "different reasons. A 429s means too many"
  },
  {
    "startMs": 1568880,
    "text": "different reasons. A 429s means too many requests."
  },
  {
    "startMs": 1570950,
    "text": "requests."
  },
  {
    "startMs": 1570960,
    "text": "requests. This particular client or agent session"
  },
  {
    "startMs": 1573269,
    "text": "This particular client or agent session"
  },
  {
    "startMs": 1573279,
    "text": "This particular client or agent session is using more than its share. The policy"
  },
  {
    "startMs": 1576390,
    "text": "is using more than its share. The policy"
  },
  {
    "startMs": 1576400,
    "text": "is using more than its share. The policy saw something like per client load, the"
  },
  {
    "startMs": 1578710,
    "text": "saw something like per client load, the"
  },
  {
    "startMs": 1578720,
    "text": "saw something like per client load, the number of concurrent calls or a kota."
  },
  {
    "startMs": 1581669,
    "text": "number of concurrent calls or a kota."
  },
  {
    "startMs": 1581679,
    "text": "number of concurrent calls or a kota. The problem is the caller and then the"
  },
  {
    "startMs": 1584870,
    "text": "The problem is the caller and then the"
  },
  {
    "startMs": 1584880,
    "text": "The problem is the caller and then the 529 means the provider is overloaded."
  },
  {
    "startMs": 1588070,
    "text": "529 means the provider is overloaded."
  },
  {
    "startMs": 1588080,
    "text": "529 means the provider is overloaded. Enthropic uses this code. Every replica"
  },
  {
    "startMs": 1591269,
    "text": "Enthropic uses this code. Every replica"
  },
  {
    "startMs": 1591279,
    "text": "Enthropic uses this code. Every replica is sold. So nobody is getting in right"
  },
  {
    "startMs": 1594390,
    "text": "is sold. So nobody is getting in right"
  },
  {
    "startMs": 1594400,
    "text": "is sold. So nobody is getting in right now."
  },
  {
    "startMs": 1596630,
    "text": "now."
  },
  {
    "startMs": 1596640,
    "text": "now. The problem is pool not the caller."
  },
  {
    "startMs": 1600789,
    "text": "The problem is pool not the caller."
  },
  {
    "startMs": 1600799,
    "text": "The problem is pool not the caller. Now uh handling a reject. So how should"
  },
  {
    "startMs": 1604710,
    "text": "Now uh handling a reject. So how should"
  },
  {
    "startMs": 1604720,
    "text": "Now uh handling a reject. So how should agent react on 429? Reduce your own"
  },
  {
    "startMs": 1608230,
    "text": "agent react on 429? Reduce your own"
  },
  {
    "startMs": 1608240,
    "text": "agent react on 429? Reduce your own load. Cap how many calls that agent has"
  },
  {
    "startMs": 1611190,
    "text": "load. Cap how many calls that agent has"
  },
  {
    "startMs": 1611200,
    "text": "load. Cap how many calls that agent has in flight at once."
  },
  {
    "startMs": 1613990,
    "text": "in flight at once."
  },
  {
    "startMs": 1614000,
    "text": "in flight at once. And on 529 retrying harder only makes"
  },
  {
    "startMs": 1617669,
    "text": "And on 529 retrying harder only makes"
  },
  {
    "startMs": 1617679,
    "text": "And on 529 retrying harder only makes this worse. Waiting longer switching to"
  },
  {
    "startMs": 1620390,
    "text": "this worse. Waiting longer switching to"
  },
  {
    "startMs": 1620400,
    "text": "this worse. Waiting longer switching to different model provider or park the"
  },
  {
    "startMs": 1622549,
    "text": "different model provider or park the"
  },
  {
    "startMs": 1622559,
    "text": "different model provider or park the task for later. and one either and on"
  },
  {
    "startMs": 1627269,
    "text": "task for later. and one either and on"
  },
  {
    "startMs": 1627279,
    "text": "task for later. and one either and on either one keep a retry budget. Use"
  },
  {
    "startMs": 1629830,
    "text": "either one keep a retry budget. Use"
  },
  {
    "startMs": 1629840,
    "text": "either one keep a retry budget. Use expansion exponential back off with"
  },
  {
    "startMs": 1632470,
    "text": "expansion exponential back off with"
  },
  {
    "startMs": 1632480,
    "text": "expansion exponential back off with jitter which means waiting longer after"
  },
  {
    "startMs": 1635350,
    "text": "jitter which means waiting longer after"
  },
  {
    "startMs": 1635360,
    "text": "jitter which means waiting longer after each failed try and after few tries tell"
  },
  {
    "startMs": 1639510,
    "text": "each failed try and after few tries tell"
  },
  {
    "startMs": 1639520,
    "text": "each failed try and after few tries tell the user what's going on because a"
  },
  {
    "startMs": 1641350,
    "text": "the user what's going on because a"
  },
  {
    "startMs": 1641360,
    "text": "the user what's going on because a reject that gets retrieded at full speed"
  },
  {
    "startMs": 1643990,
    "text": "reject that gets retrieded at full speed"
  },
  {
    "startMs": 1644000,
    "text": "reject that gets retrieded at full speed is how you end up in one of the postmodm"
  },
  {
    "startMs": 1646950,
    "text": "is how you end up in one of the postmodm"
  },
  {
    "startMs": 1646960,
    "text": "is how you end up in one of the postmodm from the talk at we described at the"
  },
  {
    "startMs": 1649830,
    "text": "from the talk at we described at the"
  },
  {
    "startMs": 1649840,
    "text": "from the talk at we described at the starting."
  },
  {
    "startMs": 1653760,
    "text": "Now over to you Hershel"
  },
  {
    "startMs": 1660000,
    "text": "three."
  },
  {
    "startMs": 1661510,
    "text": "three."
  },
  {
    "startMs": 1661520,
    "text": "three. Thank you P. Uh so this is the module"
  },
  {
    "startMs": 1663990,
    "text": "Thank you P. Uh so this is the module"
  },
  {
    "startMs": 1664000,
    "text": "Thank you P. Uh so this is the module three where we will try to learn about"
  },
  {
    "startMs": 1666149,
    "text": "three where we will try to learn about"
  },
  {
    "startMs": 1666159,
    "text": "three where we will try to learn about the different strategies uh that have"
  },
  {
    "startMs": 1668549,
    "text": "the different strategies uh that have"
  },
  {
    "startMs": 1668559,
    "text": "the different strategies uh that have been rolled out for the admission"
  },
  {
    "startMs": 1670310,
    "text": "been rolled out for the admission"
  },
  {
    "startMs": 1670320,
    "text": "been rolled out for the admission control. So far we have seen that no"
  },
  {
    "startMs": 1672950,
    "text": "control. So far we have seen that no"
  },
  {
    "startMs": 1672960,
    "text": "control. So far we have seen that no layer in the today's stack asks where uh"
  },
  {
    "startMs": 1675830,
    "text": "layer in the today's stack asks where uh"
  },
  {
    "startMs": 1675840,
    "text": "layer in the today's stack asks where uh whether there is a room and that layer"
  },
  {
    "startMs": 1678389,
    "text": "whether there is a room and that layer"
  },
  {
    "startMs": 1678399,
    "text": "whether there is a room and that layer is ours to add and now we give it a"
  },
  {
    "startMs": 1681110,
    "text": "is ours to add and now we give it a"
  },
  {
    "startMs": 1681120,
    "text": "is ours to add and now we give it a brain. So we will look at why counting"
  },
  {
    "startMs": 1683909,
    "text": "brain. So we will look at why counting"
  },
  {
    "startMs": 1683919,
    "text": "brain. So we will look at why counting requests isn't enough. Breaking a policy"
  },
  {
    "startMs": 1686149,
    "text": "requests isn't enough. Breaking a policy"
  },
  {
    "startMs": 1686159,
    "text": "requests isn't enough. Breaking a policy down into four different choices. See"
  },
  {
    "startMs": 1688710,
    "text": "down into four different choices. See"
  },
  {
    "startMs": 1688720,
    "text": "down into four different choices. See what four different research paper chose"
  },
  {
    "startMs": 1692230,
    "text": "what four different research paper chose"
  },
  {
    "startMs": 1692240,
    "text": "what four different research paper chose and finish with a decision tree that you"
  },
  {
    "startMs": 1695110,
    "text": "and finish with a decision tree that you"
  },
  {
    "startMs": 1695120,
    "text": "and finish with a decision tree that you can use onto your own ecosystem."
  },
  {
    "startMs": 1700320,
    "text": "So let's start with what most team"
  },
  {
    "startMs": 1702950,
    "text": "So let's start with what most team"
  },
  {
    "startMs": 1702960,
    "text": "So let's start with what most team already have which is counting. Yes."
  },
  {
    "startMs": 1706710,
    "text": "already have which is counting. Yes."
  },
  {
    "startMs": 1706720,
    "text": "already have which is counting. Yes. Counting the number of the requests."
  },
  {
    "startMs": 1710149,
    "text": "Counting the number of the requests."
  },
  {
    "startMs": 1710159,
    "text": "Counting the number of the requests. A simple idea is to rate limit at the"
  },
  {
    "startMs": 1712789,
    "text": "A simple idea is to rate limit at the"
  },
  {
    "startMs": 1712799,
    "text": "A simple idea is to rate limit at the gateway which gives API key a fixed"
  },
  {
    "startMs": 1715430,
    "text": "gateway which gives API key a fixed"
  },
  {
    "startMs": 1715440,
    "text": "gateway which gives API key a fixed allowance set in advance. It's great for"
  },
  {
    "startMs": 1718470,
    "text": "allowance set in advance. It's great for"
  },
  {
    "startMs": 1718480,
    "text": "allowance set in advance. It's great for keeping the clients fair, keeping your"
  },
  {
    "startMs": 1720549,
    "text": "keeping the clients fair, keeping your"
  },
  {
    "startMs": 1720559,
    "text": "keeping the clients fair, keeping your tenants fair with each other,"
  },
  {
    "startMs": 1723430,
    "text": "tenants fair with each other,"
  },
  {
    "startMs": 1723440,
    "text": "tenants fair with each other, but it does nothing to protect your GPUs"
  },
  {
    "startMs": 1727750,
    "text": "but it does nothing to protect your GPUs"
  },
  {
    "startMs": 1727760,
    "text": "but it does nothing to protect your GPUs because it doesn't change with the load"
  },
  {
    "startMs": 1730149,
    "text": "because it doesn't change with the load"
  },
  {
    "startMs": 1730159,
    "text": "because it doesn't change with the load that is happening onto your GPUs or your"
  },
  {
    "startMs": 1732870,
    "text": "that is happening onto your GPUs or your"
  },
  {
    "startMs": 1732880,
    "text": "that is happening onto your GPUs or your cluster. A fixed cap at the proxy gets"
  },
  {
    "startMs": 1737350,
    "text": "cluster. A fixed cap at the proxy gets"
  },
  {
    "startMs": 1737360,
    "text": "cluster. A fixed cap at the proxy gets closer. It counts how many requests are"
  },
  {
    "startMs": 1739990,
    "text": "closer. It counts how many requests are"
  },
  {
    "startMs": 1740000,
    "text": "closer. It counts how many requests are in flight on each replica. The problem"
  },
  {
    "startMs": 1743029,
    "text": "in flight on each replica. The problem"
  },
  {
    "startMs": 1743039,
    "text": "in flight on each replica. The problem is that the quick crep and a long"
  },
  {
    "startMs": 1745909,
    "text": "is that the quick crep and a long"
  },
  {
    "startMs": 1745919,
    "text": "is that the quick crep and a long planning to call"
  },
  {
    "startMs": 1748230,
    "text": "planning to call"
  },
  {
    "startMs": 1748240,
    "text": "planning to call count exactly the same while the real"
  },
  {
    "startMs": 1750789,
    "text": "count exactly the same while the real"
  },
  {
    "startMs": 1750799,
    "text": "count exactly the same while the real capacity of the GPU changes with every"
  },
  {
    "startMs": 1752870,
    "text": "capacity of the GPU changes with every"
  },
  {
    "startMs": 1752880,
    "text": "capacity of the GPU changes with every batch. So in practice it mostly cues and"
  },
  {
    "startMs": 1757510,
    "text": "batch. So in practice it mostly cues and"
  },
  {
    "startMs": 1757520,
    "text": "batch. So in practice it mostly cues and clients timeout ends up doing the"
  },
  {
    "startMs": 1760149,
    "text": "clients timeout ends up doing the"
  },
  {
    "startMs": 1760159,
    "text": "clients timeout ends up doing the refusing."
  },
  {
    "startMs": 1762630,
    "text": "refusing."
  },
  {
    "startMs": 1762640,
    "text": "refusing. Neither one knows whether our 40 calls"
  },
  {
    "startMs": 1765269,
    "text": "Neither one knows whether our 40 calls"
  },
  {
    "startMs": 1765279,
    "text": "Neither one knows whether our 40 calls are cheap or expensive."
  },
  {
    "startMs": 1767830,
    "text": "are cheap or expensive."
  },
  {
    "startMs": 1767840,
    "text": "are cheap or expensive. The cost of air policy does though and"
  },
  {
    "startMs": 1771269,
    "text": "The cost of air policy does though and"
  },
  {
    "startMs": 1771279,
    "text": "The cost of air policy does though and it estimates each request will actually"
  },
  {
    "startMs": 1775750,
    "text": "it estimates each request will actually"
  },
  {
    "startMs": 1775760,
    "text": "it estimates each request will actually cost what each request will cost in the"
  },
  {
    "startMs": 1779269,
    "text": "cost what each request will cost in the"
  },
  {
    "startMs": 1779279,
    "text": "cost what each request will cost in the KV memory and the tokens using the live"
  },
  {
    "startMs": 1782070,
    "text": "KV memory and the tokens using the live"
  },
  {
    "startMs": 1782080,
    "text": "KV memory and the tokens using the live state from your engine. And it says no"
  },
  {
    "startMs": 1785350,
    "text": "state from your engine. And it says no"
  },
  {
    "startMs": 1785360,
    "text": "state from your engine. And it says no before the request actually takes up the"
  },
  {
    "startMs": 1787750,
    "text": "before the request actually takes up the"
  },
  {
    "startMs": 1787760,
    "text": "before the request actually takes up the memory. So the short version is count is"
  },
  {
    "startMs": 1790630,
    "text": "memory. So the short version is count is"
  },
  {
    "startMs": 1790640,
    "text": "memory. So the short version is count is not a cost."
  },
  {
    "startMs": 1795039,
    "text": "How do you build your own policy? So"
  },
  {
    "startMs": 1797190,
    "text": "How do you build your own policy? So"
  },
  {
    "startMs": 1797200,
    "text": "How do you build your own policy? So there are four choices here. The first"
  },
  {
    "startMs": 1799990,
    "text": "there are four choices here. The first"
  },
  {
    "startMs": 1800000,
    "text": "there are four choices here. The first is the unit"
  },
  {
    "startMs": 1802149,
    "text": "is the unit"
  },
  {
    "startMs": 1802159,
    "text": "is the unit which tells you what you are actually"
  },
  {
    "startMs": 1803990,
    "text": "which tells you what you are actually"
  },
  {
    "startMs": 1804000,
    "text": "which tells you what you are actually admitting. Are you admitting a single"
  },
  {
    "startMs": 1806070,
    "text": "admitting. Are you admitting a single"
  },
  {
    "startMs": 1806080,
    "text": "admitting. Are you admitting a single request? Are you admitting an agent"
  },
  {
    "startMs": 1809029,
    "text": "request? Are you admitting an agent"
  },
  {
    "startMs": 1809039,
    "text": "request? Are you admitting an agent session or are you admitting a tenant?"
  },
  {
    "startMs": 1813430,
    "text": "session or are you admitting a tenant?"
  },
  {
    "startMs": 1813440,
    "text": "session or are you admitting a tenant? The two second is the signal"
  },
  {
    "startMs": 1817269,
    "text": "The two second is the signal"
  },
  {
    "startMs": 1817279,
    "text": "The two second is the signal which means what is that the policy is"
  },
  {
    "startMs": 1820389,
    "text": "which means what is that the policy is"
  },
  {
    "startMs": 1820399,
    "text": "which means what is that the policy is watching for taking an admission"
  },
  {
    "startMs": 1822549,
    "text": "watching for taking an admission"
  },
  {
    "startMs": 1822559,
    "text": "watching for taking an admission decision. This could be as simple as"
  },
  {
    "startMs": 1824950,
    "text": "decision. This could be as simple as"
  },
  {
    "startMs": 1824960,
    "text": "decision. This could be as simple as your memory pressure which is your KB"
  },
  {
    "startMs": 1827110,
    "text": "your memory pressure which is your KB"
  },
  {
    "startMs": 1827120,
    "text": "your memory pressure which is your KB pressure. It could be your Q depth. It"
  },
  {
    "startMs": 1829750,
    "text": "pressure. It could be your Q depth. It"
  },
  {
    "startMs": 1829760,
    "text": "pressure. It could be your Q depth. It could be your deadline slack. That means"
  },
  {
    "startMs": 1832549,
    "text": "could be your deadline slack. That means"
  },
  {
    "startMs": 1832559,
    "text": "could be your deadline slack. That means whether your request is able to meet"
  },
  {
    "startMs": 1834710,
    "text": "whether your request is able to meet"
  },
  {
    "startMs": 1834720,
    "text": "whether your request is able to meet your SLOs's or not."
  },
  {
    "startMs": 1837750,
    "text": "your SLOs's or not."
  },
  {
    "startMs": 1837760,
    "text": "your SLOs's or not. And whatever it is, it has to actually"
  },
  {
    "startMs": 1839909,
    "text": "And whatever it is, it has to actually"
  },
  {
    "startMs": 1839919,
    "text": "And whatever it is, it has to actually move when the system gets busy. So three"
  },
  {
    "startMs": 1843430,
    "text": "move when the system gets busy. So three"
  },
  {
    "startMs": 1843440,
    "text": "move when the system gets busy. So three is the rule which means how does the"
  },
  {
    "startMs": 1846470,
    "text": "is the rule which means how does the"
  },
  {
    "startMs": 1846480,
    "text": "is the rule which means how does the signal gets turned into a decision."
  },
  {
    "startMs": 1850710,
    "text": "signal gets turned into a decision."
  },
  {
    "startMs": 1850720,
    "text": "signal gets turned into a decision. So rule is used for that and that can be"
  },
  {
    "startMs": 1854310,
    "text": "So rule is used for that and that can be"
  },
  {
    "startMs": 1854320,
    "text": "So rule is used for that and that can be anything from a simple threshold that"
  },
  {
    "startMs": 1856470,
    "text": "anything from a simple threshold that"
  },
  {
    "startMs": 1856480,
    "text": "anything from a simple threshold that takes microscond to a full optimizer"
  },
  {
    "startMs": 1858789,
    "text": "takes microscond to a full optimizer"
  },
  {
    "startMs": 1858799,
    "text": "takes microscond to a full optimizer that takes seconds."
  },
  {
    "startMs": 1861269,
    "text": "that takes seconds."
  },
  {
    "startMs": 1861279,
    "text": "that takes seconds. Fourth is the action. Once you have the"
  },
  {
    "startMs": 1863990,
    "text": "Fourth is the action. Once you have the"
  },
  {
    "startMs": 1864000,
    "text": "Fourth is the action. Once you have the rule that gives you a decision what do"
  },
  {
    "startMs": 1866789,
    "text": "rule that gives you a decision what do"
  },
  {
    "startMs": 1866799,
    "text": "rule that gives you a decision what do you do with it?"
  },
  {
    "startMs": 1869110,
    "text": "you do with it?"
  },
  {
    "startMs": 1869120,
    "text": "you do with it? What does your how does your client gets"
  },
  {
    "startMs": 1871590,
    "text": "What does your how does your client gets"
  },
  {
    "startMs": 1871600,
    "text": "What does your how does your client gets impacted through those actions?"
  },
  {
    "startMs": 1874710,
    "text": "impacted through those actions?"
  },
  {
    "startMs": 1874720,
    "text": "impacted through those actions? That's the admit defer and the reject"
  },
  {
    "startMs": 1877590,
    "text": "That's the admit defer and the reject"
  },
  {
    "startMs": 1877600,
    "text": "That's the admit defer and the reject that per just talked about in the module"
  },
  {
    "startMs": 1880149,
    "text": "that per just talked about in the module"
  },
  {
    "startMs": 1880159,
    "text": "that per just talked about in the module two. Each choice limits the next one."
  },
  {
    "startMs": 1884389,
    "text": "two. Each choice limits the next one."
  },
  {
    "startMs": 1884399,
    "text": "two. Each choice limits the next one. For example, a deadline signal only"
  },
  {
    "startMs": 1886389,
    "text": "For example, a deadline signal only"
  },
  {
    "startMs": 1886399,
    "text": "For example, a deadline signal only works if your request actually carry the"
  },
  {
    "startMs": 1888950,
    "text": "works if your request actually carry the"
  },
  {
    "startMs": 1888960,
    "text": "works if your request actually carry the deadlines. So let's go through the first"
  },
  {
    "startMs": 1891830,
    "text": "deadlines. So let's go through the first"
  },
  {
    "startMs": 1891840,
    "text": "deadlines. So let's go through the first three."
  },
  {
    "startMs": 1897600,
    "text": "Choice one. Do you admit a single"
  },
  {
    "startMs": 1899830,
    "text": "Choice one. Do you admit a single"
  },
  {
    "startMs": 1899840,
    "text": "Choice one. Do you admit a single request or do you admit a poll sessions?"
  },
  {
    "startMs": 1904230,
    "text": "request or do you admit a poll sessions?"
  },
  {
    "startMs": 1904240,
    "text": "request or do you admit a poll sessions? Say you have two agent task 20 calls"
  },
  {
    "startMs": 1906630,
    "text": "Say you have two agent task 20 calls"
  },
  {
    "startMs": 1906640,
    "text": "Say you have two agent task 20 calls each room for almost half of them."
  },
  {
    "startMs": 1910310,
    "text": "each room for almost half of them."
  },
  {
    "startMs": 1910320,
    "text": "each room for almost half of them. You can uh point you can see the top"
  },
  {
    "startMs": 1914149,
    "text": "You can uh point you can see the top"
  },
  {
    "startMs": 1914159,
    "text": "You can uh point you can see the top row. So if you admit per request about"
  },
  {
    "startMs": 1917509,
    "text": "row. So if you admit per request about"
  },
  {
    "startMs": 1917519,
    "text": "row. So if you admit per request about half of each task gets in but neither"
  },
  {
    "startMs": 1920870,
    "text": "half of each task gets in but neither"
  },
  {
    "startMs": 1920880,
    "text": "half of each task gets in but neither task can finish because each one is"
  },
  {
    "startMs": 1923350,
    "text": "task can finish because each one is"
  },
  {
    "startMs": 1923360,
    "text": "task can finish because each one is stuck waiting on the calls that didn't"
  },
  {
    "startMs": 1925509,
    "text": "stuck waiting on the calls that didn't"
  },
  {
    "startMs": 1925519,
    "text": "stuck waiting on the calls that didn't make it. You have used all the capacity"
  },
  {
    "startMs": 1928950,
    "text": "make it. You have used all the capacity"
  },
  {
    "startMs": 1928960,
    "text": "make it. You have used all the capacity and completed nothing. If you admit per"
  },
  {
    "startMs": 1932149,
    "text": "and completed nothing. If you admit per"
  },
  {
    "startMs": 1932159,
    "text": "and completed nothing. If you admit per session, session A gets all of its call"
  },
  {
    "startMs": 1935029,
    "text": "session, session A gets all of its call"
  },
  {
    "startMs": 1935039,
    "text": "session, session A gets all of its call and finishes. Session B gets one clear"
  },
  {
    "startMs": 1937590,
    "text": "and finishes. Session B gets one clear"
  },
  {
    "startMs": 1937600,
    "text": "and finishes. Session B gets one clear answer. Come back later. And this"
  },
  {
    "startMs": 1940310,
    "text": "answer. Come back later. And this"
  },
  {
    "startMs": 1940320,
    "text": "answer. Come back later. And this picture is illustrative, but the logic"
  },
  {
    "startMs": 1942470,
    "text": "picture is illustrative, but the logic"
  },
  {
    "startMs": 1942480,
    "text": "picture is illustrative, but the logic holds."
  },
  {
    "startMs": 1944070,
    "text": "holds."
  },
  {
    "startMs": 1944080,
    "text": "holds. Our 40 calls are one session. Coding"
  },
  {
    "startMs": 1947110,
    "text": "Our 40 calls are one session. Coding"
  },
  {
    "startMs": 1947120,
    "text": "Our 40 calls are one session. Coding agents send sessions and yet most"
  },
  {
    "startMs": 1950230,
    "text": "agents send sessions and yet most"
  },
  {
    "startMs": 1950240,
    "text": "agents send sessions and yet most published policies admit only the single"
  },
  {
    "startMs": 1953350,
    "text": "published policies admit only the single"
  },
  {
    "startMs": 1953360,
    "text": "published policies admit only the single requests."
  },
  {
    "startMs": 1958080,
    "text": "So which metric do you watch for what"
  },
  {
    "startMs": 1960870,
    "text": "So which metric do you watch for what"
  },
  {
    "startMs": 1960880,
    "text": "So which metric do you watch for what particular problem?"
  },
  {
    "startMs": 1964559,
    "text": "This is the"
  },
  {
    "startMs": 1966630,
    "text": "This is the"
  },
  {
    "startMs": 1966640,
    "text": "This is the question that we are trying to answer."
  },
  {
    "startMs": 1969909,
    "text": "question that we are trying to answer."
  },
  {
    "startMs": 1969919,
    "text": "question that we are trying to answer. It depends on the problem that you are"
  },
  {
    "startMs": 1971750,
    "text": "It depends on the problem that you are"
  },
  {
    "startMs": 1971760,
    "text": "It depends on the problem that you are trying to prevent. If you are if you"
  },
  {
    "startMs": 1974149,
    "text": "trying to prevent. If you are if you"
  },
  {
    "startMs": 1974159,
    "text": "trying to prevent. If you are if you want to keep the cash from overflowing,"
  },
  {
    "startMs": 1977430,
    "text": "want to keep the cash from overflowing,"
  },
  {
    "startMs": 1977440,
    "text": "want to keep the cash from overflowing, you definitely have to watch your KV"
  },
  {
    "startMs": 1979590,
    "text": "you definitely have to watch your KV"
  },
  {
    "startMs": 1979600,
    "text": "you definitely have to watch your KV pressure."
  },
  {
    "startMs": 1982070,
    "text": "pressure."
  },
  {
    "startMs": 1982080,
    "text": "pressure. If you want to avoid your wasted"
  },
  {
    "startMs": 1984070,
    "text": "If you want to avoid your wasted"
  },
  {
    "startMs": 1984080,
    "text": "If you want to avoid your wasted recomputee, you have to watch out"
  },
  {
    "startMs": 1987430,
    "text": "recomputee, you have to watch out"
  },
  {
    "startMs": 1987440,
    "text": "recomputee, you have to watch out the KV used together with the cash hit"
  },
  {
    "startMs": 1989990,
    "text": "the KV used together with the cash hit"
  },
  {
    "startMs": 1990000,
    "text": "the KV used together with the cash hit rate. A full cache is fine if it's being"
  },
  {
    "startMs": 1992710,
    "text": "rate. A full cache is fine if it's being"
  },
  {
    "startMs": 1992720,
    "text": "rate. A full cache is fine if it's being reused."
  },
  {
    "startMs": 1995029,
    "text": "reused."
  },
  {
    "startMs": 1995039,
    "text": "reused. If you want to keep your weights to be"
  },
  {
    "startMs": 1998389,
    "text": "If you want to keep your weights to be"
  },
  {
    "startMs": 1998399,
    "text": "If you want to keep your weights to be shorter, you have to watch for your"
  },
  {
    "startMs": 2000549,
    "text": "shorter, you have to watch for your"
  },
  {
    "startMs": 2000559,
    "text": "shorter, you have to watch for your queue depth and you have to take actions"
  },
  {
    "startMs": 2002549,
    "text": "queue depth and you have to take actions"
  },
  {
    "startMs": 2002559,
    "text": "queue depth and you have to take actions based on that. If you want to meet your"
  },
  {
    "startMs": 2005430,
    "text": "based on that. If you want to meet your"
  },
  {
    "startMs": 2005440,
    "text": "based on that. If you want to meet your latency targets, you need to watch your"
  },
  {
    "startMs": 2007590,
    "text": "latency targets, you need to watch your"
  },
  {
    "startMs": 2007600,
    "text": "latency targets, you need to watch your deadline slack. Take actions based on"
  },
  {
    "startMs": 2009990,
    "text": "deadline slack. Take actions based on"
  },
  {
    "startMs": 2010000,
    "text": "deadline slack. Take actions based on that. If you need to plan for your long"
  },
  {
    "startMs": 2012710,
    "text": "that. If you need to plan for your long"
  },
  {
    "startMs": 2012720,
    "text": "that. If you need to plan for your long answers, you need to predict for the"
  },
  {
    "startMs": 2014950,
    "text": "answers, you need to predict for the"
  },
  {
    "startMs": 2014960,
    "text": "answers, you need to predict for the output length."
  },
  {
    "startMs": 2017269,
    "text": "output length."
  },
  {
    "startMs": 2017279,
    "text": "output length. And look at the last column"
  },
  {
    "startMs": 2020710,
    "text": "And look at the last column"
  },
  {
    "startMs": 2020720,
    "text": "And look at the last column fails when it tells you when each signal"
  },
  {
    "startMs": 2024710,
    "text": "fails when it tells you when each signal"
  },
  {
    "startMs": 2024720,
    "text": "fails when it tells you when each signal stops working. Look at the first row. KV"
  },
  {
    "startMs": 2028070,
    "text": "stops working. Look at the first row. KV"
  },
  {
    "startMs": 2028080,
    "text": "stops working. Look at the first row. KV pressure fails when the cache never"
  },
  {
    "startMs": 2030310,
    "text": "pressure fails when the cache never"
  },
  {
    "startMs": 2030320,
    "text": "pressure fails when the cache never fills up."
  },
  {
    "startMs": 2032149,
    "text": "fills up."
  },
  {
    "startMs": 2032159,
    "text": "fills up. Remember that row? It comes back to bite"
  },
  {
    "startMs": 2034389,
    "text": "Remember that row? It comes back to bite"
  },
  {
    "startMs": 2034399,
    "text": "Remember that row? It comes back to bite someone in the part four."
  },
  {
    "startMs": 2041200,
    "text": "So what are the four ways to take that"
  },
  {
    "startMs": 2043909,
    "text": "So what are the four ways to take that"
  },
  {
    "startMs": 2043919,
    "text": "So what are the four ways to take that decision?"
  },
  {
    "startMs": 2047519,
    "text": "That is how do you convert your signal"
  },
  {
    "startMs": 2049909,
    "text": "That is how do you convert your signal"
  },
  {
    "startMs": 2049919,
    "text": "That is how do you convert your signal into a decision from the cheapest to the"
  },
  {
    "startMs": 2052389,
    "text": "into a decision from the cheapest to the"
  },
  {
    "startMs": 2052399,
    "text": "into a decision from the cheapest to the most expensive. Here are the ways a"
  },
  {
    "startMs": 2056389,
    "text": "most expensive. Here are the ways a"
  },
  {
    "startMs": 2056399,
    "text": "most expensive. Here are the ways a threshold just compares one number to"
  },
  {
    "startMs": 2058550,
    "text": "threshold just compares one number to"
  },
  {
    "startMs": 2058560,
    "text": "threshold just compares one number to the one limit. Example is a policy named"
  },
  {
    "startMs": 2063270,
    "text": "the one limit. Example is a policy named"
  },
  {
    "startMs": 2063280,
    "text": "the one limit. Example is a policy named clued weight that was uh created"
  },
  {
    "startMs": 2067030,
    "text": "clued weight that was uh created"
  },
  {
    "startMs": 2067040,
    "text": "clued weight that was uh created somewhere in 2026."
  },
  {
    "startMs": 2069430,
    "text": "somewhere in 2026."
  },
  {
    "startMs": 2069440,
    "text": "somewhere in 2026. It takes tens of microsconds per"
  },
  {
    "startMs": 2071589,
    "text": "It takes tens of microsconds per"
  },
  {
    "startMs": 2071599,
    "text": "It takes tens of microsconds per decision."
  },
  {
    "startMs": 2073990,
    "text": "decision."
  },
  {
    "startMs": 2074000,
    "text": "decision. Second is a dual gate which needs two"
  },
  {
    "startMs": 2077430,
    "text": "Second is a dual gate which needs two"
  },
  {
    "startMs": 2077440,
    "text": "Second is a dual gate which needs two signals to agree before it acts. Example"
  },
  {
    "startMs": 2080069,
    "text": "signals to agree before it acts. Example"
  },
  {
    "startMs": 2080079,
    "text": "signals to agree before it acts. Example is a paper called concour"
  },
  {
    "startMs": 2083589,
    "text": "is a paper called concour"
  },
  {
    "startMs": 2083599,
    "text": "is a paper called concour uh which checks like KVUs and the cash"
  },
  {
    "startMs": 2085909,
    "text": "uh which checks like KVUs and the cash"
  },
  {
    "startMs": 2085919,
    "text": "uh which checks like KVUs and the cash hits under 100 microsconds."
  },
  {
    "startMs": 2089270,
    "text": "hits under 100 microsconds."
  },
  {
    "startMs": 2089280,
    "text": "hits under 100 microsconds. Third is an optimizer which solves like"
  },
  {
    "startMs": 2091669,
    "text": "Third is an optimizer which solves like"
  },
  {
    "startMs": 2091679,
    "text": "Third is an optimizer which solves like one big planning problem across all the"
  },
  {
    "startMs": 2093990,
    "text": "one big planning problem across all the"
  },
  {
    "startMs": 2094000,
    "text": "one big planning problem across all the cues at once. Example is a QLM."
  },
  {
    "startMs": 2097510,
    "text": "cues at once. Example is a QLM."
  },
  {
    "startMs": 2097520,
    "text": "cues at once. Example is a QLM. It's thorough but it takes about 5"
  },
  {
    "startMs": 2099910,
    "text": "It's thorough but it takes about 5"
  },
  {
    "startMs": 2099920,
    "text": "It's thorough but it takes about 5 seconds per decision."
  },
  {
    "startMs": 2102069,
    "text": "seconds per decision."
  },
  {
    "startMs": 2102079,
    "text": "seconds per decision. And the fourth is the formal bound uh"
  },
  {
    "startMs": 2105829,
    "text": "And the fourth is the formal bound uh"
  },
  {
    "startMs": 2105839,
    "text": "And the fourth is the formal bound uh which does the math on the worst case."
  },
  {
    "startMs": 2108790,
    "text": "which does the math on the worst case."
  },
  {
    "startMs": 2108800,
    "text": "which does the math on the worst case. Can this request still meets its"
  },
  {
    "startMs": 2110710,
    "text": "Can this request still meets its"
  },
  {
    "startMs": 2110720,
    "text": "Can this request still meets its deadline?"
  },
  {
    "startMs": 2112950,
    "text": "deadline?"
  },
  {
    "startMs": 2112960,
    "text": "deadline? The example is the Kronos on which uses"
  },
  {
    "startMs": 2116630,
    "text": "The example is the Kronos on which uses"
  },
  {
    "startMs": 2116640,
    "text": "The example is the Kronos on which uses the like time to first token and it uh"
  },
  {
    "startMs": 2119910,
    "text": "the like time to first token and it uh"
  },
  {
    "startMs": 2119920,
    "text": "the like time to first token and it uh provides you like a microcond level"
  },
  {
    "startMs": 2121829,
    "text": "provides you like a microcond level"
  },
  {
    "startMs": 2121839,
    "text": "provides you like a microcond level latency."
  },
  {
    "startMs": 2123829,
    "text": "latency."
  },
  {
    "startMs": 2123839,
    "text": "latency. And now hold that cost column up against"
  },
  {
    "startMs": 2126150,
    "text": "And now hold that cost column up against"
  },
  {
    "startMs": 2126160,
    "text": "And now hold that cost column up against our agent. It sends 40 calls in 2"
  },
  {
    "startMs": 2128870,
    "text": "our agent. It sends 40 calls in 2"
  },
  {
    "startMs": 2128880,
    "text": "our agent. It sends 40 calls in 2 seconds. A 5-second decision shows up"
  },
  {
    "startMs": 2131349,
    "text": "seconds. A 5-second decision shows up"
  },
  {
    "startMs": 2131359,
    "text": "seconds. A 5-second decision shows up after every party is over."
  },
  {
    "startMs": 2138400,
    "text": "So what the four papers did"
  },
  {
    "startMs": 2142150,
    "text": "So what the four papers did"
  },
  {
    "startMs": 2142160,
    "text": "So what the four papers did as the complete policy fluid weight"
  },
  {
    "startMs": 2144310,
    "text": "as the complete policy fluid weight"
  },
  {
    "startMs": 2144320,
    "text": "as the complete policy fluid weight admits single request uses the threshold"
  },
  {
    "startMs": 2146870,
    "text": "admits single request uses the threshold"
  },
  {
    "startMs": 2146880,
    "text": "admits single request uses the threshold on your KV use and reports 10 to 25%"
  },
  {
    "startMs": 2150550,
    "text": "on your KV use and reports 10 to 25%"
  },
  {
    "startMs": 2150560,
    "text": "on your KV use and reports 10 to 25% improvement. Conqueror is the one that"
  },
  {
    "startMs": 2153030,
    "text": "improvement. Conqueror is the one that"
  },
  {
    "startMs": 2153040,
    "text": "improvement. Conqueror is the one that admits your VO agents. It uses your dual"
  },
  {
    "startMs": 2156390,
    "text": "admits your VO agents. It uses your dual"
  },
  {
    "startMs": 2156400,
    "text": "admits your VO agents. It uses your dual gate and reports four to 4x"
  },
  {
    "startMs": 2160069,
    "text": "gate and reports four to 4x"
  },
  {
    "startMs": 2160079,
    "text": "gate and reports four to 4x uh the throughput of the SG lang but it"
  },
  {
    "startMs": 2162950,
    "text": "uh the throughput of the SG lang but it"
  },
  {
    "startMs": 2162960,
    "text": "uh the throughput of the SG lang but it was tested offline."
  },
  {
    "startMs": 2165270,
    "text": "was tested offline."
  },
  {
    "startMs": 2165280,
    "text": "was tested offline. Uh QLM admits the request plans across"
  },
  {
    "startMs": 2169109,
    "text": "Uh QLM admits the request plans across"
  },
  {
    "startMs": 2169119,
    "text": "Uh QLM admits the request plans across the W feed and reports 40 to 90% gains"
  },
  {
    "startMs": 2173750,
    "text": "the W feed and reports 40 to 90% gains"
  },
  {
    "startMs": 2173760,
    "text": "the W feed and reports 40 to 90% gains at about 5 seconds per decision. Kronos"
  },
  {
    "startMs": 2176950,
    "text": "at about 5 seconds per decision. Kronos"
  },
  {
    "startMs": 2176960,
    "text": "at about 5 seconds per decision. Kronos admits requests against their deadline"
  },
  {
    "startMs": 2178950,
    "text": "admits requests against their deadline"
  },
  {
    "startMs": 2178960,
    "text": "admits requests against their deadline slack with a formal bound and takes"
  },
  {
    "startMs": 2181589,
    "text": "slack with a formal bound and takes"
  },
  {
    "startMs": 2181599,
    "text": "slack with a formal bound and takes deadline misses from 99% down to zero."
  },
  {
    "startMs": 2185910,
    "text": "deadline misses from 99% down to zero."
  },
  {
    "startMs": 2185920,
    "text": "deadline misses from 99% down to zero. Definitely very amazing accomplishments."
  },
  {
    "startMs": 2189589,
    "text": "Definitely very amazing accomplishments."
  },
  {
    "startMs": 2189599,
    "text": "Definitely very amazing accomplishments. So each purper used a different model"
  },
  {
    "startMs": 2192710,
    "text": "So each purper used a different model"
  },
  {
    "startMs": 2192720,
    "text": "So each purper used a different model engine and a workload. So please don't"
  },
  {
    "startMs": 2194950,
    "text": "engine and a workload. So please don't"
  },
  {
    "startMs": 2194960,
    "text": "engine and a workload. So please don't read this as a leaderboard."
  },
  {
    "startMs": 2197430,
    "text": "read this as a leaderboard."
  },
  {
    "startMs": 2197440,
    "text": "read this as a leaderboard. The question that matters for this room"
  },
  {
    "startMs": 2199910,
    "text": "The question that matters for this room"
  },
  {
    "startMs": 2199920,
    "text": "The question that matters for this room is does any of them fit a coding agent"
  },
  {
    "startMs": 2205920,
    "text": "and the answer is no. We placed every"
  },
  {
    "startMs": 2209829,
    "text": "and the answer is no. We placed every"
  },
  {
    "startMs": 2209839,
    "text": "and the answer is no. We placed every paper on the two axises. One axis is"
  },
  {
    "startMs": 2212950,
    "text": "paper on the two axises. One axis is"
  },
  {
    "startMs": 2212960,
    "text": "paper on the two axises. One axis is what it optimizes deadlines or the"
  },
  {
    "startMs": 2215190,
    "text": "what it optimizes deadlines or the"
  },
  {
    "startMs": 2215200,
    "text": "what it optimizes deadlines or the throughputs. Other is what it admits"
  },
  {
    "startMs": 2217829,
    "text": "throughputs. Other is what it admits"
  },
  {
    "startMs": 2217839,
    "text": "throughputs. Other is what it admits single request or the VO agent sessions."
  },
  {
    "startMs": 2221270,
    "text": "single request or the VO agent sessions."
  },
  {
    "startMs": 2221280,
    "text": "single request or the VO agent sessions. A coding agent needs a policy that"
  },
  {
    "startMs": 2223349,
    "text": "A coding agent needs a policy that"
  },
  {
    "startMs": 2223359,
    "text": "A coding agent needs a policy that admits bold sessions, works online with"
  },
  {
    "startMs": 2225990,
    "text": "admits bold sessions, works online with"
  },
  {
    "startMs": 2226000,
    "text": "admits bold sessions, works online with live traffic and runs engine you haven't"
  },
  {
    "startMs": 2228390,
    "text": "live traffic and runs engine you haven't"
  },
  {
    "startMs": 2228400,
    "text": "live traffic and runs engine you haven't fought"
  },
  {
    "startMs": 2231030,
    "text": "fought"
  },
  {
    "startMs": 2231040,
    "text": "fought and nothing sits there cleanly. Conc"
  },
  {
    "startMs": 2233430,
    "text": "and nothing sits there cleanly. Conc"
  },
  {
    "startMs": 2233440,
    "text": "and nothing sits there cleanly. Conc doesn't need a for but it was only"
  },
  {
    "startMs": 2236230,
    "text": "doesn't need a for but it was only"
  },
  {
    "startMs": 2236240,
    "text": "doesn't need a for but it was only tested offline."
  },
  {
    "startMs": 2238790,
    "text": "tested offline."
  },
  {
    "startMs": 2238800,
    "text": "tested offline. So in practice you will pick the closest"
  },
  {
    "startMs": 2240870,
    "text": "So in practice you will pick the closest"
  },
  {
    "startMs": 2240880,
    "text": "So in practice you will pick the closest match and build just build your own. And"
  },
  {
    "startMs": 2243670,
    "text": "match and build just build your own. And"
  },
  {
    "startMs": 2243680,
    "text": "match and build just build your own. And here is how you can choose which polic"
  },
  {
    "startMs": 2247109,
    "text": "here is how you can choose which polic"
  },
  {
    "startMs": 2247119,
    "text": "here is how you can choose which polic like which kind of a policy you want to"
  },
  {
    "startMs": 2249430,
    "text": "like which kind of a policy you want to"
  },
  {
    "startMs": 2249440,
    "text": "like which kind of a policy you want to try it out."
  },
  {
    "startMs": 2251829,
    "text": "try it out."
  },
  {
    "startMs": 2251839,
    "text": "try it out. So here is that decision tree."
  },
  {
    "startMs": 2254870,
    "text": "So here is that decision tree."
  },
  {
    "startMs": 2254880,
    "text": "So here is that decision tree. First"
  },
  {
    "startMs": 2256630,
    "text": "First"
  },
  {
    "startMs": 2256640,
    "text": "First the question that you have to answer is"
  },
  {
    "startMs": 2258790,
    "text": "the question that you have to answer is"
  },
  {
    "startMs": 2258800,
    "text": "the question that you have to answer is does your request carry any deadlines?"
  },
  {
    "startMs": 2260790,
    "text": "does your request carry any deadlines?"
  },
  {
    "startMs": 2260800,
    "text": "does your request carry any deadlines? If yes, set up a some sort of a formal"
  },
  {
    "startMs": 2264230,
    "text": "If yes, set up a some sort of a formal"
  },
  {
    "startMs": 2264240,
    "text": "If yes, set up a some sort of a formal bound on that deadline. This will take"
  },
  {
    "startMs": 2266550,
    "text": "bound on that deadline. This will take"
  },
  {
    "startMs": 2266560,
    "text": "bound on that deadline. This will take some microsconds level of latency."
  },
  {
    "startMs": 2270790,
    "text": "some microsconds level of latency."
  },
  {
    "startMs": 2270800,
    "text": "some microsconds level of latency. If there is no deadline requirement, the"
  },
  {
    "startMs": 2273910,
    "text": "If there is no deadline requirement, the"
  },
  {
    "startMs": 2273920,
    "text": "If there is no deadline requirement, the second question that you can ask is does"
  },
  {
    "startMs": 2276950,
    "text": "second question that you can ask is does"
  },
  {
    "startMs": 2276960,
    "text": "second question that you can ask is does the KV cache fill at the peak load? If"
  },
  {
    "startMs": 2280950,
    "text": "the KV cache fill at the peak load? If"
  },
  {
    "startMs": 2280960,
    "text": "the KV cache fill at the peak load? If your KV cache fills at the peak load,"
  },
  {
    "startMs": 2285109,
    "text": "your KV cache fills at the peak load,"
  },
  {
    "startMs": 2285119,
    "text": "your KV cache fills at the peak load, then"
  },
  {
    "startMs": 2287750,
    "text": "then"
  },
  {
    "startMs": 2287760,
    "text": "then you would want something like a way to"
  },
  {
    "startMs": 2291109,
    "text": "you would want something like a way to"
  },
  {
    "startMs": 2291119,
    "text": "you would want something like a way to detect the cache prefixes used. So you"
  },
  {
    "startMs": 2294150,
    "text": "detect the cache prefixes used. So you"
  },
  {
    "startMs": 2294160,
    "text": "detect the cache prefixes used. So you can use some sort of a dual gate"
  },
  {
    "startMs": 2295910,
    "text": "can use some sort of a dual gate"
  },
  {
    "startMs": 2295920,
    "text": "can use some sort of a dual gate mechanism or a KV threshold mechanism."
  },
  {
    "startMs": 2299829,
    "text": "mechanism or a KV threshold mechanism."
  },
  {
    "startMs": 2299839,
    "text": "mechanism or a KV threshold mechanism. Um and then if your KV cache gets filled"
  },
  {
    "startMs": 2304310,
    "text": "Um and then if your KV cache gets filled"
  },
  {
    "startMs": 2304320,
    "text": "Um and then if your KV cache gets filled at the peak load, it does not gets"
  },
  {
    "startMs": 2307190,
    "text": "at the peak load, it does not gets"
  },
  {
    "startMs": 2307200,
    "text": "at the peak load, it does not gets filled at the peak load, then you what"
  },
  {
    "startMs": 2308870,
    "text": "filled at the peak load, then you what"
  },
  {
    "startMs": 2308880,
    "text": "filled at the peak load, then you what you could do is you can set up a Q level"
  },
  {
    "startMs": 2310870,
    "text": "you could do is you can set up a Q level"
  },
  {
    "startMs": 2310880,
    "text": "you could do is you can set up a Q level of threshold on it."
  },
  {
    "startMs": 2315280,
    "text": "And that gets us to the benchmarking"
  },
  {
    "startMs": 2318630,
    "text": "And that gets us to the benchmarking"
  },
  {
    "startMs": 2318640,
    "text": "And that gets us to the benchmarking like once you have a policy, how do you"
  },
  {
    "startMs": 2321190,
    "text": "like once you have a policy, how do you"
  },
  {
    "startMs": 2321200,
    "text": "like once you have a policy, how do you benchmark?"
  },
  {
    "startMs": 2323030,
    "text": "benchmark?"
  },
  {
    "startMs": 2323040,
    "text": "benchmark? So I will be handing it over to pop."
  },
  {
    "startMs": 2330800,
    "text": "Thank you. So now moving on to part four"
  },
  {
    "startMs": 2333510,
    "text": "Thank you. So now moving on to part four"
  },
  {
    "startMs": 2333520,
    "text": "Thank you. So now moving on to part four which is benchmarking. As we just saw no"
  },
  {
    "startMs": 2337030,
    "text": "which is benchmarking. As we just saw no"
  },
  {
    "startMs": 2337040,
    "text": "which is benchmarking. As we just saw no publish policy fit a coding agent"
  },
  {
    "startMs": 2339990,
    "text": "publish policy fit a coding agent"
  },
  {
    "startMs": 2340000,
    "text": "publish policy fit a coding agent cleanly. So you are going to adopt one"
  },
  {
    "startMs": 2342710,
    "text": "cleanly. So you are going to adopt one"
  },
  {
    "startMs": 2342720,
    "text": "cleanly. So you are going to adopt one or you you will have to build your own"
  },
  {
    "startMs": 2345270,
    "text": "or you you will have to build your own"
  },
  {
    "startMs": 2345280,
    "text": "or you you will have to build your own and then you have to prove if it's"
  },
  {
    "startMs": 2347670,
    "text": "and then you have to prove if it's"
  },
  {
    "startMs": 2347680,
    "text": "and then you have to prove if it's actually working or not. So to now"
  },
  {
    "startMs": 2351270,
    "text": "actually working or not. So to now"
  },
  {
    "startMs": 2351280,
    "text": "actually working or not. So to now moving on to the next slide"
  },
  {
    "startMs": 2354550,
    "text": "moving on to the next slide"
  },
  {
    "startMs": 2354560,
    "text": "moving on to the next slide you would have to test your policies"
  },
  {
    "startMs": 2356950,
    "text": "you would have to test your policies"
  },
  {
    "startMs": 2356960,
    "text": "you would have to test your policies right so for that you need four things"
  },
  {
    "startMs": 2359910,
    "text": "right so for that you need four things"
  },
  {
    "startMs": 2359920,
    "text": "right so for that you need four things first is your cluster your policy and"
  },
  {
    "startMs": 2363030,
    "text": "first is your cluster your policy and"
  },
  {
    "startMs": 2363040,
    "text": "first is your cluster your policy and trace and score. So let's dig deeper"
  },
  {
    "startMs": 2366310,
    "text": "trace and score. So let's dig deeper"
  },
  {
    "startMs": 2366320,
    "text": "trace and score. So let's dig deeper into each. So your cluster of the first"
  },
  {
    "startMs": 2369910,
    "text": "into each. So your cluster of the first"
  },
  {
    "startMs": 2369920,
    "text": "into each. So your cluster of the first measure it it is capacity with no"
  },
  {
    "startMs": 2373109,
    "text": "measure it it is capacity with no"
  },
  {
    "startMs": 2373119,
    "text": "measure it it is capacity with no admission control at all. The load your"
  },
  {
    "startMs": 2375910,
    "text": "admission control at all. The load your"
  },
  {
    "startMs": 2375920,
    "text": "admission control at all. The load your replica can sustain on their own. Every"
  },
  {
    "startMs": 2378870,
    "text": "replica can sustain on their own. Every"
  },
  {
    "startMs": 2378880,
    "text": "replica can sustain on their own. Every load level at after this is described as"
  },
  {
    "startMs": 2382470,
    "text": "load level at after this is described as"
  },
  {
    "startMs": 2382480,
    "text": "load level at after this is described as multiple of your number like multiple of"
  },
  {
    "startMs": 2385670,
    "text": "multiple of your number like multiple of"
  },
  {
    "startMs": 2385680,
    "text": "multiple of your number like multiple of that number and uh about the policy u"
  },
  {
    "startMs": 2389349,
    "text": "that number and uh about the policy u"
  },
  {
    "startMs": 2389359,
    "text": "that number and uh about the policy u which signals it reads what limit it"
  },
  {
    "startMs": 2392230,
    "text": "which signals it reads what limit it"
  },
  {
    "startMs": 2392240,
    "text": "which signals it reads what limit it uses and what it does when it says no."
  },
  {
    "startMs": 2396230,
    "text": "uses and what it does when it says no."
  },
  {
    "startMs": 2396240,
    "text": "uses and what it does when it says no. The third one is trace which is a record"
  },
  {
    "startMs": 2398950,
    "text": "The third one is trace which is a record"
  },
  {
    "startMs": 2398960,
    "text": "The third one is trace which is a record of real traffic that you replay using"
  },
  {
    "startMs": 2401990,
    "text": "of real traffic that you replay using"
  },
  {
    "startMs": 2402000,
    "text": "of real traffic that you replay using code coding agent traffic ramping from"
  },
  {
    "startMs": 2404790,
    "text": "code coding agent traffic ramping from"
  },
  {
    "startMs": 2404800,
    "text": "code coding agent traffic ramping from half your capacity up to twice your"
  },
  {
    "startMs": 2407990,
    "text": "half your capacity up to twice your"
  },
  {
    "startMs": 2408000,
    "text": "half your capacity up to twice your capacity."
  },
  {
    "startMs": 2409990,
    "text": "capacity."
  },
  {
    "startMs": 2410000,
    "text": "capacity. Four is a score. Then they run the same"
  },
  {
    "startMs": 2413589,
    "text": "Four is a score. Then they run the same"
  },
  {
    "startMs": 2413599,
    "text": "Four is a score. Then they run the same trace twice. Once with the policy and"
  },
  {
    "startMs": 2416790,
    "text": "trace twice. Once with the policy and"
  },
  {
    "startMs": 2416800,
    "text": "trace twice. Once with the policy and once without and then you can compare"
  },
  {
    "startMs": 2419270,
    "text": "once without and then you can compare"
  },
  {
    "startMs": 2419280,
    "text": "once without and then you can compare your result. The score is where the most"
  },
  {
    "startMs": 2422470,
    "text": "your result. The score is where the most"
  },
  {
    "startMs": 2422480,
    "text": "your result. The score is where the most people go wrong. So let's look look at"
  },
  {
    "startMs": 2425349,
    "text": "people go wrong. So let's look look at"
  },
  {
    "startMs": 2425359,
    "text": "people go wrong. So let's look look at that. Now moving on to the next slide"
  },
  {
    "startMs": 2428470,
    "text": "that. Now moving on to the next slide"
  },
  {
    "startMs": 2428480,
    "text": "that. Now moving on to the next slide which is uh how how you score. The score"
  },
  {
    "startMs": 2432550,
    "text": "which is uh how how you score. The score"
  },
  {
    "startMs": 2432560,
    "text": "which is uh how how you score. The score we recommend is called goodput under"
  },
  {
    "startMs": 2435750,
    "text": "we recommend is called goodput under"
  },
  {
    "startMs": 2435760,
    "text": "we recommend is called goodput under admission. So you can see the formula on"
  },
  {
    "startMs": 2438470,
    "text": "admission. So you can see the formula on"
  },
  {
    "startMs": 2438480,
    "text": "admission. So you can see the formula on the screen. The top of the fraction is"
  },
  {
    "startMs": 2441430,
    "text": "the screen. The top of the fraction is"
  },
  {
    "startMs": 2441440,
    "text": "the screen. The top of the fraction is the request that met their SLO their"
  },
  {
    "startMs": 2444710,
    "text": "the request that met their SLO their"
  },
  {
    "startMs": 2444720,
    "text": "the request that met their SLO their latency target. That means the answer"
  },
  {
    "startMs": 2447910,
    "text": "latency target. That means the answer"
  },
  {
    "startMs": 2447920,
    "text": "latency target. That means the answer started on time and it never stalled for"
  },
  {
    "startMs": 2450870,
    "text": "started on time and it never stalled for"
  },
  {
    "startMs": 2450880,
    "text": "started on time and it never stalled for too long between tokens."
  },
  {
    "startMs": 2453589,
    "text": "too long between tokens."
  },
  {
    "startMs": 2453599,
    "text": "too long between tokens. The bottom of the fraction is every"
  },
  {
    "startMs": 2455990,
    "text": "The bottom of the fraction is every"
  },
  {
    "startMs": 2456000,
    "text": "The bottom of the fraction is every request that arrived including the one"
  },
  {
    "startMs": 2459109,
    "text": "request that arrived including the one"
  },
  {
    "startMs": 2459119,
    "text": "request that arrived including the one which got rejected. Why does the bottom"
  },
  {
    "startMs": 2462069,
    "text": "which got rejected. Why does the bottom"
  },
  {
    "startMs": 2462079,
    "text": "which got rejected. Why does the bottom matter so much?"
  },
  {
    "startMs": 2465349,
    "text": "matter so much?"
  },
  {
    "startMs": 2465359,
    "text": "matter so much? So try this. Uh refuse 95% of your"
  },
  {
    "startMs": 2469190,
    "text": "So try this. Uh refuse 95% of your"
  },
  {
    "startMs": 2469200,
    "text": "So try this. Uh refuse 95% of your traffic and serve the remaining 5%"
  },
  {
    "startMs": 2471670,
    "text": "traffic and serve the remaining 5%"
  },
  {
    "startMs": 2471680,
    "text": "traffic and serve the remaining 5% perfectly. If you only count what you"
  },
  {
    "startMs": 2475109,
    "text": "perfectly. If you only count what you"
  },
  {
    "startMs": 2475119,
    "text": "perfectly. If you only count what you admitted, your SLO attainment is 100%."
  },
  {
    "startMs": 2479109,
    "text": "admitted, your SLO attainment is 100%."
  },
  {
    "startMs": 2479119,
    "text": "admitted, your SLO attainment is 100%. Looks amazing, right? Just ship it."
  },
  {
    "startMs": 2482630,
    "text": "Looks amazing, right? Just ship it."
  },
  {
    "startMs": 2482640,
    "text": "Looks amazing, right? Just ship it. But no like good put under admission"
  },
  {
    "startMs": 2485430,
    "text": "But no like good put under admission"
  },
  {
    "startMs": 2485440,
    "text": "But no like good put under admission says 5%. Shedding looks three when you"
  },
  {
    "startMs": 2489750,
    "text": "says 5%. Shedding looks three when you"
  },
  {
    "startMs": 2489760,
    "text": "says 5%. Shedding looks three when you only account the request you kept. It"
  },
  {
    "startMs": 2493270,
    "text": "only account the request you kept. It"
  },
  {
    "startMs": 2493280,
    "text": "only account the request you kept. It isn't."
  },
  {
    "startMs": 2495030,
    "text": "isn't."
  },
  {
    "startMs": 2495040,
    "text": "isn't. So moving on to slide 42."
  },
  {
    "startMs": 2499839,
    "text": "The second thing to get right is the"
  },
  {
    "startMs": 2502150,
    "text": "The second thing to get right is the"
  },
  {
    "startMs": 2502160,
    "text": "The second thing to get right is the workload. A benchmark only tell you"
  },
  {
    "startMs": 2504790,
    "text": "workload. A benchmark only tell you"
  },
  {
    "startMs": 2504800,
    "text": "workload. A benchmark only tell you something if it looks like your real"
  },
  {
    "startMs": 2507270,
    "text": "something if it looks like your real"
  },
  {
    "startMs": 2507280,
    "text": "something if it looks like your real traffic."
  },
  {
    "startMs": 2509589,
    "text": "traffic."
  },
  {
    "startMs": 2509599,
    "text": "traffic. So replay agent traces with their fan"
  },
  {
    "startMs": 2512470,
    "text": "So replay agent traces with their fan"
  },
  {
    "startMs": 2512480,
    "text": "So replay agent traces with their fan out retries session not uniform random"
  },
  {
    "startMs": 2516390,
    "text": "out retries session not uniform random"
  },
  {
    "startMs": 2516400,
    "text": "out retries session not uniform random arrival."
  },
  {
    "startMs": 2517910,
    "text": "arrival."
  },
  {
    "startMs": 2517920,
    "text": "arrival. Describe load as multiple of the"
  },
  {
    "startMs": 2520630,
    "text": "Describe load as multiple of the"
  },
  {
    "startMs": 2520640,
    "text": "Describe load as multiple of the capacity you measured and sweep from"
  },
  {
    "startMs": 2523510,
    "text": "capacity you measured and sweep from"
  },
  {
    "startMs": 2523520,
    "text": "capacity you measured and sweep from half of it to double it."
  },
  {
    "startMs": 2528309,
    "text": "half of it to double it."
  },
  {
    "startMs": 2528319,
    "text": "half of it to double it. Set your guideline as multiple of your"
  },
  {
    "startMs": 2531190,
    "text": "Set your guideline as multiple of your"
  },
  {
    "startMs": 2531200,
    "text": "Set your guideline as multiple of your latency with no load. And remember a"
  },
  {
    "startMs": 2534790,
    "text": "latency with no load. And remember a"
  },
  {
    "startMs": 2534800,
    "text": "latency with no load. And remember a benchmark that never goes past capacity"
  },
  {
    "startMs": 2537349,
    "text": "benchmark that never goes past capacity"
  },
  {
    "startMs": 2537359,
    "text": "benchmark that never goes past capacity never actually tests the policy because"
  },
  {
    "startMs": 2539670,
    "text": "never actually tests the policy because"
  },
  {
    "startMs": 2539680,
    "text": "never actually tests the policy because the policy never has to say no. So you"
  },
  {
    "startMs": 2542470,
    "text": "the policy never has to say no. So you"
  },
  {
    "startMs": 2542480,
    "text": "the policy never has to say no. So you have to make sure that you reach at that"
  },
  {
    "startMs": 2545750,
    "text": "have to make sure that you reach at that"
  },
  {
    "startMs": 2545760,
    "text": "have to make sure that you reach at that point."
  },
  {
    "startMs": 2547589,
    "text": "point."
  },
  {
    "startMs": 2547599,
    "text": "point. Now"
  },
  {
    "startMs": 2549670,
    "text": "Now"
  },
  {
    "startMs": 2549680,
    "text": "Now move moving on to the seven uh seven"
  },
  {
    "startMs": 2552870,
    "text": "move moving on to the seven uh seven"
  },
  {
    "startMs": 2552880,
    "text": "move moving on to the seven uh seven things every admission result should"
  },
  {
    "startMs": 2554630,
    "text": "things every admission result should"
  },
  {
    "startMs": 2554640,
    "text": "things every admission result should disclose."
  },
  {
    "startMs": 2556790,
    "text": "disclose."
  },
  {
    "startMs": 2556800,
    "text": "disclose. Signal liess is one of the seven things"
  },
  {
    "startMs": 2559910,
    "text": "Signal liess is one of the seven things"
  },
  {
    "startMs": 2559920,
    "text": "Signal liess is one of the seven things every admission result should disclose."
  },
  {
    "startMs": 2562390,
    "text": "every admission result should disclose."
  },
  {
    "startMs": 2562400,
    "text": "every admission result should disclose. Here is the full full list that you can"
  },
  {
    "startMs": 2565109,
    "text": "Here is the full full list that you can"
  },
  {
    "startMs": 2565119,
    "text": "Here is the full full list that you can see on your screen."
  },
  {
    "startMs": 2567750,
    "text": "see on your screen."
  },
  {
    "startMs": 2567760,
    "text": "see on your screen. The load as a multiple of measured"
  },
  {
    "startMs": 2569990,
    "text": "The load as a multiple of measured"
  },
  {
    "startMs": 2570000,
    "text": "The load as a multiple of measured capacity"
  },
  {
    "startMs": 2571829,
    "text": "capacity"
  },
  {
    "startMs": 2571839,
    "text": "capacity deadline sets relative to latency with"
  },
  {
    "startMs": 2574470,
    "text": "deadline sets relative to latency with"
  },
  {
    "startMs": 2574480,
    "text": "deadline sets relative to latency with no load."
  },
  {
    "startMs": 2576470,
    "text": "no load."
  },
  {
    "startMs": 2576480,
    "text": "no load. Signal liveless which is no paper"
  },
  {
    "startMs": 2579190,
    "text": "Signal liveless which is no paper"
  },
  {
    "startMs": 2579200,
    "text": "Signal liveless which is no paper report."
  },
  {
    "startMs": 2581910,
    "text": "report."
  },
  {
    "startMs": 2581920,
    "text": "report. How many times you repeated the run and"
  },
  {
    "startMs": 2584150,
    "text": "How many times you repeated the run and"
  },
  {
    "startMs": 2584160,
    "text": "How many times you repeated the run and how much the result varied not one lucky"
  },
  {
    "startMs": 2587030,
    "text": "how much the result varied not one lucky"
  },
  {
    "startMs": 2587040,
    "text": "how much the result varied not one lucky run. Where each parameters comes from,"
  },
  {
    "startMs": 2590390,
    "text": "run. Where each parameters comes from,"
  },
  {
    "startMs": 2590400,
    "text": "run. Where each parameters comes from, turned on this setup, borrowed from"
  },
  {
    "startMs": 2592870,
    "text": "turned on this setup, borrowed from"
  },
  {
    "startMs": 2592880,
    "text": "turned on this setup, borrowed from somewhere else or left at the default."
  },
  {
    "startMs": 2596710,
    "text": "somewhere else or left at the default."
  },
  {
    "startMs": 2596720,
    "text": "somewhere else or left at the default. Whether the rejected request count in a"
  },
  {
    "startMs": 2599030,
    "text": "Whether the rejected request count in a"
  },
  {
    "startMs": 2599040,
    "text": "Whether the rejected request count in a good put denominator and the full"
  },
  {
    "startMs": 2602230,
    "text": "good put denominator and the full"
  },
  {
    "startMs": 2602240,
    "text": "good put denominator and the full configuration"
  },
  {
    "startMs": 2605760,
    "text": "and the full configuration is model,"
  },
  {
    "startMs": 2608230,
    "text": "and the full configuration is model,"
  },
  {
    "startMs": 2608240,
    "text": "and the full configuration is model, engine, version, hardware, workload,"
  },
  {
    "startMs": 2611910,
    "text": "engine, version, hardware, workload,"
  },
  {
    "startMs": 2611920,
    "text": "engine, version, hardware, workload, SLOs's and baseline. None of this ask"
  },
  {
    "startMs": 2615270,
    "text": "SLOs's and baseline. None of this ask"
  },
  {
    "startMs": 2615280,
    "text": "SLOs's and baseline. None of this ask anyone to change how they evaluate. It"
  },
  {
    "startMs": 2618069,
    "text": "anyone to change how they evaluate. It"
  },
  {
    "startMs": 2618079,
    "text": "anyone to change how they evaluate. It just ask them to say what they did. Use"
  },
  {
    "startMs": 2621589,
    "text": "just ask them to say what they did. Use"
  },
  {
    "startMs": 2621599,
    "text": "just ask them to say what they did. Use it as a checklist when you read on on a"
  },
  {
    "startMs": 2624390,
    "text": "it as a checklist when you read on on a"
  },
  {
    "startMs": 2624400,
    "text": "it as a checklist when you read on on a paper and when your benchmark and when"
  },
  {
    "startMs": 2627270,
    "text": "paper and when your benchmark and when"
  },
  {
    "startMs": 2627280,
    "text": "paper and when your benchmark and when you benchmark your own policy. Doing all"
  },
  {
    "startMs": 2631030,
    "text": "you benchmark your own policy. Doing all"
  },
  {
    "startMs": 2631040,
    "text": "you benchmark your own policy. Doing all seven by hand is painful. So we built a"
  },
  {
    "startMs": 2634230,
    "text": "seven by hand is painful. So we built a"
  },
  {
    "startMs": 2634240,
    "text": "seven by hand is painful. So we built a tool that records all of this uh as as a"
  },
  {
    "startMs": 2638069,
    "text": "tool that records all of this uh as as a"
  },
  {
    "startMs": 2638079,
    "text": "tool that records all of this uh as as a default."
  },
  {
    "startMs": 2640710,
    "text": "default."
  },
  {
    "startMs": 2640720,
    "text": "default. So uh the tool is admin perf and it's it"
  },
  {
    "startMs": 2645910,
    "text": "So uh the tool is admin perf and it's it"
  },
  {
    "startMs": 2645920,
    "text": "So uh the tool is admin perf and it's it it's two things in one repository. So in"
  },
  {
    "startMs": 2649349,
    "text": "it's two things in one repository. So in"
  },
  {
    "startMs": 2649359,
    "text": "it's two things in one repository. So in production if you see it's a library you"
  },
  {
    "startMs": 2653030,
    "text": "production if you see it's a library you"
  },
  {
    "startMs": 2653040,
    "text": "production if you see it's a library you put in front of VLLM or SG lang it"
  },
  {
    "startMs": 2656630,
    "text": "put in front of VLLM or SG lang it"
  },
  {
    "startMs": 2656640,
    "text": "put in front of VLLM or SG lang it decides admit defer or reject for every"
  },
  {
    "startMs": 2660069,
    "text": "decides admit defer or reject for every"
  },
  {
    "startMs": 2660079,
    "text": "decides admit defer or reject for every request that's ingress layer from part"
  },
  {
    "startMs": 2663349,
    "text": "request that's ingress layer from part"
  },
  {
    "startMs": 2663359,
    "text": "request that's ingress layer from part two."
  },
  {
    "startMs": 2666560,
    "text": "So if you also see we on the right side"
  },
  {
    "startMs": 2671109,
    "text": "So if you also see we on the right side"
  },
  {
    "startMs": 2671119,
    "text": "So if you also see we on the right side uh it's uh in in a benchmark it's a"
  },
  {
    "startMs": 2674150,
    "text": "uh it's uh in in a benchmark it's a"
  },
  {
    "startMs": 2674160,
    "text": "uh it's uh in in a benchmark it's a harness that runs any policy against a"
  },
  {
    "startMs": 2676870,
    "text": "harness that runs any policy against a"
  },
  {
    "startMs": 2676880,
    "text": "harness that runs any policy against a real engine on a real GPU and reports"
  },
  {
    "startMs": 2680790,
    "text": "real engine on a real GPU and reports"
  },
  {
    "startMs": 2680800,
    "text": "real engine on a real GPU and reports what the policy cost"
  },
  {
    "startMs": 2683510,
    "text": "what the policy cost"
  },
  {
    "startMs": 2683520,
    "text": "what the policy cost both sides calls the same function"
  },
  {
    "startMs": 2685670,
    "text": "both sides calls the same function"
  },
  {
    "startMs": 2685680,
    "text": "both sides calls the same function policy decide. So the policy you"
  },
  {
    "startMs": 2689349,
    "text": "policy decide. So the policy you"
  },
  {
    "startMs": 2689359,
    "text": "policy decide. So the policy you benchmark is exactly the policy you"
  },
  {
    "startMs": 2691430,
    "text": "benchmark is exactly the policy you"
  },
  {
    "startMs": 2691440,
    "text": "benchmark is exactly the policy you deploy."
  },
  {
    "startMs": 2693190,
    "text": "deploy."
  },
  {
    "startMs": 2693200,
    "text": "deploy. We build it because across those 16"
  },
  {
    "startMs": 2696470,
    "text": "We build it because across those 16"
  },
  {
    "startMs": 2696480,
    "text": "We build it because across those 16 paper no two share the same baseline"
  },
  {
    "startMs": 2700230,
    "text": "paper no two share the same baseline"
  },
  {
    "startMs": 2700240,
    "text": "paper no two share the same baseline engine workload or SLO admitf makes a"
  },
  {
    "startMs": 2703990,
    "text": "engine workload or SLO admitf makes a"
  },
  {
    "startMs": 2704000,
    "text": "engine workload or SLO admitf makes a head-to-head comparison possible."
  },
  {
    "startMs": 2707750,
    "text": "head-to-head comparison possible."
  },
  {
    "startMs": 2707760,
    "text": "head-to-head comparison possible. Now moving on to the next slide."
  },
  {
    "startMs": 2711190,
    "text": "Now moving on to the next slide."
  },
  {
    "startMs": 2711200,
    "text": "Now moving on to the next slide. So using its pive command. So you can"
  },
  {
    "startMs": 2715510,
    "text": "So using its pive command. So you can"
  },
  {
    "startMs": 2715520,
    "text": "So using its pive command. So you can see on the screen we have five commands."
  },
  {
    "startMs": 2718390,
    "text": "see on the screen we have five commands."
  },
  {
    "startMs": 2718400,
    "text": "see on the screen we have five commands. So infra up starts a real VLM on a real"
  },
  {
    "startMs": 2721990,
    "text": "So infra up starts a real VLM on a real"
  },
  {
    "startMs": 2722000,
    "text": "So infra up starts a real VLM on a real GPU. Infra smoke checks that it's"
  },
  {
    "startMs": 2725190,
    "text": "GPU. Infra smoke checks that it's"
  },
  {
    "startMs": 2725200,
    "text": "GPU. Infra smoke checks that it's actually serving. Then we have bench"
  },
  {
    "startMs": 2727829,
    "text": "actually serving. Then we have bench"
  },
  {
    "startMs": 2727839,
    "text": "actually serving. Then we have bench runs every policy several times each."
  },
  {
    "startMs": 2731750,
    "text": "runs every policy several times each."
  },
  {
    "startMs": 2731760,
    "text": "runs every policy several times each. Then we have bench computer tell you"
  },
  {
    "startMs": 2734309,
    "text": "Then we have bench computer tell you"
  },
  {
    "startMs": 2734319,
    "text": "Then we have bench computer tell you which policy won and how by how much."
  },
  {
    "startMs": 2737750,
    "text": "which policy won and how by how much."
  },
  {
    "startMs": 2737760,
    "text": "which policy won and how by how much. And then we have infraown which which"
  },
  {
    "startMs": 2740710,
    "text": "And then we have infraown which which"
  },
  {
    "startMs": 2740720,
    "text": "And then we have infraown which which basically means it should shuts all all"
  },
  {
    "startMs": 2743430,
    "text": "basically means it should shuts all all"
  },
  {
    "startMs": 2743440,
    "text": "basically means it should shuts all all off so you stop paying. And that's my"
  },
  {
    "startMs": 2746790,
    "text": "off so you stop paying. And that's my"
  },
  {
    "startMs": 2746800,
    "text": "off so you stop paying. And that's my favorite one."
  },
  {
    "startMs": 2748550,
    "text": "favorite one."
  },
  {
    "startMs": 2748560,
    "text": "favorite one. Uh and so now provisioning and"
  },
  {
    "startMs": 2750870,
    "text": "Uh and so now provisioning and"
  },
  {
    "startMs": 2750880,
    "text": "Uh and so now provisioning and benchmarking are separate commands on"
  },
  {
    "startMs": 2752710,
    "text": "benchmarking are separate commands on"
  },
  {
    "startMs": 2752720,
    "text": "benchmarking are separate commands on purpose. Loading a model takes minutes"
  },
  {
    "startMs": 2755589,
    "text": "purpose. Loading a model takes minutes"
  },
  {
    "startMs": 2755599,
    "text": "purpose. Loading a model takes minutes and every policy has to face exactly"
  },
  {
    "startMs": 2758150,
    "text": "and every policy has to face exactly"
  },
  {
    "startMs": 2758160,
    "text": "and every policy has to face exactly same deployment."
  },
  {
    "startMs": 2759990,
    "text": "same deployment."
  },
  {
    "startMs": 2760000,
    "text": "same deployment. If you are reprovisioned between policy,"
  },
  {
    "startMs": 2762390,
    "text": "If you are reprovisioned between policy,"
  },
  {
    "startMs": 2762400,
    "text": "If you are reprovisioned between policy, you would change the very thing you are"
  },
  {
    "startMs": 2765670,
    "text": "you would change the very thing you are"
  },
  {
    "startMs": 2765680,
    "text": "you would change the very thing you are trying to hold a constant."
  },
  {
    "startMs": 2768550,
    "text": "trying to hold a constant."
  },
  {
    "startMs": 2768560,
    "text": "trying to hold a constant. The deployment is constant. The policy"
  },
  {
    "startMs": 2770790,
    "text": "The deployment is constant. The policy"
  },
  {
    "startMs": 2770800,
    "text": "The deployment is constant. The policy is variable."
  },
  {
    "startMs": 2775520,
    "text": "So now uh here a policy must declare the"
  },
  {
    "startMs": 2779430,
    "text": "So now uh here a policy must declare the"
  },
  {
    "startMs": 2779440,
    "text": "So now uh here a policy must declare the signal it reads. This is how admit perf"
  },
  {
    "startMs": 2782630,
    "text": "signal it reads. This is how admit perf"
  },
  {
    "startMs": 2782640,
    "text": "signal it reads. This is how admit perf deals with the trap I just showed you."
  },
  {
    "startMs": 2785430,
    "text": "deals with the trap I just showed you."
  },
  {
    "startMs": 2785440,
    "text": "deals with the trap I just showed you. Every policy has to declare the signal"
  },
  {
    "startMs": 2787829,
    "text": "Every policy has to declare the signal"
  },
  {
    "startMs": 2787839,
    "text": "Every policy has to declare the signal it needs. This KV cache threshold policy"
  },
  {
    "startMs": 2791270,
    "text": "it needs. This KV cache threshold policy"
  },
  {
    "startMs": 2791280,
    "text": "it needs. This KV cache threshold policy says I require KV used correction which"
  },
  {
    "startMs": 2795510,
    "text": "says I require KV used correction which"
  },
  {
    "startMs": 2795520,
    "text": "says I require KV used correction which is the share of the KV cache in use."
  },
  {
    "startMs": 2800230,
    "text": "is the share of the KV cache in use."
  },
  {
    "startMs": 2800240,
    "text": "is the share of the KV cache in use. If the engine cannot report that signal,"
  },
  {
    "startMs": 2802870,
    "text": "If the engine cannot report that signal,"
  },
  {
    "startMs": 2802880,
    "text": "If the engine cannot report that signal, the run stops at the startup with an"
  },
  {
    "startMs": 2805270,
    "text": "the run stops at the startup with an"
  },
  {
    "startMs": 2805280,
    "text": "the run stops at the startup with an error. You can actually read without the"
  },
  {
    "startMs": 2808150,
    "text": "error. You can actually read without the"
  },
  {
    "startMs": 2808160,
    "text": "error. You can actually read without the check a missing signal reads as a zero"
  },
  {
    "startMs": 2811670,
    "text": "check a missing signal reads as a zero"
  },
  {
    "startMs": 2811680,
    "text": "check a missing signal reads as a zero and KV cache threshold that always sees"
  },
  {
    "startMs": 2814630,
    "text": "and KV cache threshold that always sees"
  },
  {
    "startMs": 2814640,
    "text": "and KV cache threshold that always sees zero admits everything. It's quietly"
  },
  {
    "startMs": 2817829,
    "text": "zero admits everything. It's quietly"
  },
  {
    "startMs": 2817839,
    "text": "zero admits everything. It's quietly turned into the no admit baseline while"
  },
  {
    "startMs": 2821109,
    "text": "turned into the no admit baseline while"
  },
  {
    "startMs": 2821119,
    "text": "turned into the no admit baseline while your results looks completely normal. So"
  },
  {
    "startMs": 2824309,
    "text": "your results looks completely normal. So"
  },
  {
    "startMs": 2824319,
    "text": "your results looks completely normal. So policy that cannot see its signal should"
  },
  {
    "startMs": 2827910,
    "text": "policy that cannot see its signal should"
  },
  {
    "startMs": 2827920,
    "text": "policy that cannot see its signal should fail loudly and not just fail silently."
  },
  {
    "startMs": 2834079,
    "text": "Um so here"
  },
  {
    "startMs": 2837270,
    "text": "Um so here"
  },
  {
    "startMs": 2837280,
    "text": "Um so here what we did is every decision is"
  },
  {
    "startMs": 2840069,
    "text": "what we did is every decision is"
  },
  {
    "startMs": 2840079,
    "text": "what we did is every decision is recorded with the state it saw. Every"
  },
  {
    "startMs": 2842550,
    "text": "recorded with the state it saw. Every"
  },
  {
    "startMs": 2842560,
    "text": "recorded with the state it saw. Every run leaves behind a folder you can open"
  },
  {
    "startMs": 2845270,
    "text": "run leaves behind a folder you can open"
  },
  {
    "startMs": 2845280,
    "text": "run leaves behind a folder you can open inspect. There would be four files as"
  },
  {
    "startMs": 2847910,
    "text": "inspect. There would be four files as"
  },
  {
    "startMs": 2847920,
    "text": "inspect. There would be four files as you can see on the screen. Those are"
  },
  {
    "startMs": 2850230,
    "text": "you can see on the screen. Those are"
  },
  {
    "startMs": 2850240,
    "text": "you can see on the screen. Those are manifest JSON summary JSON decision JSON"
  },
  {
    "startMs": 2853510,
    "text": "manifest JSON summary JSON decision JSON"
  },
  {
    "startMs": 2853520,
    "text": "manifest JSON summary JSON decision JSON and uh out outcome JSON. So"
  },
  {
    "startMs": 2859030,
    "text": "and uh out outcome JSON. So"
  },
  {
    "startMs": 2859040,
    "text": "and uh out outcome JSON. So what each does"
  },
  {
    "startMs": 2861270,
    "text": "what each does"
  },
  {
    "startMs": 2861280,
    "text": "what each does is"
  },
  {
    "startMs": 2862870,
    "text": "is"
  },
  {
    "startMs": 2862880,
    "text": "is uh you can take a look deeply once you"
  },
  {
    "startMs": 2866309,
    "text": "uh you can take a look deeply once you"
  },
  {
    "startMs": 2866319,
    "text": "uh you can take a look deeply once you run the results. Now we we will be"
  },
  {
    "startMs": 2868470,
    "text": "run the results. Now we we will be"
  },
  {
    "startMs": 2868480,
    "text": "run the results. Now we we will be looking at the shading cut. So shade"
  },
  {
    "startMs": 2871750,
    "text": "looking at the shading cut. So shade"
  },
  {
    "startMs": 2871760,
    "text": "looking at the shading cut. So shade shading 37% cut tail latency. So here"
  },
  {
    "startMs": 2875910,
    "text": "shading 37% cut tail latency. So here"
  },
  {
    "startMs": 2875920,
    "text": "shading 37% cut tail latency. So here are the four results how they will look"
  },
  {
    "startMs": 2877750,
    "text": "are the four results how they will look"
  },
  {
    "startMs": 2877760,
    "text": "are the four results how they will look like on a real GPU. The setup small"
  },
  {
    "startMs": 2880550,
    "text": "like on a real GPU. The setup small"
  },
  {
    "startMs": 2880560,
    "text": "like on a real GPU. The setup small model we used quen 2.5 uh and it's a"
  },
  {
    "startMs": 2884870,
    "text": "model we used quen 2.5 uh and it's a"
  },
  {
    "startMs": 2884880,
    "text": "model we used quen 2.5 uh and it's a half billion"
  },
  {
    "startMs": 2886950,
    "text": "half billion"
  },
  {
    "startMs": 2886960,
    "text": "half billion model on a A10 GPU. The engine was kept"
  },
  {
    "startMs": 2890630,
    "text": "model on a A10 GPU. The engine was kept"
  },
  {
    "startMs": 2890640,
    "text": "model on a A10 GPU. The engine was kept out at four sequ sequences at a time."
  },
  {
    "startMs": 2894870,
    "text": "out at four sequ sequences at a time."
  },
  {
    "startMs": 2894880,
    "text": "out at four sequ sequences at a time. Eight request at 14 per second two"
  },
  {
    "startMs": 2897829,
    "text": "Eight request at 14 per second two"
  },
  {
    "startMs": 2897839,
    "text": "Eight request at 14 per second two repeats."
  },
  {
    "startMs": 2900630,
    "text": "repeats."
  },
  {
    "startMs": 2900640,
    "text": "repeats. If you look at the first row with no"
  },
  {
    "startMs": 2903829,
    "text": "If you look at the first row with no"
  },
  {
    "startMs": 2903839,
    "text": "If you look at the first row with no admission control, the P95 time to first"
  },
  {
    "startMs": 2907430,
    "text": "admission control, the P95 time to first"
  },
  {
    "startMs": 2907440,
    "text": "admission control, the P95 time to first token was about two two and two and a"
  },
  {
    "startMs": 2909829,
    "text": "token was about two two and two and a"
  },
  {
    "startMs": 2909839,
    "text": "token was about two two and two and a half around 2 and a half seconds and P"
  },
  {
    "startMs": 2912950,
    "text": "half around 2 and a half seconds and P"
  },
  {
    "startMs": 2912960,
    "text": "half around 2 and a half seconds and P 45 sorry P95 means 95% of the requests"
  },
  {
    "startMs": 2917270,
    "text": "45 sorry P95 means 95% of the requests"
  },
  {
    "startMs": 2917280,
    "text": "45 sorry P95 means 95% of the requests were faster than that and it swung by"
  },
  {
    "startMs": 2920309,
    "text": "were faster than that and it swung by"
  },
  {
    "startMs": 2920319,
    "text": "were faster than that and it swung by about 700 milliseconds either way."
  },
  {
    "startMs": 2924230,
    "text": "about 700 milliseconds either way."
  },
  {
    "startMs": 2924240,
    "text": "about 700 milliseconds either way. Now the last row if you see with a queue"
  },
  {
    "startMs": 2927510,
    "text": "Now the last row if you see with a queue"
  },
  {
    "startMs": 2927520,
    "text": "Now the last row if you see with a queue up policy that allows at most two"
  },
  {
    "startMs": 2930069,
    "text": "up policy that allows at most two"
  },
  {
    "startMs": 2930079,
    "text": "up policy that allows at most two request waiting 60 63% of requests were"
  },
  {
    "startMs": 2934470,
    "text": "request waiting 60 63% of requests were"
  },
  {
    "startMs": 2934480,
    "text": "request waiting 60 63% of requests were admitted and the P95 dropped to about"
  },
  {
    "startMs": 2938790,
    "text": "admitted and the P95 dropped to about"
  },
  {
    "startMs": 2938800,
    "text": "admitted and the P95 dropped to about around 1 second that's 2.5 times better"
  },
  {
    "startMs": 2943510,
    "text": "around 1 second that's 2.5 times better"
  },
  {
    "startMs": 2943520,
    "text": "around 1 second that's 2.5 times better with the exact the good same put"
  },
  {
    "startMs": 2947030,
    "text": "with the exact the good same put"
  },
  {
    "startMs": 2947040,
    "text": "with the exact the good same put now if we look at the signal matter more"
  },
  {
    "startMs": 2950069,
    "text": "now if we look at the signal matter more"
  },
  {
    "startMs": 2950079,
    "text": "now if we look at the signal matter more than the threshold"
  },
  {
    "startMs": 2952710,
    "text": "than the threshold"
  },
  {
    "startMs": 2952720,
    "text": "than the threshold but this This is the result I love most."
  },
  {
    "startMs": 2955349,
    "text": "but this This is the result I love most."
  },
  {
    "startMs": 2955359,
    "text": "but this This is the result I love most. Same run. KV cache usage peaked at half"
  },
  {
    "startMs": 2958710,
    "text": "Same run. KV cache usage peaked at half"
  },
  {
    "startMs": 2958720,
    "text": "Same run. KV cache usage peaked at half a percent. The queue got 24 requests"
  },
  {
    "startMs": 2961910,
    "text": "a percent. The queue got 24 requests"
  },
  {
    "startMs": 2961920,
    "text": "a percent. The queue got 24 requests deep on a small model. KV cache is huge"
  },
  {
    "startMs": 2965109,
    "text": "deep on a small model. KV cache is huge"
  },
  {
    "startMs": 2965119,
    "text": "deep on a small model. KV cache is huge compared to the four short sequences. So"
  },
  {
    "startMs": 2968390,
    "text": "compared to the four short sequences. So"
  },
  {
    "startMs": 2968400,
    "text": "compared to the four short sequences. So it never comes close to filling. All of"
  },
  {
    "startMs": 2971270,
    "text": "it never comes close to filling. All of"
  },
  {
    "startMs": 2971280,
    "text": "it never comes close to filling. All of the pressure showed up in the queue."
  },
  {
    "startMs": 2973109,
    "text": "the pressure showed up in the queue."
  },
  {
    "startMs": 2973119,
    "text": "the pressure showed up in the queue. Instead, a KV cache pressure policy"
  },
  {
    "startMs": 2975910,
    "text": "Instead, a KV cache pressure policy"
  },
  {
    "startMs": 2975920,
    "text": "Instead, a KV cache pressure policy would have botched a flat line and"
  },
  {
    "startMs": 2978309,
    "text": "would have botched a flat line and"
  },
  {
    "startMs": 2978319,
    "text": "would have botched a flat line and admitted everything. that that's the row"
  },
  {
    "startMs": 2982790,
    "text": "admitted everything. that that's the row"
  },
  {
    "startMs": 2982800,
    "text": "admitted everything. that that's the row of choice from two. So"
  },
  {
    "startMs": 2988000,
    "text": "now here and you you can try it today."
  },
  {
    "startMs": 2990950,
    "text": "now here and you you can try it today."
  },
  {
    "startMs": 2990960,
    "text": "now here and you you can try it today. You don't need a GPU. There is a small"
  },
  {
    "startMs": 2993589,
    "text": "You don't need a GPU. There is a small"
  },
  {
    "startMs": 2993599,
    "text": "You don't need a GPU. There is a small fake engine that speaks the three"
  },
  {
    "startMs": 2995750,
    "text": "fake engine that speaks the three"
  },
  {
    "startMs": 2995760,
    "text": "fake engine that speaks the three endpoints admit puff uses and it gets"
  },
  {
    "startMs": 2998630,
    "text": "endpoints admit puff uses and it gets"
  },
  {
    "startMs": 2998640,
    "text": "endpoints admit puff uses and it gets busy under load just like a real one. So"
  },
  {
    "startMs": 3002069,
    "text": "busy under load just like a real one. So"
  },
  {
    "startMs": 3002079,
    "text": "busy under load just like a real one. So you can test the whole setup on your"
  },
  {
    "startMs": 3003990,
    "text": "you can test the whole setup on your"
  },
  {
    "startMs": 3004000,
    "text": "you can test the whole setup on your laptop. If your right policy is your own"
  },
  {
    "startMs": 3006870,
    "text": "laptop. If your right policy is your own"
  },
  {
    "startMs": 3006880,
    "text": "laptop. If your right policy is your own package,"
  },
  {
    "startMs": 3008710,
    "text": "package,"
  },
  {
    "startMs": 3008720,
    "text": "package, admit perf discovers it automatically."
  },
  {
    "startMs": 3011270,
    "text": "admit perf discovers it automatically."
  },
  {
    "startMs": 3011280,
    "text": "admit perf discovers it automatically. No change to the repo and contribute"
  },
  {
    "startMs": 3013829,
    "text": "No change to the repo and contribute"
  },
  {
    "startMs": 3013839,
    "text": "No change to the repo and contribute contributions are very welcome. New"
  },
  {
    "startMs": 3016470,
    "text": "contributions are very welcome. New"
  },
  {
    "startMs": 3016480,
    "text": "contributions are very welcome. New policies, workload, engine adapter or"
  },
  {
    "startMs": 3019670,
    "text": "policies, workload, engine adapter or"
  },
  {
    "startMs": 3019680,
    "text": "policies, workload, engine adapter or matrices."
  },
  {
    "startMs": 3025440,
    "text": "Okay."
  },
  {
    "startMs": 3029839,
    "text": "So if you want to remember like one"
  },
  {
    "startMs": 3032630,
    "text": "So if you want to remember like one"
  },
  {
    "startMs": 3032640,
    "text": "So if you want to remember like one slide just make it this one that"
  },
  {
    "startMs": 3037750,
    "text": "slide just make it this one that"
  },
  {
    "startMs": 3037760,
    "text": "slide just make it this one that when you are accepting the request in"
  },
  {
    "startMs": 3040150,
    "text": "when you are accepting the request in"
  },
  {
    "startMs": 3040160,
    "text": "when you are accepting the request in your inference clusters"
  },
  {
    "startMs": 3042790,
    "text": "your inference clusters"
  },
  {
    "startMs": 3042800,
    "text": "your inference clusters you need to have a mechanism to say no"
  },
  {
    "startMs": 3045109,
    "text": "you need to have a mechanism to say no"
  },
  {
    "startMs": 3045119,
    "text": "you need to have a mechanism to say no early something that works at the"
  },
  {
    "startMs": 3047829,
    "text": "early something that works at the"
  },
  {
    "startMs": 3047839,
    "text": "early something that works at the ingress something before saying yes to"
  },
  {
    "startMs": 3051030,
    "text": "ingress something before saying yes to"
  },
  {
    "startMs": 3051040,
    "text": "ingress something before saying yes to the request before that request goes"
  },
  {
    "startMs": 3054309,
    "text": "the request before that request goes"
  },
  {
    "startMs": 3054319,
    "text": "the request before that request goes into the GPU before that request takes"
  },
  {
    "startMs": 3057349,
    "text": "into the GPU before that request takes"
  },
  {
    "startMs": 3057359,
    "text": "into the GPU before that request takes up your memory."
  },
  {
    "startMs": 3060800,
    "text": "and per added clearly. There are a"
  },
  {
    "startMs": 3063589,
    "text": "and per added clearly. There are a"
  },
  {
    "startMs": 3063599,
    "text": "and per added clearly. There are a couple of error codes that your"
  },
  {
    "startMs": 3065990,
    "text": "couple of error codes that your"
  },
  {
    "startMs": 3066000,
    "text": "couple of error codes that your admission policies return. You need to"
  },
  {
    "startMs": 3068870,
    "text": "admission policies return. You need to"
  },
  {
    "startMs": 3068880,
    "text": "admission policies return. You need to handle those requests and those errors."
  },
  {
    "startMs": 3075280,
    "text": "And with that we would like to come to"
  },
  {
    "startMs": 3078230,
    "text": "And with that we would like to come to"
  },
  {
    "startMs": 3078240,
    "text": "And with that we would like to come to the closure. We would like to thank you"
  },
  {
    "startMs": 3080790,
    "text": "the closure. We would like to thank you"
  },
  {
    "startMs": 3080800,
    "text": "the closure. We would like to thank you all for your patience, for your time."
  },
  {
    "startMs": 3084790,
    "text": "all for your patience, for your time."
  },
  {
    "startMs": 3084800,
    "text": "all for your patience, for your time. And here are like two of our projects."
  },
  {
    "startMs": 3088150,
    "text": "And here are like two of our projects."
  },
  {
    "startMs": 3088160,
    "text": "And here are like two of our projects. It all started as part of the LLM"
  },
  {
    "startMs": 3090309,
    "text": "It all started as part of the LLM"
  },
  {
    "startMs": 3090319,
    "text": "It all started as part of the LLM inference at scale handbook. We and when"
  },
  {
    "startMs": 3093750,
    "text": "inference at scale handbook. We and when"
  },
  {
    "startMs": 3093760,
    "text": "inference at scale handbook. We and when we were doing the survey, we found the"
  },
  {
    "startMs": 3095670,
    "text": "we were doing the survey, we found the"
  },
  {
    "startMs": 3095680,
    "text": "we were doing the survey, we found the gaps and we prepared admit puff. So here"
  },
  {
    "startMs": 3099510,
    "text": "gaps and we prepared admit puff. So here"
  },
  {
    "startMs": 3099520,
    "text": "gaps and we prepared admit puff. So here is the source code for it. Feel free to"
  },
  {
    "startMs": 3102630,
    "text": "is the source code for it. Feel free to"
  },
  {
    "startMs": 3102640,
    "text": "is the source code for it. Feel free to go to it and like experiment with it."
  },
  {
    "startMs": 3105990,
    "text": "go to it and like experiment with it."
  },
  {
    "startMs": 3106000,
    "text": "go to it and like experiment with it. Raise the issues, contribute to it. idea"
  },
  {
    "startMs": 3109430,
    "text": "Raise the issues, contribute to it. idea"
  },
  {
    "startMs": 3109440,
    "text": "Raise the issues, contribute to it. idea is to"
  },
  {
    "startMs": 3112150,
    "text": "is to"
  },
  {
    "startMs": 3112160,
    "text": "is to build your admission policies, benchmark"
  },
  {
    "startMs": 3114630,
    "text": "build your admission policies, benchmark"
  },
  {
    "startMs": 3114640,
    "text": "build your admission policies, benchmark them and be confident that it uh helps"
  },
  {
    "startMs": 3118790,
    "text": "them and be confident that it uh helps"
  },
  {
    "startMs": 3118800,
    "text": "them and be confident that it uh helps you keep your cluster stable. Yeah."
  },
  {
    "startMs": 3121589,
    "text": "you keep your cluster stable. Yeah."
  },
  {
    "startMs": 3121599,
    "text": "you keep your cluster stable. Yeah. Thank you everyone."
  },
  {
    "startMs": 3122790,
    "text": "Thank you everyone."
  },
  {
    "startMs": 3122800,
    "text": "Thank you everyone. Thank you everyone."
  }
]