[
  {
    "startMs": 13599,
    "text": "Uh so good afternoon everyone. Um my"
  },
  {
    "startMs": 16720,
    "text": "name is Hershel Jan and he is Tanisha."
  },
  {
    "startMs": 20320,
    "text": "Uh and we would like to welcome you all"
  },
  {
    "startMs": 22800,
    "text": "in this two hours workshop on the LLM"
  },
  {
    "startMs": 25359,
    "text": "inference. Uh so the goal of this"
  },
  {
    "startMs": 28480,
    "text": "workshop is to understand this domain"
  },
  {
    "startMs": 32640,
    "text": "from the first principles uh dive deeper"
  },
  {
    "startMs": 35840,
    "text": "into it and like understand what's going"
  },
  {
    "startMs": 38160,
    "text": "on throughout the industry."
  },
  {
    "startMs": 41440,
    "text": "Uh a bit of background about us. So I am"
  },
  {
    "startMs": 45680,
    "text": "a senior software engineer at Audible."
  },
  {
    "startMs": 48160,
    "text": "uh have been building MLA data platforms"
  },
  {
    "startMs": 51440,
    "text": "for the past five years and on the sides"
  },
  {
    "startMs": 54559,
    "text": "I have been writing this opensource"
  },
  {
    "startMs": 56399,
    "text": "handbook on LLM inference"
  },
  {
    "startMs": 59359,
    "text": "and Tanme he is the senior quantitative"
  },
  {
    "startMs": 63199,
    "text": "modeler at XAN cup bank corporation he"
  },
  {
    "startMs": 66960,
    "text": "recently completed his PhD and he has"
  },
  {
    "startMs": 69920,
    "text": "been actively doing research in the"
  },
  {
    "startMs": 72560,
    "text": "agent verifiers and the world models"
  },
  {
    "startMs": 78880,
    "text": "Uh so a quick show of hands here. Uh Vu"
  },
  {
    "startMs": 82560,
    "text": "here is like brand new to the LLM"
  },
  {
    "startMs": 84640,
    "text": "inference."
  },
  {
    "startMs": 87200,
    "text": "Okay, great. And Vu here has like"
  },
  {
    "startMs": 89920,
    "text": "deployed these models in production."
  },
  {
    "startMs": 93119,
    "text": "They have been tuning it. They have been"
  },
  {
    "startMs": 95280,
    "text": "serving the production traffic."
  },
  {
    "startMs": 98720,
    "text": "Okay, great."
  },
  {
    "startMs": 101119,
    "text": "So this workshop is targeted towards the"
  },
  {
    "startMs": 104640,
    "text": "beginner and the intermediate level. U"
  },
  {
    "startMs": 108079,
    "text": "and all of the slides and exercises they"
  },
  {
    "startMs": 110240,
    "text": "are in the repo. I will share that soon."
  },
  {
    "startMs": 115119,
    "text": "Here is the quick agenda for the"
  },
  {
    "startMs": 117920,
    "text": "workshop. We will start with the problem"
  },
  {
    "startMs": 120159,
    "text": "statement. We will try to understand few"
  },
  {
    "startMs": 122960,
    "text": "of the pain points around LLM inference."
  },
  {
    "startMs": 126399,
    "text": "uh then we understand what causes those"
  },
  {
    "startMs": 128399,
    "text": "pain points and build our foundations"
  },
  {
    "startMs": 130399,
    "text": "from there."
  },
  {
    "startMs": 132160,
    "text": "Then we will dive into like two kind of"
  },
  {
    "startMs": 135280,
    "text": "the optimizations that we do like the"
  },
  {
    "startMs": 137760,
    "text": "model optimizations and the serving"
  },
  {
    "startMs": 139599,
    "text": "optimizations."
  },
  {
    "startMs": 141360,
    "text": "uh and then we start learning about"
  },
  {
    "startMs": 143599,
    "text": "different serving engines that are"
  },
  {
    "startMs": 145280,
    "text": "available to deploy our LLM inference"
  },
  {
    "startMs": 149200,
    "text": "solutions in production and we will"
  },
  {
    "startMs": 151599,
    "text": "showcase some benchmarks and the"
  },
  {
    "startMs": 153840,
    "text": "decision chart on like which engine to"
  },
  {
    "startMs": 156480,
    "text": "use."
  },
  {
    "startMs": 160239,
    "text": "Cool. So to understand the pain points"
  },
  {
    "startMs": 163760,
    "text": "first we need to know what is like LLM"
  },
  {
    "startMs": 166319,
    "text": "inference. So, and probably a lot of us"
  },
  {
    "startMs": 169599,
    "text": "already know this. Um, but yeah,"
  },
  {
    "startMs": 172640,
    "text": "anything that you ask your AI to do like"
  },
  {
    "startMs": 175440,
    "text": "whether it be generate a video, audio,"
  },
  {
    "startMs": 178239,
    "text": "analyze any text, uh, analyze your"
  },
  {
    "startMs": 181120,
    "text": "medical reports or like your tax bills,"
  },
  {
    "startMs": 184080,
    "text": "all of that is like an LLM inference."
  },
  {
    "startMs": 188080,
    "text": "And this market is like approximately"
  },
  {
    "startMs": 191200,
    "text": "$23 billion today."
  },
  {
    "startMs": 195040,
    "text": "uh semi analysis recently shared that if"
  },
  {
    "startMs": 198400,
    "text": "you want to model like a Google search"
  },
  {
    "startMs": 200560,
    "text": "queries with LLMs, you need like a"
  },
  {
    "startMs": 203599,
    "text": "profit drain of like $36 billion"
  },
  {
    "startMs": 207120,
    "text": "and query cost has to be less than 0.5"
  },
  {
    "startMs": 211280,
    "text": "cents to keep your search business"
  },
  {
    "startMs": 213840,
    "text": "profitable."
  },
  {
    "startMs": 215840,
    "text": "On the other hand, the business insider"
  },
  {
    "startMs": 218640,
    "text": "mentioned like your AI has to be put on"
  },
  {
    "startMs": 220720,
    "text": "diet"
  },
  {
    "startMs": 222400,
    "text": "and everyone has to start auditing and"
  },
  {
    "startMs": 224720,
    "text": "budgeting their token usage and all of"
  },
  {
    "startMs": 227680,
    "text": "this is happening. Why? Because your"
  },
  {
    "startMs": 230159,
    "text": "hardware is limited, compute is"
  },
  {
    "startMs": 232080,
    "text": "expensive, your inference is expensive"
  },
  {
    "startMs": 235440,
    "text": "and with the growing need of like more"
  },
  {
    "startMs": 238239,
    "text": "and more AI usage, this inference cost"
  },
  {
    "startMs": 240400,
    "text": "is rising more and more."
  },
  {
    "startMs": 243760,
    "text": "So"
  },
  {
    "startMs": 245599,
    "text": "this stat, it's an old stat from the"
  },
  {
    "startMs": 248080,
    "text": "open AI, but it's still it's still true."
  },
  {
    "startMs": 251599,
    "text": "So if you look at the like training cost"
  },
  {
    "startMs": 254640,
    "text": "of the GPT3, it was like around $4.6"
  },
  {
    "startMs": 257199,
    "text": "million. It was a one-time cost. But if"
  },
  {
    "startMs": 260400,
    "text": "you see the inference cost that has been"
  },
  {
    "startMs": 263360,
    "text": "like uh it's a recurring cost because"
  },
  {
    "startMs": 266080,
    "text": "it's a operating cost that scales with"
  },
  {
    "startMs": 268320,
    "text": "every user that comes in that every"
  },
  {
    "startMs": 270240,
    "text": "token that comes in every session that"
  },
  {
    "startMs": 272880,
    "text": "uh is being initiated on the like AI"
  },
  {
    "startMs": 279520,
    "text": "and there are only two ways to basically"
  },
  {
    "startMs": 282560,
    "text": "counter this. Uh one way is you reduce"
  },
  {
    "startMs": 285600,
    "text": "your token usage."
  },
  {
    "startMs": 287919,
    "text": "um alternative is you should try to"
  },
  {
    "startMs": 290880,
    "text": "optimize your inference solutions as a"
  },
  {
    "startMs": 293680,
    "text": "inference service provider for your"
  },
  {
    "startMs": 296320,
    "text": "customers and for yourself. And so this"
  },
  {
    "startMs": 299680,
    "text": "um we have been seeing like lot and lot"
  },
  {
    "startMs": 302080,
    "text": "of like new solutions coming out every"
  },
  {
    "startMs": 304560,
    "text": "then and now. Um and so the idea would"
  },
  {
    "startMs": 308080,
    "text": "be like okay we will try to build those"
  },
  {
    "startMs": 310240,
    "text": "foundations that will help us understand"
  },
  {
    "startMs": 313680,
    "text": "and evaluate like whatever ships next."
  },
  {
    "startMs": 317360,
    "text": "Um so yeah to get started like we will"
  },
  {
    "startMs": 321840,
    "text": "do a quick demo like it's a short demo"
  },
  {
    "startMs": 324080,
    "text": "of like what are the different pain"
  },
  {
    "startMs": 325840,
    "text": "points around inference"
  },
  {
    "startMs": 328720,
    "text": "u and so this is the repo uh I mean you"
  },
  {
    "startMs": 333039,
    "text": "can pull it or you can also open it on"
  },
  {
    "startMs": 335440,
    "text": "the GitHub"
  },
  {
    "startMs": 337199,
    "text": "uh it's called LLM inference at scale a"
  },
  {
    "startMs": 339759,
    "text": "bit of background here like four months"
  },
  {
    "startMs": 342080,
    "text": "back when I didn't knew anything on the"
  },
  {
    "startMs": 343919,
    "text": "LLM inference um I started learning it I"
  },
  {
    "startMs": 347120,
    "text": "saw like lot of resources were"
  },
  {
    "startMs": 348639,
    "text": "scattered. So we started putting it uh"
  },
  {
    "startMs": 351280,
    "text": "like all together in one place uh so"
  },
  {
    "startMs": 353840,
    "text": "that it could benefit people."
  },
  {
    "startMs": 356720,
    "text": "Um yes. So let me actually get out of"
  },
  {
    "startMs": 360000,
    "text": "this slideshow mode and probably"
  },
  {
    "startMs": 364080,
    "text": "go into"
  },
  {
    "startMs": 368319,
    "text": "I will go to this extended mode."
  },
  {
    "startMs": 373120,
    "text": "Um"
  },
  {
    "startMs": 376560,
    "text": "okay great."
  },
  {
    "startMs": 379199,
    "text": "Uh yeah so in this uh repository if you"
  },
  {
    "startMs": 383919,
    "text": "see a readme file there is like a link"
  },
  {
    "startMs": 386400,
    "text": "to the slides. Uh so this it will be"
  },
  {
    "startMs": 389520,
    "text": "like this folder where you have like a"
  },
  {
    "startMs": 391280,
    "text": "pptx and there is like a benchmark"
  },
  {
    "startMs": 393600,
    "text": "report in there."
  },
  {
    "startMs": 396000,
    "text": "uh you can always like download it and"
  },
  {
    "startMs": 399440,
    "text": "then for the demo purposes uh we have"
  },
  {
    "startMs": 402000,
    "text": "couple of Jupyter notebooks. Uh we have"
  },
  {
    "startMs": 404800,
    "text": "like collaborated with Moab who are the"
  },
  {
    "startMs": 407680,
    "text": "like Google collab alternative"
  },
  {
    "startMs": 410240,
    "text": "and what they basically provide you is"
  },
  {
    "startMs": 412880,
    "text": "like a free RTX 6000 GPU. It's a 100 GB"
  },
  {
    "startMs": 416000,
    "text": "V RAM GPU."
  },
  {
    "startMs": 418240,
    "text": "So"
  },
  {
    "startMs": 419759,
    "text": "and we have like already set up these"
  },
  {
    "startMs": 421440,
    "text": "notebooks so that it becomes easy to"
  },
  {
    "startMs": 424319,
    "text": "like experiment with and like all of the"
  },
  {
    "startMs": 427919,
    "text": "assets and everything are preset for"
  },
  {
    "startMs": 429919,
    "text": "you. Uh"
  },
  {
    "startMs": 432400,
    "text": "so uh we will start with like a simple"
  },
  {
    "startMs": 435680,
    "text": "demo a"
  },
  {
    "startMs": 438000,
    "text": "probably"
  },
  {
    "startMs": 441599,
    "text": "let me just see"
  },
  {
    "startMs": 449039,
    "text": "Okay."
  },
  {
    "startMs": 451280,
    "text": "Um yeah. So"
  },
  {
    "startMs": 455120,
    "text": "when it comes to the inference, you need"
  },
  {
    "startMs": 457199,
    "text": "to do an inference on a certain model,"
  },
  {
    "startMs": 460000,
    "text": "right? Uh for the workshop purposes, we"
  },
  {
    "startMs": 462880,
    "text": "are using a simple ML 7B model. Uh it's"
  },
  {
    "startMs": 466160,
    "text": "a small model of around 15GB in size. So"
  },
  {
    "startMs": 469759,
    "text": "we we are going to like load that into"
  },
  {
    "startMs": 471919,
    "text": "the GPU. So,"
  },
  {
    "startMs": 475440,
    "text": "and we would look like some of the GPU"
  },
  {
    "startMs": 477360,
    "text": "stats as well. So, we see like okay, we"
  },
  {
    "startMs": 479520,
    "text": "are working on the 6,000 Blackwell. Uh,"
  },
  {
    "startMs": 482000,
    "text": "and you might be thinking I'm not"
  },
  {
    "startMs": 483440,
    "text": "running the cells because I don't trust"
  },
  {
    "startMs": 485599,
    "text": "the Wi-Fi at conferences."
  },
  {
    "startMs": 488639,
    "text": "So, yeah. So, I would probably be just"
  },
  {
    "startMs": 491039,
    "text": "going over the results uh that we kind"
  },
  {
    "startMs": 493840,
    "text": "of ran previously."
  },
  {
    "startMs": 498479,
    "text": "Uh, yeah. So, we have like a GPU which"
  },
  {
    "startMs": 500960,
    "text": "is like 102GB."
  },
  {
    "startMs": 503280,
    "text": "Uh now the first thing that comes to my"
  },
  {
    "startMs": 506160,
    "text": "mind is like what's my memory"
  },
  {
    "startMs": 507759,
    "text": "consumption looks like when I do the LLM"
  },
  {
    "startMs": 509919,
    "text": "inference. So I load this model"
  },
  {
    "startMs": 513760,
    "text": "uh and I see like okay I have like a 15"
  },
  {
    "startMs": 516479,
    "text": "GB here. So I have roughly like 87.5GB."
  },
  {
    "startMs": 520959,
    "text": "And now when I do the like inference"
  },
  {
    "startMs": 523599,
    "text": "here"
  },
  {
    "startMs": 525360,
    "text": "uh what I notice is like the more the"
  },
  {
    "startMs": 528560,
    "text": "number of inputs I pass more is the"
  },
  {
    "startMs": 531279,
    "text": "memory that I need."
  },
  {
    "startMs": 533200,
    "text": "uh and it's increasing slowly but it's"
  },
  {
    "startMs": 535519,
    "text": "still increasing. [snorts] So imagine"
  },
  {
    "startMs": 537360,
    "text": "like if you have a context length of"
  },
  {
    "startMs": 539360,
    "text": "like around 4,000 or 16,000 or 32,000"
  },
  {
    "startMs": 543360,
    "text": "uh tokens. Uh so this memory could like"
  },
  {
    "startMs": 546959,
    "text": "really grow big and it you could"
  },
  {
    "startMs": 548720,
    "text": "actually get like all of those out of"
  },
  {
    "startMs": 550880,
    "text": "memory issues."
  },
  {
    "startMs": 553120,
    "text": "Uh so definitely this is like your"
  },
  {
    "startMs": 555040,
    "text": "problem one like your memory increasing"
  },
  {
    "startMs": 557360,
    "text": "with the increase in tokens. So in form"
  },
  {
    "startMs": 559680,
    "text": "of like a simple visualization it looks"
  },
  {
    "startMs": 561839,
    "text": "like this."
  },
  {
    "startMs": 565200,
    "text": "The second problem that you would see is"
  },
  {
    "startMs": 567440,
    "text": "like"
  },
  {
    "startMs": 568959,
    "text": "the time to your first token it's very"
  },
  {
    "startMs": 571360,
    "text": "very slow. Uh we measure it by a metric"
  },
  {
    "startMs": 574399,
    "text": "called TTFT. It's a short short form of"
  },
  {
    "startMs": 577360,
    "text": "it. uh and when you try to like measure"
  },
  {
    "startMs": 581920,
    "text": "the TTFT uh with the like input size you"
  },
  {
    "startMs": 586160,
    "text": "would see like longer the context"
  },
  {
    "startMs": 589760,
    "text": "you would see like this uh TTFT being"
  },
  {
    "startMs": 592640,
    "text": "slow. So now there are two problems."
  },
  {
    "startMs": 594640,
    "text": "Your memory increases with the token"
  },
  {
    "startMs": 596320,
    "text": "size. Your TTFT increases with the token"
  },
  {
    "startMs": 599279,
    "text": "size."
  },
  {
    "startMs": 600800,
    "text": "Uh sorry not the token size, the context"
  },
  {
    "startMs": 602880,
    "text": "size."
  },
  {
    "startMs": 606800,
    "text": "Uh and then the third is the like"
  },
  {
    "startMs": 608959,
    "text": "throughput. The throughput is like how"
  },
  {
    "startMs": 611519,
    "text": "many tokens can you serve per second and"
  },
  {
    "startMs": 614079,
    "text": "then how many users can you serve per"
  },
  {
    "startMs": 616079,
    "text": "second. So if you take a very very"
  },
  {
    "startMs": 618480,
    "text": "vanilla implementation on your local"
  },
  {
    "startMs": 620640,
    "text": "system"
  },
  {
    "startMs": 622240,
    "text": "uh it would be like very sequential. So"
  },
  {
    "startMs": 624800,
    "text": "if you send like five requests all those"
  },
  {
    "startMs": 627519,
    "text": "five requests would be catered like"
  },
  {
    "startMs": 629440,
    "text": "sequentially rather than parallelly."
  },
  {
    "startMs": 632640,
    "text": "Uh and so like your request basically"
  },
  {
    "startMs": 636399,
    "text": "takes more time to complete if you have"
  },
  {
    "startMs": 638240,
    "text": "like multiple users."
  },
  {
    "startMs": 641040,
    "text": "So these are the like three problems."
  },
  {
    "startMs": 643440,
    "text": "There is a fourth one. I haven't"
  },
  {
    "startMs": 644800,
    "text": "described it here. probably we will"
  },
  {
    "startMs": 646800,
    "text": "build that intuition as we move forward."
  },
  {
    "startMs": 650480,
    "text": "Uh but let's remember like these are the"
  },
  {
    "startMs": 652640,
    "text": "three problems the memory TTFT and the"
  },
  {
    "startMs": 655279,
    "text": "throughput."
  },
  {
    "startMs": 658160,
    "text": "Cool. Uh I will go back to the slides."
  },
  {
    "startMs": 667680,
    "text": "Okay, perfect."
  },
  {
    "startMs": 675680,
    "text": "Uh so it should be this."
  },
  {
    "startMs": 680880,
    "text": ">> Uh is it visible?"
  },
  {
    "startMs": 695120,
    "text": "Oh, you you need the"
  },
  {
    "startMs": 698640,
    "text": "Oh, okay. Workshop. Okay."
  },
  {
    "startMs": 707600,
    "text": "Yeah. So, within that repository, if you"
  },
  {
    "startMs": 710800,
    "text": "see a workshop folder, you see that"
  },
  {
    "startMs": 713519,
    "text": "readme and then the readme has all the"
  },
  {
    "startMs": 715760,
    "text": "links, the slides and the demos."
  },
  {
    "startMs": 724959,
    "text": "Uh, does that work?"
  },
  {
    "startMs": 729680,
    "text": "Okay, perfect."
  },
  {
    "startMs": 736399,
    "text": "Okay. uh so let's start working through"
  },
  {
    "startMs": 738959,
    "text": "the foundations like let's start"
  },
  {
    "startMs": 740880,
    "text": "understanding"
  },
  {
    "startMs": 742399,
    "text": "uh what are the reasons behind those"
  },
  {
    "startMs": 744480,
    "text": "pain points and for that like we have to"
  },
  {
    "startMs": 747279,
    "text": "look at this inference pipeline"
  },
  {
    "startMs": 749920,
    "text": "um"
  },
  {
    "startMs": 752000,
    "text": "so we get like an input text uh that"
  },
  {
    "startMs": 755920,
    "text": "text could have like any number of words"
  },
  {
    "startMs": 759279,
    "text": "you convert those into the tokens so for"
  },
  {
    "startMs": 762800,
    "text": "simplicity you can assume one word equal"
  },
  {
    "startMs": 765200,
    "text": "to one token"
  },
  {
    "startMs": 767040,
    "text": "uh then you kind of convert them into"
  },
  {
    "startMs": 769440,
    "text": "like the embeddings and then you send it"
  },
  {
    "startMs": 771839,
    "text": "to the like transformers"
  },
  {
    "startMs": 774000,
    "text": "uh like there are 32 layers of"
  },
  {
    "startMs": 776079,
    "text": "transformers but that's specific to the"
  },
  {
    "startMs": 777920,
    "text": "Mistl 7B different models have different"
  },
  {
    "startMs": 780000,
    "text": "kind number of layers uh and then you"
  },
  {
    "startMs": 783120,
    "text": "generate a new token and that token"
  },
  {
    "startMs": 785360,
    "text": "basically goes back to the input then"
  },
  {
    "startMs": 787120,
    "text": "you generate another token and that"
  },
  {
    "startMs": 788880,
    "text": "keeps on going now in this entire"
  },
  {
    "startMs": 791920,
    "text": "pipeline you would see like 95% of your"
  },
  {
    "startMs": 795680,
    "text": "compute is like taken by these"
  },
  {
    "startMs": 797360,
    "text": "transformer layers. So it's worth"
  },
  {
    "startMs": 799760,
    "text": "looking at like what goes within this"
  },
  {
    "startMs": 801680,
    "text": "transformer layer."
  },
  {
    "startMs": 805120,
    "text": "Within this transformer layer you would"
  },
  {
    "startMs": 806959,
    "text": "have like more layers. You have like a"
  },
  {
    "startMs": 809760,
    "text": "normalization layer, you have an"
  },
  {
    "startMs": 811360,
    "text": "attention layer, you have a feed forward"
  },
  {
    "startMs": 814000,
    "text": "layer and all."
  },
  {
    "startMs": 815920,
    "text": "And attention layer is the one I think"
  },
  {
    "startMs": 818959,
    "text": "that has been very very famous."
  },
  {
    "startMs": 820560,
    "text": "Attention is all you need paper. I think"
  },
  {
    "startMs": 822880,
    "text": "that's very well known. So attention is"
  },
  {
    "startMs": 825920,
    "text": "the most compute intensive layer and we"
  },
  {
    "startMs": 829600,
    "text": "need to understand what goes within that"
  },
  {
    "startMs": 831920,
    "text": "attention layer."
  },
  {
    "startMs": 834560,
    "text": "So what does attention do? Attention uh"
  },
  {
    "startMs": 838079,
    "text": "so if you have an input text it needs to"
  },
  {
    "startMs": 840399,
    "text": "find the attention scores of every token"
  },
  {
    "startMs": 843040,
    "text": "with respect to all of the previous"
  },
  {
    "startMs": 844800,
    "text": "tokens."
  },
  {
    "startMs": 846720,
    "text": "And to do that what it needs to do is"
  },
  {
    "startMs": 849120,
    "text": "like it needs to project every token"
  },
  {
    "startMs": 851040,
    "text": "into like a key query and the value"
  },
  {
    "startMs": 853519,
    "text": "space. So in like in a simpler terms"
  },
  {
    "startMs": 858800,
    "text": "just u understand this like if you have"
  },
  {
    "startMs": 862160,
    "text": "10 tokens then it needs like the 10"
  },
  {
    "startMs": 864320,
    "text": "different query key and the value"
  },
  {
    "startMs": 866079,
    "text": "vectors. If there are 100 tokens you"
  },
  {
    "startMs": 869040,
    "text": "would need 100 key and the value"
  },
  {
    "startMs": 870639,
    "text": "vectors. If there are thousand tokens"
  },
  {
    "startMs": 872959,
    "text": "you would need thousand key value"
  },
  {
    "startMs": 874480,
    "text": "vectors. And so like your number of the"
  },
  {
    "startMs": 876959,
    "text": "key and the value vectors they increase"
  },
  {
    "startMs": 878800,
    "text": "as you increase the input size."
  },
  {
    "startMs": 884720,
    "text": "Uh and if you calculate the like KV size"
  },
  {
    "startMs": 888480,
    "text": "per token"
  },
  {
    "startMs": 890320,
    "text": "uh for a mist 7B it comes out to be 131"
  },
  {
    "startMs": 893360,
    "text": "KV. Uh this is because like"
  },
  {
    "startMs": 897680,
    "text": "uh you have two vectors K and V. You"
  },
  {
    "startMs": 900240,
    "text": "have to multiply the size uh one vector"
  },
  {
    "startMs": 903040,
    "text": "is like 128 dimensions. you have to"
  },
  {
    "startMs": 905279,
    "text": "multi multiply it by 32 transformer"
  },
  {
    "startMs": 908880,
    "text": "layers and then you have to multiply it"
  },
  {
    "startMs": 910959,
    "text": "by the KV heads for ML 7B it's gate KV"
  },
  {
    "startMs": 915040,
    "text": "heads it's not like 32 because it uses a"
  },
  {
    "startMs": 918639,
    "text": "different kind of an attention mechanism"
  },
  {
    "startMs": 921600,
    "text": "u which we will talk about for sure but"
  },
  {
    "startMs": 925120,
    "text": "yeah so the KV size per token is like"
  },
  {
    "startMs": 929360,
    "text": "your 131 KB now imagine if you have 4K"
  },
  {
    "startMs": 933199,
    "text": "context uh So that size becomes like"
  },
  {
    "startMs": 935920,
    "text": "half a GB. Uh if you do like 16k context"
  },
  {
    "startMs": 940000,
    "text": "that size becomes 2.1 GB. Uh now"
  },
  {
    "startMs": 944000,
    "text": "multiplay by the users like assume you"
  },
  {
    "startMs": 946800,
    "text": "can serve multiple users together"
  },
  {
    "startMs": 950000,
    "text": "at the same time within that GPU you"
  },
  {
    "startMs": 953360,
    "text": "could have like 42GB with a 4K context"
  },
  {
    "startMs": 956959,
    "text": "and 80 users. Um and if your GPU is only"
  },
  {
    "startMs": 960639,
    "text": "like let's say 24GB, you are already"
  },
  {
    "startMs": 963360,
    "text": "running out of the memory. So you cannot"
  },
  {
    "startMs": 965759,
    "text": "serve that many users with that many"
  },
  {
    "startMs": 967680,
    "text": "context."
  },
  {
    "startMs": 972320,
    "text": "To visualize this, look at a GPU memory."
  },
  {
    "startMs": 975279,
    "text": "So the GPU memory has like a model"
  },
  {
    "startMs": 977600,
    "text": "weights which are pretty fixed. These"
  },
  {
    "startMs": 980160,
    "text": "are pre-trained weights. Uh there is"
  },
  {
    "startMs": 983040,
    "text": "like an overhead that is also fixed."
  },
  {
    "startMs": 985279,
    "text": "that also like that changes but it does"
  },
  {
    "startMs": 987680,
    "text": "not change that much u overall you can"
  },
  {
    "startMs": 990959,
    "text": "assume it's fixed and then there is like"
  },
  {
    "startMs": 993759,
    "text": "a leftover memory so this leftover"
  },
  {
    "startMs": 996399,
    "text": "memory is what being used by your KB me"
  },
  {
    "startMs": 999279,
    "text": "like key and the value vectors so"
  },
  {
    "startMs": 1004399,
    "text": "assume like you have a one user you can"
  },
  {
    "startMs": 1008560,
    "text": "only serve that many key and the value"
  },
  {
    "startMs": 1011680,
    "text": "vectors or that many tokens which can"
  },
  {
    "startMs": 1014480,
    "text": "like fit in this entire 80GB uh like"
  },
  {
    "startMs": 1018399,
    "text": "memory that is left."
  },
  {
    "startMs": 1022959,
    "text": "Uh"
  },
  {
    "startMs": 1024720,
    "text": "so we can show this with a simple demo"
  },
  {
    "startMs": 1027919,
    "text": "too. Uh"
  },
  {
    "startMs": 1033199,
    "text": "okay, let me"
  },
  {
    "startMs": 1038959,
    "text": "Okay, great."
  },
  {
    "startMs": 1043120,
    "text": "Okay, great."
  },
  {
    "startMs": 1045600,
    "text": "Uh, let me see if I can actually run"
  },
  {
    "startMs": 1049120,
    "text": "this."
  },
  {
    "startMs": 1051520,
    "text": "What"
  },
  {
    "startMs": 1056480,
    "text": "probably?"
  },
  {
    "startMs": 1058799,
    "text": "Where the heck is this happening?"
  },
  {
    "startMs": 1069840,
    "text": "Okay, great."
  },
  {
    "startMs": 1072640,
    "text": "Yeah. So you would see like the GPU is"
  },
  {
    "startMs": 1075280,
    "text": "attached."
  },
  {
    "startMs": 1087600,
    "text": "So here we are just trying to confirm"
  },
  {
    "startMs": 1089840,
    "text": "the like memory based on the maths and"
  },
  {
    "startMs": 1091919,
    "text": "based on the intuition that we have"
  },
  {
    "startMs": 1093840,
    "text": "built. So the model memory is like let's"
  },
  {
    "startMs": 1096400,
    "text": "say if you have a 7 billion parameters"
  },
  {
    "startMs": 1098400,
    "text": "you are doing a 16 bit precision your"
  },
  {
    "startMs": 1101120,
    "text": "total memory comes out to be 14.6 GB"
  },
  {
    "startMs": 1103919,
    "text": "with you can basically verify that with"
  },
  {
    "startMs": 1106960,
    "text": "the maths. So if you do all that math"
  },
  {
    "startMs": 1110160,
    "text": "that comes out to be the 14.6GB."
  },
  {
    "startMs": 1113919,
    "text": "Uh now comes the KVK and the KV size. So"
  },
  {
    "startMs": 1118080,
    "text": "this KV size is like your 131 KB per"
  },
  {
    "startMs": 1120960,
    "text": "token."
  },
  {
    "startMs": 1122559,
    "text": "Uh and if you do that maths and you try"
  },
  {
    "startMs": 1125360,
    "text": "to like visualize this."
  },
  {
    "startMs": 1129232,
    "text": "[sighs]"
  },
  {
    "startMs": 1132160,
    "text": "Oh, sure."
  },
  {
    "startMs": 1140640,
    "text": "Wait."
  },
  {
    "startMs": 1143840,
    "text": "Okay. And then let's just visualize"
  },
  {
    "startMs": 1147520,
    "text": "this."
  },
  {
    "startMs": 1149919,
    "text": "Okay. Great."
  },
  {
    "startMs": 1151840,
    "text": "Uh yeah so this is the like a memory"
  },
  {
    "startMs": 1154799,
    "text": "chart. So if you see like as your"
  },
  {
    "startMs": 1157120,
    "text": "context increases your memory keeps"
  },
  {
    "startMs": 1159760,
    "text": "increasing."
  },
  {
    "startMs": 1161679,
    "text": "Then another thing to realize is like as"
  },
  {
    "startMs": 1164240,
    "text": "your users increase"
  },
  {
    "startMs": 1166720,
    "text": "uh then also your memory increases. So"
  },
  {
    "startMs": 1169280,
    "text": "if you want to serve like 160 users"
  },
  {
    "startMs": 1173440,
    "text": "uh on a GPU you can support like uh you"
  },
  {
    "startMs": 1177200,
    "text": "can only support like a lesser context"
  },
  {
    "startMs": 1179360,
    "text": "length. So there is always a tradeoff"
  },
  {
    "startMs": 1182000,
    "text": "between what context length you can"
  },
  {
    "startMs": 1184640,
    "text": "serve versus how much cost you can save"
  },
  {
    "startMs": 1188080,
    "text": "by like putting your multiple"
  },
  {
    "startMs": 1191039,
    "text": "uh users or the concurrent users into"
  },
  {
    "startMs": 1193360,
    "text": "like a single GPU. So you have to always"
  },
  {
    "startMs": 1196720,
    "text": "take that tradeoff and and we will go"
  },
  {
    "startMs": 1198880,
    "text": "through that like uh in couple of more"
  },
  {
    "startMs": 1201600,
    "text": "slides."
  },
  {
    "startMs": 1206320,
    "text": ">> Uh can you repeat please?"
  },
  {
    "startMs": 1214400,
    "text": "Uh I'm sorry I'm cannot hear you."
  },
  {
    "startMs": 1224880,
    "text": ">> Yeah."
  },
  {
    "startMs": 1233200,
    "text": "Cool."
  },
  {
    "startMs": 1235039,
    "text": "Uh okay. Great. So let me pull back."
  },
  {
    "startMs": 1241120,
    "text": "So that was like memory. Uh we need to"
  },
  {
    "startMs": 1244559,
    "text": "understand uh why we had like uh slower"
  },
  {
    "startMs": 1248880,
    "text": "time to first token"
  },
  {
    "startMs": 1251600,
    "text": "uh when we increase the context length."
  },
  {
    "startMs": 1254240,
    "text": "So for that like we need to understand"
  },
  {
    "startMs": 1256720,
    "text": "the two phases of inference and those"
  },
  {
    "startMs": 1259120,
    "text": "phases are like the prefill and the"
  },
  {
    "startMs": 1260880,
    "text": "decode phase. I think you would all seen"
  },
  {
    "startMs": 1263200,
    "text": "like a lot of articles but we just"
  },
  {
    "startMs": 1265039,
    "text": "wanted to explain it. Uh so when you"
  },
  {
    "startMs": 1269360,
    "text": "send like a lot of uh like when you send"
  },
  {
    "startMs": 1271840,
    "text": "these input tokens"
  },
  {
    "startMs": 1274080,
    "text": "what you want to do is uh you want to"
  },
  {
    "startMs": 1276880,
    "text": "build those key and the value vectors"
  },
  {
    "startMs": 1278559,
    "text": "that I mentioned for all the tokens."
  },
  {
    "startMs": 1281440,
    "text": "Then you want to compute the attention"
  },
  {
    "startMs": 1283440,
    "text": "scores of every token with respect to"
  },
  {
    "startMs": 1286080,
    "text": "the previous token. All this operation"
  },
  {
    "startMs": 1288799,
    "text": "that you do it's a very very metricsh"
  },
  {
    "startMs": 1291120,
    "text": "heavy uh it's a very very compute heavy"
  },
  {
    "startMs": 1294000,
    "text": "operation and we all know like the GPUs"
  },
  {
    "startMs": 1297520,
    "text": "they are like very well suited for a"
  },
  {
    "startMs": 1300159,
    "text": "heavy compute workload so we call like a"
  },
  {
    "startMs": 1304559,
    "text": "prefill to be like a compute bound uh"
  },
  {
    "startMs": 1307679,
    "text": "and it does take some time to complete"
  },
  {
    "startMs": 1310480,
    "text": "so whatever time that this phase takes"
  },
  {
    "startMs": 1312880,
    "text": "to complete that's your time to the"
  },
  {
    "startMs": 1315200,
    "text": "first token"
  },
  {
    "startMs": 1317120,
    "text": "So if if you have like a more input"
  },
  {
    "startMs": 1319200,
    "text": "tokens, you have to generate more key"
  },
  {
    "startMs": 1322240,
    "text": "value vectors. You have to do lot more"
  },
  {
    "startMs": 1324960,
    "text": "attention math and because of that your"
  },
  {
    "startMs": 1328480,
    "text": "TTFT becomes more more slower."
  },
  {
    "startMs": 1332640,
    "text": "Where is if once you generate one token"
  },
  {
    "startMs": 1335440,
    "text": "you need to keep doing this to generate"
  },
  {
    "startMs": 1337760,
    "text": "another tokens sequentially one after"
  },
  {
    "startMs": 1340080,
    "text": "another. But in that process every time"
  },
  {
    "startMs": 1343760,
    "text": "you have to build the key and the value"
  },
  {
    "startMs": 1346480,
    "text": "vectors of all the previous tokens"
  },
  {
    "startMs": 1349440,
    "text": "which is same as prefill like you were"
  },
  {
    "startMs": 1351200,
    "text": "building key value vectors there also"
  },
  {
    "startMs": 1353120,
    "text": "here also but in decode phase you are"
  },
  {
    "startMs": 1356640,
    "text": "only computing the attention math for"
  },
  {
    "startMs": 1358960,
    "text": "the new token"
  },
  {
    "startMs": 1361039,
    "text": "and that is why it's a very less it's"
  },
  {
    "startMs": 1364080,
    "text": "lesser compute oriented"
  },
  {
    "startMs": 1366720,
    "text": "and it's also called as memory bound. We"
  },
  {
    "startMs": 1368720,
    "text": "will see it shortly why it's called as"
  },
  {
    "startMs": 1370799,
    "text": "memory bound."
  },
  {
    "startMs": 1374960,
    "text": "So in a classic timeline you would see"
  },
  {
    "startMs": 1378240,
    "text": "prefill and decode phase like this. So"
  },
  {
    "startMs": 1381120,
    "text": "time taken by prefill that's your time"
  },
  {
    "startMs": 1383120,
    "text": "to first token and then your time taken"
  },
  {
    "startMs": 1386400,
    "text": "by every decode step that's your"
  },
  {
    "startMs": 1390720,
    "text": "uh basically your inter token latency."
  },
  {
    "startMs": 1393760,
    "text": "So that's like the fourth metric"
  },
  {
    "startMs": 1396720,
    "text": "uh that you need to worry about like"
  },
  {
    "startMs": 1399280,
    "text": "what's the time being taken by your"
  },
  {
    "startMs": 1401120,
    "text": "decode step."
  },
  {
    "startMs": 1405760,
    "text": "Okay, cool. Uh"
  },
  {
    "startMs": 1408720,
    "text": "now why why does the like decode step or"
  },
  {
    "startMs": 1412480,
    "text": "why does decode takes time and why it's"
  },
  {
    "startMs": 1416400,
    "text": "being called as like a memory bound"
  },
  {
    "startMs": 1418320,
    "text": "operation? Let's try to understand that."
  },
  {
    "startMs": 1420960,
    "text": "uh to understand that we need to look at"
  },
  {
    "startMs": 1424640,
    "text": "how the metrics map basically works on"
  },
  {
    "startMs": 1426799,
    "text": "the GPU on a high level. So GPU has two"
  },
  {
    "startMs": 1429679,
    "text": "kind of memories. You have a high"
  },
  {
    "startMs": 1431919,
    "text": "bandwidth memory. You have a shared"
  },
  {
    "startMs": 1434799,
    "text": "memory."
  },
  {
    "startMs": 1436480,
    "text": "So the high bandwidth memory is a larger"
  },
  {
    "startMs": 1438559,
    "text": "size but a lower me like lower"
  },
  {
    "startMs": 1440720,
    "text": "bandwidth."
  },
  {
    "startMs": 1442240,
    "text": "By lower bandwidth I mean like you can"
  },
  {
    "startMs": 1444480,
    "text": "transfer data out of it at a lower rate"
  },
  {
    "startMs": 1447440,
    "text": "compared to the shared memory. So the"
  },
  {
    "startMs": 1449919,
    "text": "shared memory is smaller in size but it"
  },
  {
    "startMs": 1452400,
    "text": "has a very very high bandwidth. Uh that"
  },
  {
    "startMs": 1455039,
    "text": "means you can transfer data in and out"
  },
  {
    "startMs": 1457279,
    "text": "of it with a very first thing. So when a"
  },
  {
    "startMs": 1460000,
    "text": "when you have to do a metric math so you"
  },
  {
    "startMs": 1463279,
    "text": "have to pick the data in chunks from the"
  },
  {
    "startMs": 1466480,
    "text": "high bandwidth memory you have to put it"
  },
  {
    "startMs": 1468480,
    "text": "into the shared memory."
  },
  {
    "startMs": 1471039,
    "text": "Do that math write back the result into"
  },
  {
    "startMs": 1473840,
    "text": "the high bandwidth memory."
  },
  {
    "startMs": 1477840,
    "text": "uh for the prefill phase when you have"
  },
  {
    "startMs": 1482159,
    "text": "to do this you have to do this matrix"
  },
  {
    "startMs": 1485039,
    "text": "math only once but for the decode phase"
  },
  {
    "startMs": 1488080,
    "text": "you have to do this metric math uh like"
  },
  {
    "startMs": 1493520,
    "text": "uh again and again because you're"
  },
  {
    "startMs": 1495679,
    "text": "generating each and every token"
  },
  {
    "startMs": 1497520,
    "text": "sequentially"
  },
  {
    "startMs": 1499520,
    "text": "and so like you it doesn't matter like"
  },
  {
    "startMs": 1502799,
    "text": "how fast is your decode"
  },
  {
    "startMs": 1505600,
    "text": "because now you can transfer your data"
  },
  {
    "startMs": 1508320,
    "text": "out of the high bandwidth memory into"
  },
  {
    "startMs": 1511279,
    "text": "the S shared memory at a certain speed"
  },
  {
    "startMs": 1514880,
    "text": "because you are limited by the high"
  },
  {
    "startMs": 1516480,
    "text": "bandwidth memory bandwidth speed"
  },
  {
    "startMs": 1519919,
    "text": "and so that governs your like token"
  },
  {
    "startMs": 1522240,
    "text": "sealing like at what rate can you"
  },
  {
    "startMs": 1524880,
    "text": "actually generate tokens out of the"
  },
  {
    "startMs": 1527120,
    "text": "decode step."
  },
  {
    "startMs": 1532559,
    "text": "If you look at this in the roof line"
  },
  {
    "startMs": 1535039,
    "text": "plot"
  },
  {
    "startMs": 1537120,
    "text": "uh so there is a left section which is"
  },
  {
    "startMs": 1541360,
    "text": "called to be a memory bound."
  },
  {
    "startMs": 1543279,
    "text": "Mathematically it's governed by the"
  },
  {
    "startMs": 1545120,
    "text": "arithmetic intensity."
  },
  {
    "startMs": 1547760,
    "text": "Arithmetic intensity is the number of"
  },
  {
    "startMs": 1550400,
    "text": "flip-flop operations that you perform"
  },
  {
    "startMs": 1552799,
    "text": "per bite of data being transferred. So"
  },
  {
    "startMs": 1555679,
    "text": "for the decode step"
  },
  {
    "startMs": 1558480,
    "text": "uh decode step since you are"
  },
  {
    "startMs": 1561520,
    "text": "transferring lot of data like the key"
  },
  {
    "startMs": 1564640,
    "text": "and the value vectors of the all the"
  },
  {
    "startMs": 1566480,
    "text": "previous tokens the model weights but"
  },
  {
    "startMs": 1568799,
    "text": "you are doing the like less computation"
  },
  {
    "startMs": 1570960,
    "text": "because you're computing attention math"
  },
  {
    "startMs": 1572640,
    "text": "for only one token. Uh so it's"
  },
  {
    "startMs": 1576400,
    "text": "arithmetic intensity is very low but for"
  },
  {
    "startMs": 1579600,
    "text": "a prefill phase you are transferring the"
  },
  {
    "startMs": 1581520,
    "text": "data once but then like you are doing"
  },
  {
    "startMs": 1585360,
    "text": "this heavy computation and so it's"
  },
  {
    "startMs": 1588640,
    "text": "arithmetic intensity is very high. So"
  },
  {
    "startMs": 1590720,
    "text": "now you know like in terms of"
  },
  {
    "startMs": 1592559,
    "text": "mathematics like why the computer like"
  },
  {
    "startMs": 1596400,
    "text": "why the arithmetic intensity of prefill"
  },
  {
    "startMs": 1598559,
    "text": "is very high compared to your decor."
  },
  {
    "startMs": 1603440,
    "text": "Uh [snorts]"
  },
  {
    "startMs": 1604159,
    "text": "okay"
  },
  {
    "startMs": 1606880,
    "text": "so this is like another small small"
  },
  {
    "startMs": 1609520,
    "text": "demo. Uh"
  },
  {
    "startMs": 1612000,
    "text": "every time I have to Okay."
  },
  {
    "startMs": 1617679,
    "text": "Okay, great."
  },
  {
    "startMs": 1620000,
    "text": "I hope this is already running. So yeah,"
  },
  {
    "startMs": 1623600,
    "text": "again we are loading the model."
  },
  {
    "startMs": 1627520,
    "text": "Now this is the like the prefill cost."
  },
  {
    "startMs": 1630000,
    "text": "So what we are basically doing is uh we"
  },
  {
    "startMs": 1632320,
    "text": "are getting the like um the input uh"
  },
  {
    "startMs": 1635760,
    "text": "text and then we are trying to generate"
  },
  {
    "startMs": 1638480,
    "text": "this um the prefill step the amount of"
  },
  {
    "startMs": 1642000,
    "text": "time it takes. We see like as we"
  },
  {
    "startMs": 1644080,
    "text": "increase the like size of the input"
  },
  {
    "startMs": 1646720,
    "text": "tokens this prefill is increasing. So"
  },
  {
    "startMs": 1649600,
    "text": "you and this is the reason why your TDF"
  },
  {
    "startMs": 1652080,
    "text": "increases"
  },
  {
    "startMs": 1654320,
    "text": "and then like your decode time. So the"
  },
  {
    "startMs": 1657520,
    "text": "decode time is like on average it stays"
  },
  {
    "startMs": 1660000,
    "text": "about the same. Uh and so if it is like"
  },
  {
    "startMs": 1664159,
    "text": "assuming like you ignore the like cold"
  },
  {
    "startMs": 1666240,
    "text": "start your decode time is like"
  },
  {
    "startMs": 1668240,
    "text": "approximately around the average line."
  },
  {
    "startMs": 1670799,
    "text": "it it it it is still impacted by like u"
  },
  {
    "startMs": 1676000,
    "text": "the input size. It's not like it's a"
  },
  {
    "startMs": 1678720,
    "text": "constant uh time and it is because it"
  },
  {
    "startMs": 1682159,
    "text": "still needs to pull the key and the"
  },
  {
    "startMs": 1683760,
    "text": "value vectors from the memory for all"
  },
  {
    "startMs": 1685840,
    "text": "the previous tokens. So there is still"
  },
  {
    "startMs": 1688880,
    "text": "like uh that uh basically small increase"
  },
  {
    "startMs": 1692720,
    "text": "in time that you would see with the"
  },
  {
    "startMs": 1694240,
    "text": "decode step. And then this is the like"
  },
  {
    "startMs": 1697600,
    "text": "classic uh roof line plot."
  },
  {
    "startMs": 1701919,
    "text": "Uh okay."
  },
  {
    "startMs": 1707120,
    "text": "Presentation."
  },
  {
    "startMs": 1711279,
    "text": "Okay. Five. Okay. Great."
  },
  {
    "startMs": 1715600,
    "text": "Okay. So"
  },
  {
    "startMs": 1718640,
    "text": "now now let's try to understand like uh"
  },
  {
    "startMs": 1721760,
    "text": "the throughput dimension. You want to"
  },
  {
    "startMs": 1724559,
    "text": "understand how many users you can"
  },
  {
    "startMs": 1726880,
    "text": "actually serve and I think we saw like a"
  },
  {
    "startMs": 1729919,
    "text": "diagram of the GPU memory where we saw"
  },
  {
    "startMs": 1732240,
    "text": "okay there is some memory that is free"
  },
  {
    "startMs": 1734080,
    "text": "for the key and the value vectors to"
  },
  {
    "startMs": 1735840,
    "text": "grow."
  },
  {
    "startMs": 1737520,
    "text": "So assume like you have just a single"
  },
  {
    "startMs": 1740799,
    "text": "user"
  },
  {
    "startMs": 1742960,
    "text": "uh"
  },
  {
    "startMs": 1744799,
    "text": "what's the total KV size that you have"
  },
  {
    "startMs": 1747600,
    "text": "you can basically support it's defined"
  },
  {
    "startMs": 1749840,
    "text": "by your context limit."
  },
  {
    "startMs": 1752480,
    "text": "uh the max users that you can support is"
  },
  {
    "startMs": 1755360,
    "text": "like whatever is your GPU uh"
  },
  {
    "startMs": 1758000,
    "text": "availability like whatever is the memory"
  },
  {
    "startMs": 1760080,
    "text": "that is available in the GPU you divide"
  },
  {
    "startMs": 1762559,
    "text": "it by the key and the value size per"
  },
  {
    "startMs": 1764880,
    "text": "user uh and when you do that like it"
  },
  {
    "startMs": 1768080,
    "text": "comes out to be like your the concurrent"
  },
  {
    "startMs": 1770640,
    "text": "users."
  },
  {
    "startMs": 1773120,
    "text": "Now assume like your GPU is fixed, your"
  },
  {
    "startMs": 1778080,
    "text": "model is fixed."
  },
  {
    "startMs": 1780320,
    "text": "Uh"
  },
  {
    "startMs": 1781840,
    "text": "so"
  },
  {
    "startMs": 1783360,
    "text": "your KV size per token is fixed. There"
  },
  {
    "startMs": 1786960,
    "text": "are only two dimensions that are left"
  },
  {
    "startMs": 1788799,
    "text": "here which is context and your"
  },
  {
    "startMs": 1791200,
    "text": "concurrent users."
  },
  {
    "startMs": 1793360,
    "text": "If you want to serve more concurrent"
  },
  {
    "startMs": 1795600,
    "text": "users, you have to reduce the context"
  },
  {
    "startMs": 1797440,
    "text": "length. If you reduce the context"
  },
  {
    "startMs": 1799039,
    "text": "length, you could impact your quality."
  },
  {
    "startMs": 1802480,
    "text": "Uh so these are the two dimensions right"
  },
  {
    "startMs": 1805039,
    "text": "now that we are trading off."
  },
  {
    "startMs": 1809279,
    "text": "Then if we but can you actually serve"
  },
  {
    "startMs": 1812960,
    "text": "the like max number of concurrent users?"
  },
  {
    "startMs": 1817520,
    "text": "Uh in an ideal world probably not"
  },
  {
    "startMs": 1820080,
    "text": "because"
  },
  {
    "startMs": 1822080,
    "text": "every business has like a latency SLO"
  },
  {
    "startMs": 1825279,
    "text": "that we have to meet."
  },
  {
    "startMs": 1828159,
    "text": "So"
  },
  {
    "startMs": 1830159,
    "text": "if you remember like in the decode step"
  },
  {
    "startMs": 1832720,
    "text": "I said the time for the decode still"
  },
  {
    "startMs": 1835360,
    "text": "increases if you have more inputs."
  },
  {
    "startMs": 1838159,
    "text": "It also increases if you have more"
  },
  {
    "startMs": 1839840,
    "text": "users."
  },
  {
    "startMs": 1841440,
    "text": "So ultimately"
  },
  {
    "startMs": 1844159,
    "text": "uh your inter token latency also gets"
  },
  {
    "startMs": 1847760,
    "text": "impacted"
  },
  {
    "startMs": 1849520,
    "text": "if you have like a higher batch size and"
  },
  {
    "startMs": 1851679,
    "text": "your TTF also gets impacted. So now"
  },
  {
    "startMs": 1854559,
    "text": "there is a third dimension you have to"
  },
  {
    "startMs": 1856240,
    "text": "worry about which is like your latency."
  },
  {
    "startMs": 1858799,
    "text": "So the three dimensions that you have is"
  },
  {
    "startMs": 1861120,
    "text": "like a quality latency and the"
  },
  {
    "startMs": 1863600,
    "text": "throughput. So it comes out to be like"
  },
  {
    "startMs": 1865919,
    "text": "this trade-off triangle where you have"
  },
  {
    "startMs": 1868720,
    "text": "to choose between the two. So for a"
  },
  {
    "startMs": 1872080,
    "text": "premium chat application"
  },
  {
    "startMs": 1875200,
    "text": "you would want to prioritize definitely"
  },
  {
    "startMs": 1877120,
    "text": "the quality and you want to prioritize"
  },
  {
    "startMs": 1879120,
    "text": "the like the latency. You would not want"
  },
  {
    "startMs": 1882720,
    "text": "your users to wait infinitely for the"
  },
  {
    "startMs": 1885120,
    "text": "like or like not infinitely but probably"
  },
  {
    "startMs": 1888480,
    "text": "for the larger latency."
  },
  {
    "startMs": 1891200,
    "text": "You can always sacrifice the number of"
  },
  {
    "startMs": 1893520,
    "text": "users you can support on the GPU and"
  },
  {
    "startMs": 1895760,
    "text": "probably take that costed being more"
  },
  {
    "startMs": 1898000,
    "text": "customers"
  },
  {
    "startMs": 1900480,
    "text": "in form of like and and like if you"
  },
  {
    "startMs": 1903679,
    "text": "consider like an agent uh sorry the"
  },
  {
    "startMs": 1906240,
    "text": "async agent workload you would want to"
  },
  {
    "startMs": 1909039,
    "text": "like prioritize definitely quality and"
  },
  {
    "startMs": 1910880,
    "text": "the throughput"
  },
  {
    "startMs": 1912880,
    "text": "uh because these are the longunning"
  },
  {
    "startMs": 1915360,
    "text": "tasks"
  },
  {
    "startMs": 1916880,
    "text": "uh and you would want to like serve as"
  },
  {
    "startMs": 1919600,
    "text": "many as concurrent tasks. fast as"
  },
  {
    "startMs": 1921200,
    "text": "possible but with a very very higher"
  },
  {
    "startMs": 1923279,
    "text": "quality."
  },
  {
    "startMs": 1927840,
    "text": "And often like we think like okay if the"
  },
  {
    "startMs": 1931440,
    "text": "GPU is like a very expensive GPU"
  },
  {
    "startMs": 1935279,
    "text": "uh that might not be a good fit for us."
  },
  {
    "startMs": 1939039,
    "text": "Uh but it turns out that could actually"
  },
  {
    "startMs": 1941360,
    "text": "serve you the lowest cost per million uh"
  },
  {
    "startMs": 1944720,
    "text": "tokens."
  },
  {
    "startMs": 1947519,
    "text": "Uh but you really have to trust your"
  },
  {
    "startMs": 1950960,
    "text": "kind of calculations on the max users"
  },
  {
    "startMs": 1953120,
    "text": "that you want and like uh you really"
  },
  {
    "startMs": 1956080,
    "text": "have to make those estimations uh"
  },
  {
    "startMs": 1959519,
    "text": "correctly."
  },
  {
    "startMs": 1963840,
    "text": "Uh"
  },
  {
    "startMs": 1969679,
    "text": "so we do have like uh"
  },
  {
    "startMs": 1975120,
    "text": "let me just"
  },
  {
    "startMs": 1979679,
    "text": "Where is this?"
  },
  {
    "startMs": 1982320,
    "text": "Okay, great. So, for the capacity"
  },
  {
    "startMs": 1985519,
    "text": "calculator, uh there is like a link to"
  },
  {
    "startMs": 1988559,
    "text": "the collab because I was facing certain"
  },
  {
    "startMs": 1991120,
    "text": "issues with molab. I had to migrate out"
  },
  {
    "startMs": 1993679,
    "text": "the wall widget library and I didn't"
  },
  {
    "startMs": 1996559,
    "text": "have time. So, being lazy, I just picked"
  },
  {
    "startMs": 1999600,
    "text": "collab there. Uh apologies to Moab."
  },
  {
    "startMs": 2005519,
    "text": "Uh"
  },
  {
    "startMs": 2007360,
    "text": "so my VR is connected."
  },
  {
    "startMs": 2015840,
    "text": "Okay."
  },
  {
    "startMs": 2026080,
    "text": "Wi-Fi probably."
  },
  {
    "startMs": 2031919,
    "text": "Okay. Great."
  },
  {
    "startMs": 2042159,
    "text": "So what we have done over here is we"
  },
  {
    "startMs": 2044320,
    "text": "have like shaded some like the GPUs with"
  },
  {
    "startMs": 2047360,
    "text": "their V RAMs, bandwidths, the flip-flops"
  },
  {
    "startMs": 2050240,
    "text": "and the cost per hours. Um"
  },
  {
    "startMs": 2054639,
    "text": "then we kind of like built this simple"
  },
  {
    "startMs": 2057280,
    "text": "uh like uh capacity calculator. This is"
  },
  {
    "startMs": 2060639,
    "text": "just a KV visualizer uh where you kind"
  },
  {
    "startMs": 2063760,
    "text": "of like when you increase the number of"
  },
  {
    "startMs": 2065679,
    "text": "tokens uh you see like your KV size it"
  },
  {
    "startMs": 2068480,
    "text": "increases and when you increase the"
  },
  {
    "startMs": 2071599,
    "text": "number of users your size is like"
  },
  {
    "startMs": 2073760,
    "text": "increasing at a much faster rate"
  },
  {
    "startMs": 2077919,
    "text": "and then"
  },
  {
    "startMs": 2080960,
    "text": "in this capacity calculator uh"
  },
  {
    "startMs": 2086639,
    "text": "let it run."
  },
  {
    "startMs": 2089599,
    "text": "So we have like a model which we which"
  },
  {
    "startMs": 2092800,
    "text": "is like a 7 billion parameter model that"
  },
  {
    "startMs": 2095440,
    "text": "we selected."
  },
  {
    "startMs": 2098079,
    "text": "We set the like precision to be FP16. Uh"
  },
  {
    "startMs": 2101760,
    "text": "now we decide the way we basically go by"
  },
  {
    "startMs": 2104240,
    "text": "the GPU decision is you have to decide"
  },
  {
    "startMs": 2108000,
    "text": "what's your like you have to fix one"
  },
  {
    "startMs": 2111119,
    "text": "dimension first which you care about the"
  },
  {
    "startMs": 2113920,
    "text": "most."
  },
  {
    "startMs": 2115599,
    "text": "for premium chat I mentioned like"
  },
  {
    "startMs": 2117359,
    "text": "latency is definitely the one"
  },
  {
    "startMs": 2121040,
    "text": "uh and then like for the async workloads"
  },
  {
    "startMs": 2123520,
    "text": "the batch the minimum batch size that"
  },
  {
    "startMs": 2125760,
    "text": "you want to serve for from like a single"
  },
  {
    "startMs": 2128720,
    "text": "GPU that is the second dimension so you"
  },
  {
    "startMs": 2131680,
    "text": "want to fix these first so I will go"
  },
  {
    "startMs": 2134800,
    "text": "about like in a premium chat application"
  },
  {
    "startMs": 2138160,
    "text": "uh"
  },
  {
    "startMs": 2139680,
    "text": "so I can go ahead with like 10"
  },
  {
    "startMs": 2141839,
    "text": "milliseconds latency a minimum batch"
  },
  {
    "startMs": 2145359,
    "text": "size I don't care like I can so I'm okay"
  },
  {
    "startMs": 2147839,
    "text": "with like probably two"
  },
  {
    "startMs": 2150720,
    "text": "uh"
  },
  {
    "startMs": 2153680,
    "text": "okay so probably with the seven"
  },
  {
    "startMs": 2156240,
    "text": "concurrent users on a single GPU and"
  },
  {
    "startMs": 2159200,
    "text": "then like my context limit is very"
  },
  {
    "startMs": 2161200,
    "text": "important to me because I want to focus"
  },
  {
    "startMs": 2163040,
    "text": "on the quality as well"
  },
  {
    "startMs": 2166400,
    "text": "uh and so like I do see like some of the"
  },
  {
    "startMs": 2169440,
    "text": "GPUs so the H18GB"
  },
  {
    "startMs": 2172960,
    "text": "it's like a $8 per hour but like am I"
  },
  {
    "startMs": 2178000,
    "text": "300x is it? Yeah. So it's like around"
  },
  {
    "startMs": 2181680,
    "text": "$10 per hour but if you do all that"
  },
  {
    "startMs": 2186240,
    "text": "throughput math that we shared in the"
  },
  {
    "startMs": 2188880,
    "text": "mathematics before you could find like"
  },
  {
    "startMs": 2191359,
    "text": "your cost per million dollar tokens that"
  },
  {
    "startMs": 2194240,
    "text": "could be very very that could be like"
  },
  {
    "startMs": 2196480,
    "text": "lesser. So"
  },
  {
    "startMs": 2199200,
    "text": "you need to do such calculations by"
  },
  {
    "startMs": 2201680,
    "text": "fixing those dimensions and you need to"
  },
  {
    "startMs": 2204079,
    "text": "decide your GPU to like reduce your kind"
  },
  {
    "startMs": 2207760,
    "text": "of inference cost. This is at least the"
  },
  {
    "startMs": 2210079,
    "text": "first step that you can take towards"
  },
  {
    "startMs": 2211920,
    "text": "optimizing the inference."
  },
  {
    "startMs": 2216720,
    "text": "Okay, cool."
  },
  {
    "startMs": 2221520,
    "text": "So the next slide. So let me"
  },
  {
    "startMs": 2228320,
    "text": "Okay, great."
  },
  {
    "startMs": 2230560,
    "text": "And so like now the next thing is about"
  },
  {
    "startMs": 2233280,
    "text": "the model optimization. So we are now"
  },
  {
    "startMs": 2235760,
    "text": "basically have built that foundation"
  },
  {
    "startMs": 2237599,
    "text": "where we understood some of the pain"
  },
  {
    "startMs": 2239440,
    "text": "points, reason behind those pain points,"
  },
  {
    "startMs": 2241520,
    "text": "why those were happening"
  },
  {
    "startMs": 2243920,
    "text": "um how we could like address that GPU"
  },
  {
    "startMs": 2248000,
    "text": "capacity thing. We need to understand"
  },
  {
    "startMs": 2250560,
    "text": "what can we do like what can we further"
  },
  {
    "startMs": 2252720,
    "text": "do about it. So it it is about the model"
  },
  {
    "startMs": 2256240,
    "text": "optimization and I think I would like to"
  },
  {
    "startMs": 2258560,
    "text": "invite Tan I he can talk more about"
  },
  {
    "startMs": 2260880,
    "text": "these model optimizations provided he"
  },
  {
    "startMs": 2263680,
    "text": "has worked uh on this like during his"
  },
  {
    "startMs": 2266240,
    "text": "research times"
  },
  {
    "startMs": 2268640,
    "text": "okay I can control"
  },
  {
    "startMs": 2273920,
    "text": "yeah here okay hi everyone uh mic check"
  },
  {
    "startMs": 2278240,
    "text": "am I audible at last yeah okay so hi I'm"
  },
  {
    "startMs": 2282720,
    "text": "Tesha I work as a senior quant modeler"
  },
  {
    "startMs": 2285680,
    "text": "and also I am an AI researcher. My work"
  },
  {
    "startMs": 2288160,
    "text": "focuses on a agent verification and"
  },
  {
    "startMs": 2290880,
    "text": "right now building world models. So for"
  },
  {
    "startMs": 2293920,
    "text": "this one model optimization"
  },
  {
    "startMs": 2296640,
    "text": "before we start model optimization so I"
  },
  {
    "startMs": 2299119,
    "text": "created a research template so that it"
  },
  {
    "startMs": 2301280,
    "text": "will be easy for us to understand all"
  },
  {
    "startMs": 2303200,
    "text": "these complex things. I so our template"
  },
  {
    "startMs": 2307280,
    "text": "is simple. First we will identify the"
  },
  {
    "startMs": 2309359,
    "text": "problem. Second step we will solve the"
  },
  {
    "startMs": 2312800,
    "text": "problem using two algorithms. These are"
  },
  {
    "startMs": 2314800,
    "text": "just fake algorithms. So first algorithm"
  },
  {
    "startMs": 2316880,
    "text": "is called ostrich algorithm. Whenever we"
  },
  {
    "startMs": 2320079,
    "text": "see uh just like ostrich whenever we see"
  },
  {
    "startMs": 2322800,
    "text": "a problem ostrich put their head into"
  },
  {
    "startMs": 2325839,
    "text": "the sand. So same thing we will do"
  },
  {
    "startMs": 2327760,
    "text": "whenever we face a problem we will just"
  },
  {
    "startMs": 2330240,
    "text": "ignore it. So this is an important"
  },
  {
    "startMs": 2332800,
    "text": "algorithm we should follow. Second one"
  },
  {
    "startMs": 2335680,
    "text": "is created it is called world cup"
  },
  {
    "startMs": 2337920,
    "text": "algorithm. For example, we don't know"
  },
  {
    "startMs": 2341119,
    "text": "who will win this FIFA World Cup. So,"
  },
  {
    "startMs": 2343599,
    "text": "what organizers did, they uh break the"
  },
  {
    "startMs": 2348400,
    "text": "48 teams into 12 groups, uh then round"
  },
  {
    "startMs": 2352240,
    "text": "32. So, round 32 right now is currently"
  },
  {
    "startMs": 2354800,
    "text": "going on. Uh then round 16, then"
  },
  {
    "startMs": 2357920,
    "text": "quarterfinals, uh then semi-finals and"
  },
  {
    "startMs": 2360720,
    "text": "finals. So what they are doing is that"
  },
  {
    "startMs": 2364079,
    "text": "uh they are breaking it into a smaller"
  },
  {
    "startMs": 2366720,
    "text": "problems and the useful results are"
  },
  {
    "startMs": 2369440,
    "text": "moving forward. So same analogy or same"
  },
  {
    "startMs": 2372560,
    "text": "algorithm we will use uh to understand"
  },
  {
    "startMs": 2375760,
    "text": "this model optimization all those"
  },
  {
    "startMs": 2377760,
    "text": "things. So yeah let's start. So I have"
  },
  {
    "startMs": 2383359,
    "text": "one H100 GPU."
  },
  {
    "startMs": 2386400,
    "text": "I have to use this open-source model"
  },
  {
    "startMs": 2389119,
    "text": "what is called GPTOSS"
  },
  {
    "startMs": 2391760,
    "text": "120 billion parameter model. So right"
  },
  {
    "startMs": 2395200,
    "text": "now I think it's so they have trained it"
  },
  {
    "startMs": 2397440,
    "text": "on BF float 16 and weight is 240 GB."
  },
  {
    "startMs": 2402560,
    "text": "What should I do?"
  },
  {
    "startMs": 2404880,
    "text": "This is the problem we have. So first"
  },
  {
    "startMs": 2407520,
    "text": "thing what we have to deal do is that"
  },
  {
    "startMs": 2410800,
    "text": "240 GB and 80 uh GB H100."
  },
  {
    "startMs": 2416240,
    "text": "So and I have to fit only in one GPU or"
  },
  {
    "startMs": 2419839,
    "text": "not in multiple GPU. So what can we do?"
  },
  {
    "startMs": 2422880,
    "text": "I think simple step is that just"
  },
  {
    "startMs": 2425760,
    "text": "compress it. But how should we compress"
  },
  {
    "startMs": 2429040,
    "text": "it? Uh that's the another challenge. So"
  },
  {
    "startMs": 2431359,
    "text": "if we compress BF BF float 16 to FP8 uh"
  },
  {
    "startMs": 2436240,
    "text": "then it will be around 120 GB but our"
  },
  {
    "startMs": 2439440,
    "text": "GPU H100 is still 80 GB. So what I think"
  },
  {
    "startMs": 2443040,
    "text": "they did is that they compressed it into"
  },
  {
    "startMs": 2445680,
    "text": "further MX uh MX FP4 and I think size is"
  },
  {
    "startMs": 2451040,
    "text": "around 65 GB. So this is something we"
  },
  {
    "startMs": 2455200,
    "text": "can do uh compress but question so and"
  },
  {
    "startMs": 2460000,
    "text": "we will use over this ostrich algorithm"
  },
  {
    "startMs": 2463119,
    "text": "we are assuming that uh there is no loss"
  },
  {
    "startMs": 2465839,
    "text": "in compressing a bigger model into a a"
  },
  {
    "startMs": 2469280,
    "text": "smaller size. Second thing"
  },
  {
    "startMs": 2472960,
    "text": "in uh in this one okay yeah so in this"
  },
  {
    "startMs": 2477119,
    "text": "one in this slide we have used this"
  },
  {
    "startMs": 2478960,
    "text": "mistral 7B so 7 billion parameters so"
  },
  {
    "startMs": 2482240,
    "text": "it's a small model 7 billion parameters"
  },
  {
    "startMs": 2484480,
    "text": "so uh so if you multiply it by two bytes"
  },
  {
    "startMs": 2487920,
    "text": "so it so weight of it's around is 14"
  },
  {
    "startMs": 2491280,
    "text": "14.5 GB which can easily fit into H100"
  },
  {
    "startMs": 2495760,
    "text": "or even a a40 so"
  },
  {
    "startMs": 2499920,
    "text": "so Next uh what we can do is that like"
  },
  {
    "startMs": 2503119,
    "text": "mistral 7B instead of compressing it a"
  },
  {
    "startMs": 2506720,
    "text": "floating point 16 we can apply different"
  },
  {
    "startMs": 2509040,
    "text": "techniques like int 8 or int4 or nf4. So"
  },
  {
    "startMs": 2513280,
    "text": "basically we have to just use ostrich"
  },
  {
    "startMs": 2515920,
    "text": "algorithm and just believe that uh there"
  },
  {
    "startMs": 2518079,
    "text": "is no quality loss kind of things but"
  },
  {
    "startMs": 2521599,
    "text": "somehow we also have to mathematically"
  },
  {
    "startMs": 2523440,
    "text": "prove that by doing some kind of test"
  },
  {
    "startMs": 2525440,
    "text": "testing on some external benchmark that"
  },
  {
    "startMs": 2527520,
    "text": "whether it is working or not. So the and"
  },
  {
    "startMs": 2532480,
    "text": "this comes under post training"
  },
  {
    "startMs": 2534560,
    "text": "quantization kind of thing. One can also"
  },
  {
    "startMs": 2536720,
    "text": "do uh this one uh during finetuning one"
  },
  {
    "startMs": 2540400,
    "text": "can also do this kind of quantization."
  },
  {
    "startMs": 2542560,
    "text": "This comes under a quant training kind"
  },
  {
    "startMs": 2544720,
    "text": "of thing. So uh let's move to our next"
  },
  {
    "startMs": 2548720,
    "text": "problem."
  },
  {
    "startMs": 2551520,
    "text": "So"
  },
  {
    "startMs": 2554560,
    "text": "we have this huge matrices just just"
  },
  {
    "startMs": 2558800,
    "text": "imagine imagine uh 1,000 by 1,000 uh"
  },
  {
    "startMs": 2563760,
    "text": "dimension matrix A and another matrix"
  },
  {
    "startMs": 2567040,
    "text": "matrix um 1,000 by 1,000. So if we"
  },
  {
    "startMs": 2571680,
    "text": "multiply uh if we multiply by this two"
  },
  {
    "startMs": 2574319,
    "text": "matrices so number of operations will be"
  },
  {
    "startMs": 2577200,
    "text": "1,000 raised to the power q and this is"
  },
  {
    "startMs": 2581599,
    "text": "kind of a problem in terms of uh uh in"
  },
  {
    "startMs": 2585280,
    "text": "terms of computing. So we wondered our"
  },
  {
    "startMs": 2587920,
    "text": "matrix multiplication should be fast and"
  },
  {
    "startMs": 2590880,
    "text": "it should save memory. So what should we"
  },
  {
    "startMs": 2594000,
    "text": "do? We have a giant matrix. Okay, let's"
  },
  {
    "startMs": 2598160,
    "text": "take this one. Uh, Mr. 4096 by 4096."
  },
  {
    "startMs": 2602880,
    "text": "What should we do uh to"
  },
  {
    "startMs": 2605920,
    "text": "solve our problem of speeding up the"
  },
  {
    "startMs": 2608400,
    "text": "things and saving the memory 4096 by"
  },
  {
    "startMs": 2611440,
    "text": "4096."
  },
  {
    "startMs": 2613599,
    "text": "So first thing is that we will use just"
  },
  {
    "startMs": 2615440,
    "text": "our world cup algorithm. We can decide a"
  },
  {
    "startMs": 2618800,
    "text": "random number just break the block"
  },
  {
    "startMs": 2621119,
    "text": "vertically. It does not matter what you"
  },
  {
    "startMs": 2623839,
    "text": "are choosing it. So you have so let's"
  },
  {
    "startMs": 2627440,
    "text": "say uh we have 4096 uh columns we will"
  },
  {
    "startMs": 2631839,
    "text": "break it uh we will break it into a"
  },
  {
    "startMs": 2634560,
    "text": "group of 128 column each. So 128, 128, 128"
  },
  {
    "startMs": 2640000,
    "text": "128 uh vertical vert uh vertically so we"
  },
  {
    "startMs": 2644160,
    "text": "will get a 30 we will get this 32 blocks"
  },
  {
    "startMs": 2647040,
    "text": "if we divide this 4096"
  },
  {
    "startMs": 2649680,
    "text": "then"
  },
  {
    "startMs": 2651680,
    "text": "what will happen by doing this thing? So"
  },
  {
    "startMs": 2653680,
    "text": "if we just divide this one vertical"
  },
  {
    "startMs": 2656960,
    "text": "vertically then we can use a multiple"
  },
  {
    "startMs": 2659359,
    "text": "GPU to speed up the process. So this"
  },
  {
    "startMs": 2662240,
    "text": "kind of thing is called multi head"
  },
  {
    "startMs": 2665119,
    "text": "attention."
  },
  {
    "startMs": 2667280,
    "text": "So what else can we do? We have a big"
  },
  {
    "startMs": 2670319,
    "text": "matrix like"
  },
  {
    "startMs": 2673839,
    "text": "uh as I have mentioned that ostrich"
  },
  {
    "startMs": 2675680,
    "text": "algorithm. So our main problem is"
  },
  {
    "startMs": 2679599,
    "text": "sizing. So what we what we can do is"
  },
  {
    "startMs": 2682160,
    "text": "that instead of having all those 32 uh"
  },
  {
    "startMs": 2685920,
    "text": "32 vertical blocks we will throw away uh"
  },
  {
    "startMs": 2688880,
    "text": "31 blocks and we will assume that one"
  },
  {
    "startMs": 2691359,
    "text": "block is sufficient enough that all the"
  },
  {
    "startMs": 2694960,
    "text": "queries uh can handle those blocks. Our"
  },
  {
    "startMs": 2698160,
    "text": "loss will be almost negligible and we"
  },
  {
    "startMs": 2702560,
    "text": "come up with this algorithm uh which uh"
  },
  {
    "startMs": 2704800,
    "text": "and this algorithm is called a"
  },
  {
    "startMs": 2706480,
    "text": "multiquery attention. So as we can see"
  },
  {
    "startMs": 2710079,
    "text": "right now we are at two spectrum. One is"
  },
  {
    "startMs": 2712880,
    "text": "multi head attention where we split it"
  },
  {
    "startMs": 2716000,
    "text": "into 32 blocks and use different uh"
  },
  {
    "startMs": 2720240,
    "text": "different uh GPUs or do some parallel"
  },
  {
    "startMs": 2722720,
    "text": "processing and at the same time we are"
  },
  {
    "startMs": 2725440,
    "text": "just throwing 31 blocks and uh we are"
  },
  {
    "startMs": 2728880,
    "text": "calling this is as a multi-query"
  },
  {
    "startMs": 2731359,
    "text": "attention. So"
  },
  {
    "startMs": 2734560,
    "text": "uh so at both extreme we should be come"
  },
  {
    "startMs": 2736880,
    "text": "up with a middle ground like something"
  },
  {
    "startMs": 2739040,
    "text": "we can say that"
  },
  {
    "startMs": 2741280,
    "text": "instead of throwing all the 31 uh maybe"
  },
  {
    "startMs": 2744240,
    "text": "we can group we can group we can group"
  },
  {
    "startMs": 2746880,
    "text": "some of the blocks together so that uh"
  },
  {
    "startMs": 2750319,
    "text": "uh and we can assume that uh similar"
  },
  {
    "startMs": 2753440,
    "text": "blocks will attend to a um similar kind"
  },
  {
    "startMs": 2757040,
    "text": "of uh queries. So this kind of technique"
  },
  {
    "startMs": 2760400,
    "text": "comes under grouped query attention"
  },
  {
    "startMs": 2762240,
    "text": "which is very popular right now. Uh even"
  },
  {
    "startMs": 2764720,
    "text": "in uh even in mistral or in other models"
  },
  {
    "startMs": 2768640,
    "text": "this grouped query attention works. So"
  },
  {
    "startMs": 2771680,
    "text": "right now we have understand that we"
  },
  {
    "startMs": 2774079,
    "text": "have a big matrix uh we can divide it"
  },
  {
    "startMs": 2776960,
    "text": "the way we want and doing some"
  },
  {
    "startMs": 2778800,
    "text": "mathematical calculation prove that loss"
  },
  {
    "startMs": 2781119,
    "text": "is almost negligible kind of thing. So"
  },
  {
    "startMs": 2783520,
    "text": "what else we can do?"
  },
  {
    "startMs": 2786880,
    "text": "So after that uh after this grouped"
  },
  {
    "startMs": 2789920,
    "text": "query attention"
  },
  {
    "startMs": 2792560,
    "text": "uh"
  },
  {
    "startMs": 2794800,
    "text": "see"
  },
  {
    "startMs": 2796960,
    "text": "uh we have a big matrix uh one is one is"
  },
  {
    "startMs": 2800400,
    "text": "key and one is value."
  },
  {
    "startMs": 2802960,
    "text": "Let's compress that matrix into a latent"
  },
  {
    "startMs": 2806720,
    "text": "vector and then come up with some"
  },
  {
    "startMs": 2809359,
    "text": "algorithm to uh reconstruct from latent"
  },
  {
    "startMs": 2812319,
    "text": "vector uh to our original matrix. So"
  },
  {
    "startMs": 2815839,
    "text": "this kind of a strategy comes under this"
  },
  {
    "startMs": 2818319,
    "text": "one um multi head latent latent"
  },
  {
    "startMs": 2821839,
    "text": "attention but again it has some problems"
  },
  {
    "startMs": 2824079,
    "text": "with rope because rope is position"
  },
  {
    "startMs": 2826400,
    "text": "dependent and uh and it is position"
  },
  {
    "startMs": 2828960,
    "text": "independent kind of thing. So yeah one"
  },
  {
    "startMs": 2831359,
    "text": "needs to also include some uh index for"
  },
  {
    "startMs": 2833760,
    "text": "keys also so that one can map it. But"
  },
  {
    "startMs": 2837760,
    "text": "again main problem is that why why we"
  },
  {
    "startMs": 2841359,
    "text": "are why we are multiplying all those big"
  },
  {
    "startMs": 2844000,
    "text": "matrices. So because that's how this"
  },
  {
    "startMs": 2847359,
    "text": "attention mechanism works that"
  },
  {
    "startMs": 2850720,
    "text": "each token will pay attention to every"
  },
  {
    "startMs": 2853280,
    "text": "token. So how about let's don't pay"
  },
  {
    "startMs": 2856800,
    "text": "attention to all the previous token only"
  },
  {
    "startMs": 2858960,
    "text": "pay attention to the important tokens uh"
  },
  {
    "startMs": 2861359,
    "text": "which is important for us. So this is a"
  },
  {
    "startMs": 2864640,
    "text": "kind of uh this kind of field is uh"
  },
  {
    "startMs": 2867040,
    "text": "evolving. So this comes under sparse uh"
  },
  {
    "startMs": 2869760,
    "text": "deepseek sparse attention. So"
  },
  {
    "startMs": 2873440,
    "text": "uh yeah and yeah yeah yeah so okay next"
  },
  {
    "startMs": 2880240,
    "text": "yeah so next one is flash attention. So"
  },
  {
    "startMs": 2884319,
    "text": "uh so in flash attention so main pro so"
  },
  {
    "startMs": 2887680,
    "text": "main problem is that uh"
  },
  {
    "startMs": 2891200,
    "text": "uh so so currently so so currently not"
  },
  {
    "startMs": 2894560,
    "text": "currently so right now almost everyone"
  },
  {
    "startMs": 2896400,
    "text": "uses flash attention but way in 2022 or"
  },
  {
    "startMs": 2899440,
    "text": "2023 uh so that's how it works that's"
  },
  {
    "startMs": 2903839,
    "text": "how it works is that uh so"
  },
  {
    "startMs": 2908000,
    "text": "uh this"
  },
  {
    "startMs": 2909599,
    "text": "Q K query and A and key matrices they"
  },
  {
    "startMs": 2913839,
    "text": "were in HBM. Uh it loads uh it uh first"
  },
  {
    "startMs": 2918480,
    "text": "uh it loads into uh this one uh tensor"
  },
  {
    "startMs": 2921599,
    "text": "core and it do some uh it do some"
  },
  {
    "startMs": 2924000,
    "text": "calculation and then it will uh write it"
  },
  {
    "startMs": 2926319,
    "text": "back to uh HBM and then uh this process"
  },
  {
    "startMs": 2929760,
    "text": "goes on multiple times. So in flash"
  },
  {
    "startMs": 2932640,
    "text": "attention uh what they did is that"
  },
  {
    "startMs": 2936240,
    "text": "uh is that instead of multiplying the"
  },
  {
    "startMs": 2938880,
    "text": "whole matrices so they just divided it"
  },
  {
    "startMs": 2941200,
    "text": "into like our world cup algorithm"
  },
  {
    "startMs": 2943119,
    "text": "divided the bigger matrices into a small"
  },
  {
    "startMs": 2946000,
    "text": "tile and only put those small tiles uh"
  },
  {
    "startMs": 2948960,
    "text": "into a SBM so that uh it can process"
  },
  {
    "startMs": 2952400,
    "text": "multiplication fast and just uh keep uh"
  },
  {
    "startMs": 2956160,
    "text": "keeping track of this some three"
  },
  {
    "startMs": 2957839,
    "text": "variables so that they can calculate"
  },
  {
    "startMs": 2959839,
    "text": "this online softmax."
  },
  {
    "startMs": 2963599,
    "text": "Yeah."
  },
  {
    "startMs": 2965200,
    "text": "Next one. So, yeah. So, so this is just"
  },
  {
    "startMs": 2968559,
    "text": "mathematics. So, if we have a multi head"
  },
  {
    "startMs": 2971200,
    "text": "attention if it is 524"
  },
  {
    "startMs": 2974000,
    "text": "uh KV uh then it depends upon how much"
  },
  {
    "startMs": 2978079,
    "text": "how much grouping we want and so if"
  },
  {
    "startMs": 2982960,
    "text": "instead of 32 KV head we only want to"
  },
  {
    "startMs": 2986079,
    "text": "use uh 8 KV heads. So uh so so we can"
  },
  {
    "startMs": 2990000,
    "text": "get a compression of 4x times and this"
  },
  {
    "startMs": 2993520,
    "text": "multi head latent attention this formula"
  },
  {
    "startMs": 2996800,
    "text": "depends on the model to model how many"
  },
  {
    "startMs": 2999440,
    "text": "layers your model have. So in the"
  },
  {
    "startMs": 3001359,
    "text": "original deepseek paper uh I think they"
  },
  {
    "startMs": 3003680,
    "text": "have some 128 dimension"
  },
  {
    "startMs": 3007680,
    "text": "128 d 12 I don't remember the exact"
  },
  {
    "startMs": 3010240,
    "text": "dimension but according to that uh they"
  },
  {
    "startMs": 3013040,
    "text": "have used uh this one latent vector in"
  },
  {
    "startMs": 3016400,
    "text": "which they have used 512 as a dimension"
  },
  {
    "startMs": 3019119,
    "text": "and some 64"
  },
  {
    "startMs": 3021359,
    "text": "for for rope index. So and then they"
  },
  {
    "startMs": 3025680,
    "text": "show that it is a 50x 56x"
  },
  {
    "startMs": 3029200,
    "text": "uh more compressed than multi head"
  },
  {
    "startMs": 3032000,
    "text": "attention."
  },
  {
    "startMs": 3037280,
    "text": "Okay."
  },
  {
    "startMs": 3039839,
    "text": "Yeah. So uh so yeah so this is uh so"
  },
  {
    "startMs": 3042079,
    "text": "this is the uh this is the trade-off uh"
  },
  {
    "startMs": 3045040,
    "text": "trade-off diagram. So here I think we"
  },
  {
    "startMs": 3046960,
    "text": "have not talked about this linear"
  },
  {
    "startMs": 3048960,
    "text": "attention or mamba. So main problem is"
  },
  {
    "startMs": 3052559,
    "text": "just all this m Matrix multiplication."
  },
  {
    "startMs": 3055119,
    "text": "Right now everyone is using attention."
  },
  {
    "startMs": 3057359,
    "text": "Suppose in future"
  },
  {
    "startMs": 3060800,
    "text": "uh if we don't want to use attention or"
  },
  {
    "startMs": 3063680,
    "text": "rather than generating tokens"
  },
  {
    "startMs": 3065359,
    "text": "sequentially just use maybe diffusion"
  },
  {
    "startMs": 3067599,
    "text": "models where we can generate everything"
  },
  {
    "startMs": 3069680,
    "text": "simultaneously. So all these algorithms"
  },
  {
    "startMs": 3072640,
    "text": "will change also. But here I think they"
  },
  {
    "startMs": 3075520,
    "text": "have two more. One is linear attention"
  },
  {
    "startMs": 3077520,
    "text": "and one is mamba. So according to uh"
  },
  {
    "startMs": 3080960,
    "text": "this slide so"
  },
  {
    "startMs": 3083200,
    "text": "if we are not compressing anything so"
  },
  {
    "startMs": 3085599,
    "text": "MHA is just we are parallelizing the"
  },
  {
    "startMs": 3088960,
    "text": "process so there is no quality loss so"
  },
  {
    "startMs": 3091760,
    "text": "it's a good and then this uh grouped"
  },
  {
    "startMs": 3094319,
    "text": "query attention which is I think almost"
  },
  {
    "startMs": 3096960,
    "text": "uh every model is using uh just GQA and"
  },
  {
    "startMs": 3100720,
    "text": "DSA kind of thing or yeah"
  },
  {
    "startMs": 3105040,
    "text": "I think same thing we are providing in"
  },
  {
    "startMs": 3106880,
    "text": "the attention mechanism scorecard So uh"
  },
  {
    "startMs": 3110480,
    "text": "so I think uh this one mha quality is"
  },
  {
    "startMs": 3113040,
    "text": "good throughput is uh throughput is okay"
  },
  {
    "startMs": 3116640,
    "text": "and for grouped query attention it"
  },
  {
    "startMs": 3119760,
    "text": "depends upon your use case also though"
  },
  {
    "startMs": 3122720,
    "text": "yeah though"
  },
  {
    "startMs": 3124800,
    "text": "quality is almost similar to uh multi"
  },
  {
    "startMs": 3127040,
    "text": "head attention but use case also matters"
  },
  {
    "startMs": 3129440,
    "text": "a lot yeah multi-query attention is just"
  },
  {
    "startMs": 3132880,
    "text": "one extreme we are"
  },
  {
    "startMs": 3135839,
    "text": "I don't know why but we are just"
  },
  {
    "startMs": 3137599,
    "text": "assuming that we only need one block and"
  },
  {
    "startMs": 3140480,
    "text": "all the queries will attend to that"
  },
  {
    "startMs": 3142720,
    "text": "smaller smaller block. So, so quality is"
  },
  {
    "startMs": 3146000,
    "text": "not that great for M for MQA and this"
  },
  {
    "startMs": 3150960,
    "text": "multi head latent attention. So yeah if"
  },
  {
    "startMs": 3154000,
    "text": "you have tried some this deep seat"
  },
  {
    "startMs": 3156480,
    "text": "models so I think uh they are doing"
  },
  {
    "startMs": 3158880,
    "text": "great job yeah in in quality wise"
  },
  {
    "startMs": 3162880,
    "text": "besides that sliding window so all these"
  },
  {
    "startMs": 3165599,
    "text": "are sub techniques which"
  },
  {
    "startMs": 3168319,
    "text": "yeah yeah all these are some techniques"
  },
  {
    "startMs": 3170640,
    "text": "like I just slide the windows all those"
  },
  {
    "startMs": 3173680,
    "text": "things and instead of yeah instead of"
  },
  {
    "startMs": 3176880,
    "text": "multiplying everything so linear"
  },
  {
    "startMs": 3179200,
    "text": "attention is just saying that sum"
  },
  {
    "startMs": 3180720,
    "text": "summarize everything first uh and then"
  },
  {
    "startMs": 3183440,
    "text": "look up into it and then mamba this is"
  },
  {
    "startMs": 3186319,
    "text": "just a state space model. Yeah,"
  },
  {
    "startMs": 3192319,
    "text": "I can cover that. Okay."
  },
  {
    "startMs": 3195520,
    "text": "Uh cool. Uh thank you T."
  },
  {
    "startMs": 3198480,
    "text": "So for the model like optimizations we"
  },
  {
    "startMs": 3202000,
    "text": "also have like the two notebooks"
  },
  {
    "startMs": 3205839,
    "text": "here."
  },
  {
    "startMs": 3208720,
    "text": "So there will be"
  },
  {
    "startMs": 3212800,
    "text": "I have to go to this."
  },
  {
    "startMs": 3222800,
    "text": "Okay. Uh so for the quantization uh like"
  },
  {
    "startMs": 3227119,
    "text": "the demo"
  },
  {
    "startMs": 3229040,
    "text": "uh this is is this already run? No. Let"
  },
  {
    "startMs": 3233359,
    "text": "me just run this."
  },
  {
    "startMs": 3243280,
    "text": "Okay. So we are loading the model which"
  },
  {
    "startMs": 3245920,
    "text": "is like uh ML 7B."
  },
  {
    "startMs": 3252480,
    "text": "Uh so"
  },
  {
    "startMs": 3257359,
    "text": "this one is like with the FP16 baseline."
  },
  {
    "startMs": 3269760,
    "text": "Wait."
  },
  {
    "startMs": 3271520,
    "text": "Uh, did it run?"
  },
  {
    "startMs": 3275839,
    "text": "Okay. So, it's uh two millisecond run."
  },
  {
    "startMs": 3280160,
    "text": "Did this run? Okay. So, yeah, this time"
  },
  {
    "startMs": 3283839,
    "text": "it's fetching that model with the FP16"
  },
  {
    "startMs": 3287359,
    "text": "precision."
  },
  {
    "startMs": 3297520,
    "text": "the Wi-Fi."
  },
  {
    "startMs": 3303040,
    "text": "It's going to take time."
  },
  {
    "startMs": 3307440,
    "text": "Okay."
  },
  {
    "startMs": 3311200,
    "text": ">> Yeah, it because it's downloading the"
  },
  {
    "startMs": 3313119,
    "text": "weights from the hugging face."
  },
  {
    "startMs": 3317040,
    "text": ">> Huh."
  },
  {
    "startMs": 3320480,
    "text": "Yeah. So, MOLAB is like running online."
  },
  {
    "startMs": 3323119,
    "text": ">> Yes."
  },
  {
    "startMs": 3326240,
    "text": "because it needs to make the network"
  },
  {
    "startMs": 3327839,
    "text": "call through to the hugging phase and"
  },
  {
    "startMs": 3329520,
    "text": "like it fetching"
  },
  {
    "startMs": 3332480,
    "text": "I don't know like but it's taking time"
  },
  {
    "startMs": 3334240,
    "text": "to download probably"
  },
  {
    "startMs": 3352880,
    "text": "Okay."
  },
  {
    "startMs": 3355920,
    "text": "So good. Okay. So here we see like the"
  },
  {
    "startMs": 3359359,
    "text": "memory size is like 15 GB around"
  },
  {
    "startMs": 3362400,
    "text": "approximately with the FP16 precision."
  },
  {
    "startMs": 3365440,
    "text": "We are trying to do the 2x compression"
  },
  {
    "startMs": 3368400,
    "text": "as Tmet talked about with the int8."
  },
  {
    "startMs": 3373839,
    "text": "Let's download. Okay. So we do see like"
  },
  {
    "startMs": 3376799,
    "text": "your memory size is now like 7.5 GB."
  },
  {
    "startMs": 3381200,
    "text": "What that means is now you have a more s"
  },
  {
    "startMs": 3383680,
    "text": "more memory for your KV to basically"
  },
  {
    "startMs": 3386799,
    "text": "grow. That means you can either serve"
  },
  {
    "startMs": 3388799,
    "text": "higher context limit or you can serve"
  },
  {
    "startMs": 3390720,
    "text": "the higher concurrent users there."
  },
  {
    "startMs": 3396319,
    "text": "If you do the like in your basic you are"
  },
  {
    "startMs": 3399040,
    "text": "doing the 4x compression so that with"
  },
  {
    "startMs": 3402640,
    "text": "the 4x compression it would be more"
  },
  {
    "startMs": 3405200,
    "text": "lower. It would be I think around"
  },
  {
    "startMs": 3408960,
    "text": "3 to 4 GB."
  },
  {
    "startMs": 3411200,
    "text": "Yeah. 4.5 GB"
  },
  {
    "startMs": 3415839,
    "text": "and Yep. So this is Wait."
  },
  {
    "startMs": 3422000,
    "text": "So this is just a basic plot"
  },
  {
    "startMs": 3425359,
    "text": "of like"
  },
  {
    "startMs": 3427599,
    "text": "so these are the like theoretical"
  },
  {
    "startMs": 3429119,
    "text": "numbers. uh we are not doing the like"
  },
  {
    "startMs": 3431200,
    "text": "any throughput test here but uh usually"
  },
  {
    "startMs": 3433680,
    "text": "you would see like your memory increases"
  },
  {
    "startMs": 3435839,
    "text": "so pro you would also have like a bit of"
  },
  {
    "startMs": 3438400,
    "text": "higher uh throughput."
  },
  {
    "startMs": 3441440,
    "text": "Uh from some of the benchmarks that we"
  },
  {
    "startMs": 3443599,
    "text": "studied we saw like the intate uh"
  },
  {
    "startMs": 3446000,
    "text": "compression it does have like a lower"
  },
  {
    "startMs": 3448000,
    "text": "throughput."
  },
  {
    "startMs": 3452960,
    "text": "Okay. And then there is like a demo on"
  },
  {
    "startMs": 3456960,
    "text": "the like the attention mechanisms."
  },
  {
    "startMs": 3462480,
    "text": "So for the attention okay I have to run"
  },
  {
    "startMs": 3467040,
    "text": "this."
  },
  {
    "startMs": 3478480,
    "text": "Uh okay so it has run. Oh, wait. Why"
  },
  {
    "startMs": 3484720,
    "text": "does it say no GPU detected?"
  },
  {
    "startMs": 3489440,
    "text": "It should say the GPU should be"
  },
  {
    "startMs": 3491520,
    "text": "detected."
  },
  {
    "startMs": 3498880,
    "text": "Oh, okay."
  },
  {
    "startMs": 3517920,
    "text": "Wait,"
  },
  {
    "startMs": 3530160,
    "text": "but this is surprising."
  },
  {
    "startMs": 3537200,
    "text": "Yeah, I guess it's not like able to"
  },
  {
    "startMs": 3539280,
    "text": "detect the GPU for some reason."
  },
  {
    "startMs": 3544319,
    "text": "Uh we do have like a GPU here."
  },
  {
    "startMs": 3551280,
    "text": "Uh okay, never mind."
  },
  {
    "startMs": 3554319,
    "text": "Yeah. Yeah. So, but the like basic idea"
  },
  {
    "startMs": 3556079,
    "text": "here was more like"
  },
  {
    "startMs": 3560000,
    "text": "as you try to move towards like"
  },
  {
    "startMs": 3562559,
    "text": "compressing the computation like by"
  },
  {
    "startMs": 3565280,
    "text": "using different attention mechanisms"
  },
  {
    "startMs": 3567920,
    "text": "like moving from the multi head to the"
  },
  {
    "startMs": 3570960,
    "text": "grouped query attention and then to the"
  },
  {
    "startMs": 3573119,
    "text": "MLA you would start seeing some"
  },
  {
    "startMs": 3575680,
    "text": "optimizations."
  },
  {
    "startMs": 3578000,
    "text": "Um I think yesterday night we were doing"
  },
  {
    "startMs": 3581200,
    "text": "some benchmarking. Uh I wanted to"
  },
  {
    "startMs": 3584400,
    "text": "correct this part. Uh so it wasn't like"
  },
  {
    "startMs": 3587119,
    "text": "50 56x it was 14x. Uh basically the demo"
  },
  {
    "startMs": 3592079,
    "text": "had a mistake of like a computation uh"
  },
  {
    "startMs": 3595920,
    "text": "where it did not multiply the number of"
  },
  {
    "startMs": 3598400,
    "text": "layers."
  },
  {
    "startMs": 3601040,
    "text": "Uh yeah so apologies for that. Uh so"
  },
  {
    "startMs": 3604400,
    "text": "this MLA is like a 14x savings work in"
  },
  {
    "startMs": 3608480,
    "text": "comparison to like your multi head"
  },
  {
    "startMs": 3610480,
    "text": "attention."
  },
  {
    "startMs": 3614319,
    "text": "Uh so now that we have understanding of"
  },
  {
    "startMs": 3618160,
    "text": "the pain points, the foundations, the"
  },
  {
    "startMs": 3621119,
    "text": "one side of the optimizations which is"
  },
  {
    "startMs": 3623119,
    "text": "the model optimizations,"
  },
  {
    "startMs": 3625760,
    "text": "we want to talk about what can you do on"
  },
  {
    "startMs": 3629440,
    "text": "the like the serving side."
  },
  {
    "startMs": 3633520,
    "text": "So"
  },
  {
    "startMs": 3636000,
    "text": "the first thing is we saw like when you"
  },
  {
    "startMs": 3638640,
    "text": "perform like a simple decode step you"
  },
  {
    "startMs": 3642079,
    "text": "are pulling it you are basically pulling"
  },
  {
    "startMs": 3644559,
    "text": "the model weights and then you are"
  },
  {
    "startMs": 3645920,
    "text": "recomputing the key and the value"
  },
  {
    "startMs": 3647680,
    "text": "vectors for all the previous tokens even"
  },
  {
    "startMs": 3651599,
    "text": "though you already computed the those"
  },
  {
    "startMs": 3653839,
    "text": "vectors for the tokens."
  },
  {
    "startMs": 3656960,
    "text": "So there is definitely like a lot of"
  },
  {
    "startMs": 3659200,
    "text": "compute wastage."
  },
  {
    "startMs": 3661520,
    "text": "Uh and if you kind of analyze the time"
  },
  {
    "startMs": 3664799,
    "text": "complexity of it, it would come out to"
  },
  {
    "startMs": 3666720,
    "text": "be O of N². Uh and the way to resolve"
  },
  {
    "startMs": 3670000,
    "text": "that is like a classic trade-off against"
  },
  {
    "startMs": 3671839,
    "text": "the memory. You can maintain a memory of"
  },
  {
    "startMs": 3675839,
    "text": "those vectors against the tokens and you"
  },
  {
    "startMs": 3678240,
    "text": "can reference that memory. So that"
  },
  {
    "startMs": 3680400,
    "text": "memory was called as like KV cache."
  },
  {
    "startMs": 3684559,
    "text": "uh and the like the flow looks something"
  },
  {
    "startMs": 3686640,
    "text": "like this"
  },
  {
    "startMs": 3690160,
    "text": "and then based on this KV cache there"
  },
  {
    "startMs": 3692480,
    "text": "were like four optimizations that were"
  },
  {
    "startMs": 3694480,
    "text": "really possible."
  },
  {
    "startMs": 3696559,
    "text": "Um"
  },
  {
    "startMs": 3698240,
    "text": "the first one is about the page"
  },
  {
    "startMs": 3701359,
    "text": "detention. So what's the different"
  },
  {
    "startMs": 3704000,
    "text": "what's the problem today? So when you"
  },
  {
    "startMs": 3706160,
    "text": "send like multiple requests as the input"
  },
  {
    "startMs": 3708319,
    "text": "to the GPU"
  },
  {
    "startMs": 3710559,
    "text": "these requests are in a batch"
  },
  {
    "startMs": 3714240,
    "text": "uh every request is allocated like a"
  },
  {
    "startMs": 3717359,
    "text": "continuous memory storage let's say of"
  },
  {
    "startMs": 3721359,
    "text": "I'm just taking an example like let's"
  },
  {
    "startMs": 3723280,
    "text": "set uh 2 KB"
  },
  {
    "startMs": 3726720,
    "text": "however like your request needed only"
  },
  {
    "startMs": 3728799,
    "text": "let's say"
  },
  {
    "startMs": 3730799,
    "text": "uh 1 KB so there is like u 50% of that"
  },
  {
    "startMs": 3736880,
    "text": "memory fragmentation."
  },
  {
    "startMs": 3739839,
    "text": "Uh and this fragmentation basically"
  },
  {
    "startMs": 3742559,
    "text": "leads to the memory wastage. That means"
  },
  {
    "startMs": 3746240,
    "text": "there was a space in the memory where"
  },
  {
    "startMs": 3748079,
    "text": "you could have served more requests but"
  },
  {
    "startMs": 3751119,
    "text": "you could not because you were looking"
  },
  {
    "startMs": 3753520,
    "text": "for that contigious block of the memory."
  },
  {
    "startMs": 3756319,
    "text": "So an inspiration to was being taken"
  },
  {
    "startMs": 3759040,
    "text": "from like how the OS works like you"
  },
  {
    "startMs": 3762160,
    "text": "maintain a logical memory and you"
  },
  {
    "startMs": 3764720,
    "text": "basically have a physical memory."
  },
  {
    "startMs": 3767760,
    "text": "So in the logical memory it would still"
  },
  {
    "startMs": 3771680,
    "text": "feel like"
  },
  {
    "startMs": 3773599,
    "text": "uh that the KV vector for the like every"
  },
  {
    "startMs": 3777119,
    "text": "token is like a contiguous"
  },
  {
    "startMs": 3780079,
    "text": "but it will be mapping to a different"
  },
  {
    "startMs": 3782240,
    "text": "physical address."
  },
  {
    "startMs": 3786960,
    "text": "So that really helped like saving a lot"
  },
  {
    "startMs": 3790640,
    "text": "of memory. Uh and it was only possible"
  },
  {
    "startMs": 3794079,
    "text": "because you they considered like memory"
  },
  {
    "startMs": 3796240,
    "text": "as a set of blocks and you would be"
  },
  {
    "startMs": 3799680,
    "text": "dynamically allocating those blocks as"
  },
  {
    "startMs": 3801680,
    "text": "the request need as the like new tokens"
  },
  {
    "startMs": 3805039,
    "text": "comes in and they need that kind of"
  },
  {
    "startMs": 3807920,
    "text": "memory."
  },
  {
    "startMs": 3810960,
    "text": "The another lever is like when you are"
  },
  {
    "startMs": 3814480,
    "text": "sending multiple requests"
  },
  {
    "startMs": 3817760,
    "text": "in the batch"
  },
  {
    "startMs": 3819839,
    "text": "GPU is like taking those requests"
  },
  {
    "startMs": 3824240,
    "text": "but it does not accepts the new batch"
  },
  {
    "startMs": 3826640,
    "text": "unless all the requests in that batch"
  },
  {
    "startMs": 3828640,
    "text": "gets completed. So the diagram looks"
  },
  {
    "startMs": 3831119,
    "text": "more like a page retention but here it"
  },
  {
    "startMs": 3833680,
    "text": "is more about like when is GPU available"
  },
  {
    "startMs": 3837520,
    "text": "to take the next batch. So there is a"
  },
  {
    "startMs": 3841280,
    "text": "time period where GPU is like sitting"
  },
  {
    "startMs": 3843359,
    "text": "really idle"
  },
  {
    "startMs": 3845359,
    "text": "and you want to like resolve for that"
  },
  {
    "startMs": 3849039,
    "text": "and for that like the idea was like okay"
  },
  {
    "startMs": 3851920,
    "text": "let's do that continuous batching."
  },
  {
    "startMs": 3857520,
    "text": "So the continuous batching also really"
  },
  {
    "startMs": 3859760,
    "text": "helped with like throughput because now"
  },
  {
    "startMs": 3861599,
    "text": "you can ship more requests pretty"
  },
  {
    "startMs": 3864079,
    "text": "quickly. Keep making sure like GPU"
  },
  {
    "startMs": 3866559,
    "text": "always uh get is always like occupied"
  },
  {
    "startMs": 3870000,
    "text": "and it's not like uh sitting idle. So"
  },
  {
    "startMs": 3873839,
    "text": "you are saving on that compute."
  },
  {
    "startMs": 3876640,
    "text": "The third is the like prefix caching. So"
  },
  {
    "startMs": 3879760,
    "text": "you remember like the KV cache helped"
  },
  {
    "startMs": 3881760,
    "text": "you save the computation for a single"
  },
  {
    "startMs": 3884319,
    "text": "request across the tokens."
  },
  {
    "startMs": 3887599,
    "text": "But what if like you have the same"
  },
  {
    "startMs": 3890880,
    "text": "tokens across multiple requests? How do"
  },
  {
    "startMs": 3893839,
    "text": "you basically save against that? So the"
  },
  {
    "startMs": 3896960,
    "text": "prefix caching uh which was introduced"
  },
  {
    "startMs": 3899520,
    "text": "by VLM"
  },
  {
    "startMs": 3903039,
    "text": "exactly counters that"
  },
  {
    "startMs": 3906720,
    "text": "and then the third is like we talked"
  },
  {
    "startMs": 3909119,
    "text": "about the fourth actually. So we talked"
  },
  {
    "startMs": 3912160,
    "text": "about quantizing the model"
  },
  {
    "startMs": 3915520,
    "text": "but you could also you can also like"
  },
  {
    "startMs": 3918079,
    "text": "quantize the KV weights."
  },
  {
    "startMs": 3921520,
    "text": "So that means now you you need like a"
  },
  {
    "startMs": 3924880,
    "text": "lesser space for your key and the value"
  },
  {
    "startMs": 3927039,
    "text": "vectors. That means you can serve more"
  },
  {
    "startMs": 3929680,
    "text": "key and the value vectors in the memory."
  },
  {
    "startMs": 3932480,
    "text": "And that means like you can serve more"
  },
  {
    "startMs": 3934240,
    "text": "tokens. That means you can serve more"
  },
  {
    "startMs": 3936160,
    "text": "context context limit. And that means"
  },
  {
    "startMs": 3938960,
    "text": "like you can serve more model quality"
  },
  {
    "startMs": 3945200,
    "text": "and all of this is like uh already"
  },
  {
    "startMs": 3949680,
    "text": "present in the VLM."
  },
  {
    "startMs": 3951920,
    "text": "You don't really need to reinvent that"
  },
  {
    "startMs": 3954240,
    "text": "wheel"
  },
  {
    "startMs": 3956640,
    "text": "uh and you can like deploy this VLM in"
  },
  {
    "startMs": 3959520,
    "text": "production and you could see that"
  },
  {
    "startMs": 3962400,
    "text": "basically growth."
  },
  {
    "startMs": 3965599,
    "text": "So next we have like a benchmark that we"
  },
  {
    "startMs": 3968559,
    "text": "did. So this benchmark was"
  },
  {
    "startMs": 3972640,
    "text": "let me see if I have that"
  },
  {
    "startMs": 3977119,
    "text": "here"
  },
  {
    "startMs": 3979760,
    "text": "the demos."
  },
  {
    "startMs": 3983359,
    "text": "So doing this benchmark takes like"
  },
  {
    "startMs": 3986000,
    "text": "around 1 hour because you have to"
  },
  {
    "startMs": 3988319,
    "text": "continuously stop and like restart the"
  },
  {
    "startMs": 3990880,
    "text": "VLM servers and you have to load the"
  },
  {
    "startMs": 3993920,
    "text": "models and all. So it does take a lot of"
  },
  {
    "startMs": 3996799,
    "text": "time in doing the testing but I can like"
  },
  {
    "startMs": 3999200,
    "text": "really tell you here what we are doing."
  },
  {
    "startMs": 4002720,
    "text": "So we have kept the model as same like"
  },
  {
    "startMs": 4005680,
    "text": "the Mistful 7B."
  },
  {
    "startMs": 4008240,
    "text": "Uh and then we have like the set of"
  },
  {
    "startMs": 4011280,
    "text": "input questions that we are sending. Uh"
  },
  {
    "startMs": 4015200,
    "text": "consider them as the prompts. Then we"
  },
  {
    "startMs": 4017680,
    "text": "have couple of helper functions here"
  },
  {
    "startMs": 4019359,
    "text": "like checking the server is up or not."
  },
  {
    "startMs": 4022319,
    "text": "The server is the VLM server."
  },
  {
    "startMs": 4025760,
    "text": "Then there are helper functions to get"
  },
  {
    "startMs": 4028240,
    "text": "the VLM metrics."
  },
  {
    "startMs": 4030880,
    "text": "uh and I will talk about like what those"
  },
  {
    "startMs": 4032720,
    "text": "metrics are. Uh then there are like lot"
  },
  {
    "startMs": 4035599,
    "text": "of the benchmarks and all and then you"
  },
  {
    "startMs": 4038720,
    "text": "have to measure uh the KV usage and all."
  },
  {
    "startMs": 4043280,
    "text": "So these are the like helper functions."
  },
  {
    "startMs": 4045200,
    "text": "So the baseline is very simple like we"
  },
  {
    "startMs": 4047280,
    "text": "have a hugging phase baseline."
  },
  {
    "startMs": 4050160,
    "text": "Uh this is the raw like sending the text"
  },
  {
    "startMs": 4053680,
    "text": "to the LLM getting back the response. We"
  },
  {
    "startMs": 4057119,
    "text": "see some results here. We saw like"
  },
  {
    "startMs": 4060240,
    "text": "hugging phase has a throughput of like"
  },
  {
    "startMs": 4062000,
    "text": "around 51 tokens per second. Time to"
  },
  {
    "startMs": 4064720,
    "text": "first token was like 54 and then the"
  },
  {
    "startMs": 4066880,
    "text": "inter token latency was 19."
  },
  {
    "startMs": 4070000,
    "text": "Uh this bas uh this was all run on the"
  },
  {
    "startMs": 4072480,
    "text": "h100."
  },
  {
    "startMs": 4075200,
    "text": "Uh and then we start like a very default"
  },
  {
    "startMs": 4079440,
    "text": "VLM server. So by default VLM provides"
  },
  {
    "startMs": 4082720,
    "text": "you the page detention, continuous"
  },
  {
    "startMs": 4084400,
    "text": "batching"
  },
  {
    "startMs": 4086000,
    "text": "and the KV caching."
  },
  {
    "startMs": 4088240,
    "text": "So three things are present by default"
  },
  {
    "startMs": 4092960,
    "text": "and when you try to compare those"
  },
  {
    "startMs": 4095200,
    "text": "benchmarks you see your throughput is"
  },
  {
    "startMs": 4098640,
    "text": "like almost 15x you are able to serve"
  },
  {
    "startMs": 4102159,
    "text": "more tokens per second"
  },
  {
    "startMs": 4105040,
    "text": "then"
  },
  {
    "startMs": 4106640,
    "text": "your time to the first token"
  },
  {
    "startMs": 4110400,
    "text": "uh that also rises"
  },
  {
    "startMs": 4114239,
    "text": "and then your v the inter token latency"
  },
  {
    "startMs": 4117279,
    "text": "kind goes down and then your KV versus"
  },
  {
    "startMs": 4121120,
    "text": "users and the versus context rate"
  },
  {
    "startMs": 4122960,
    "text": "increases for sure."
  },
  {
    "startMs": 4127920,
    "text": "Now when you apply the prefix caching to"
  },
  {
    "startMs": 4130319,
    "text": "it"
  },
  {
    "startMs": 4132319,
    "text": "so with the prefix caching you see like"
  },
  {
    "startMs": 4134480,
    "text": "your throughput increases"
  },
  {
    "startMs": 4137120,
    "text": "more your TDF decreases your inter token"
  },
  {
    "startMs": 4141199,
    "text": "latency is approximately same uh and"
  },
  {
    "startMs": 4144159,
    "text": "then your KV cache usage versus the"
  },
  {
    "startMs": 4147359,
    "text": "users it's kind of going down"
  },
  {
    "startMs": 4150640,
    "text": "the vers context it's not going down"
  },
  {
    "startMs": 4152799,
    "text": "it's approximately same I think this is"
  },
  {
    "startMs": 4155839,
    "text": "also approximately same it's like not"
  },
  {
    "startMs": 4158159,
    "text": "that uh big of a deal"
  },
  {
    "startMs": 4161839,
    "text": "when you apply the like KV quantization"
  },
  {
    "startMs": 4164480,
    "text": "on top of it."
  },
  {
    "startMs": 4167600,
    "text": "So it becomes like so so you see like"
  },
  {
    "startMs": 4171040,
    "text": "your throughput is like almost similar."
  },
  {
    "startMs": 4174159,
    "text": "Your time to first token is similar."
  },
  {
    "startMs": 4176640,
    "text": "Your token latency is similar but then"
  },
  {
    "startMs": 4180159,
    "text": "your KV usage actually goes down. And"
  },
  {
    "startMs": 4182319,
    "text": "this is because like you have quantized"
  },
  {
    "startMs": 4185199,
    "text": "your key value space."
  },
  {
    "startMs": 4188560,
    "text": "Uh and then there is a concept of"
  },
  {
    "startMs": 4191359,
    "text": "speculative decoding that TME will talk"
  },
  {
    "startMs": 4194080,
    "text": "about. Uh"
  },
  {
    "startMs": 4196640,
    "text": "so when you try to benchmark those so"
  },
  {
    "startMs": 4200159,
    "text": "you also see like there is a uh like a"
  },
  {
    "startMs": 4203199,
    "text": "bit of like the less KV usage there"
  },
  {
    "startMs": 4206480,
    "text": "although like the results are"
  },
  {
    "startMs": 4207760,
    "text": "approximately same."
  },
  {
    "startMs": 4215040,
    "text": "So yeah, I mean overall like these are"
  },
  {
    "startMs": 4219520,
    "text": "the like the metrics across probably I"
  },
  {
    "startMs": 4222880,
    "text": "should"
  },
  {
    "startMs": 4225040,
    "text": "zoom out. Okay, it's not zoom out. It's"
  },
  {
    "startMs": 4227920,
    "text": "not working."
  },
  {
    "startMs": 4230239,
    "text": "Great. So yeah, this is the like VLM"
  },
  {
    "startMs": 4234159,
    "text": "benchmarks. Um it's your production"
  },
  {
    "startMs": 4236800,
    "text": "default by the way. uh we will also"
  },
  {
    "startMs": 4239440,
    "text": "share that decision tree uh when we try"
  },
  {
    "startMs": 4243360,
    "text": "to talk about like the other engines."
  },
  {
    "startMs": 4249040,
    "text": "So"
  },
  {
    "startMs": 4251199,
    "text": "yeah, so we should talk about like what"
  },
  {
    "startMs": 4254400,
    "text": "are some of the other inference"
  },
  {
    "startMs": 4256000,
    "text": "optimizations we can do on top of it and"
  },
  {
    "startMs": 4259679,
    "text": "what were some of the other solutions"
  },
  {
    "startMs": 4261440,
    "text": "that came out."
  },
  {
    "startMs": 4265199,
    "text": "Uh so I would like to again invite"
  },
  {
    "startMs": 4267760,
    "text": "Tanme. He's going to talk about like"
  },
  {
    "startMs": 4270000,
    "text": "some of these optimizations."
  },
  {
    "startMs": 4278480,
    "text": "Oh, sorry. Uh I'm so sorry. Uh I didn't"
  },
  {
    "startMs": 4281199,
    "text": "enable the slides."
  },
  {
    "startMs": 4285280,
    "text": "Uh what was the Okay, great."
  },
  {
    "startMs": 4290000,
    "text": "Perfect."
  },
  {
    "startMs": 4290800,
    "text": ">> Which one?"
  },
  {
    "startMs": 4291600,
    "text": ">> The speculative."
  },
  {
    "startMs": 4292800,
    "text": ">> Yeah. Thank you, Hersel. Yeah."
  },
  {
    "startMs": 4296080,
    "text": "So,"
  },
  {
    "startMs": 4297600,
    "text": "so all these are like speculative"
  },
  {
    "startMs": 4299920,
    "text": "decoding all these are the uh so so what"
  },
  {
    "startMs": 4303120,
    "text": "we say uh different flavors of same kind"
  },
  {
    "startMs": 4306239,
    "text": "of soda. So this uh this technique comes"
  },
  {
    "startMs": 4310320,
    "text": "under decoding accelerator. So first one"
  },
  {
    "startMs": 4313199,
    "text": "so we are only talking about this"
  },
  {
    "startMs": 4315840,
    "text": "speculative decoding but there are other"
  },
  {
    "startMs": 4318000,
    "text": "variants like self speculative eagle"
  },
  {
    "startMs": 4321120,
    "text": "medusa"
  },
  {
    "startMs": 4322640,
    "text": "I only like I think uh this one eagle"
  },
  {
    "startMs": 4325360,
    "text": "algorithm"
  },
  {
    "startMs": 4327440,
    "text": "personally I don't think"
  },
  {
    "startMs": 4330640,
    "text": "speculative decoding works because main"
  },
  {
    "startMs": 4332880,
    "text": "problem is alignment okay so let's start"
  },
  {
    "startMs": 4335520,
    "text": "with what is uh speculative decoding"
  },
  {
    "startMs": 4338480,
    "text": "main problem is that in transformer"
  },
  {
    "startMs": 4340960,
    "text": "architecture All these tokens are"
  },
  {
    "startMs": 4342960,
    "text": "generated sequentially one by one by"
  },
  {
    "startMs": 4346080,
    "text": "one. How about just use a smaller model"
  },
  {
    "startMs": 4350880,
    "text": "and let a smaller model to generate"
  },
  {
    "startMs": 4354960,
    "text": "maybe let's say four or five tokens and"
  },
  {
    "startMs": 4358159,
    "text": "this teacher model or we can say"
  },
  {
    "startMs": 4360480,
    "text": "according to our world cup algorithm we"
  },
  {
    "startMs": 4362480,
    "text": "can say referee. So referee will decide"
  },
  {
    "startMs": 4365040,
    "text": "how many uh tokens it accept and this"
  },
  {
    "startMs": 4369760,
    "text": "loop keeps on going on and our"
  },
  {
    "startMs": 4373440,
    "text": "assumption is that there are certain"
  },
  {
    "startMs": 4376159,
    "text": "domain where this kind of things will"
  },
  {
    "startMs": 4379199,
    "text": "work like maybe in decode maybe in"
  },
  {
    "startMs": 4381600,
    "text": "coding or where almost there is no"
  },
  {
    "startMs": 4384560,
    "text": "creativity uh each uh code or syntax is"
  },
  {
    "startMs": 4388000,
    "text": "almost similar. So maybe it can help it."
  },
  {
    "startMs": 4391440,
    "text": "But uh based on personal testing, I"
  },
  {
    "startMs": 4394640,
    "text": "didn't find this speculative decoding"
  },
  {
    "startMs": 4397280,
    "text": "useful at all. But other techniques like"
  },
  {
    "startMs": 4401760,
    "text": "uh self speculative decoding where"
  },
  {
    "startMs": 4404719,
    "text": "teacher model also have one head"
  },
  {
    "startMs": 4407280,
    "text": "auxiliary head and it will do same"
  },
  {
    "startMs": 4409760,
    "text": "similar kind of things what this base"
  },
  {
    "startMs": 4412400,
    "text": "model or small model is doing it. But"
  },
  {
    "startMs": 4416640,
    "text": "then this eagle came Eagle 1 2 3 I don't"
  },
  {
    "startMs": 4419760,
    "text": "know how many version versions are but"
  },
  {
    "startMs": 4422400,
    "text": "it is just saying that instead of"
  },
  {
    "startMs": 4425440,
    "text": "creating instead of generating tokens uh"
  },
  {
    "startMs": 4427760,
    "text": "let's uh train a small model inside"
  },
  {
    "startMs": 4430640,
    "text": "train a small model and just take a"
  },
  {
    "startMs": 4433520,
    "text": "features from one of its uh one of main"
  },
  {
    "startMs": 4437120,
    "text": "models layer so that instead of"
  },
  {
    "startMs": 4439280,
    "text": "generating token uh it will generate uh"
  },
  {
    "startMs": 4441679,
    "text": "this features so so uh so eagle is uh"
  },
  {
    "startMs": 4445840,
    "text": "Eagle is better compared to this other"
  },
  {
    "startMs": 4449280,
    "text": "kind of technologies and then another"
  },
  {
    "startMs": 4451920,
    "text": "one is Medusa which is just saying that"
  },
  {
    "startMs": 4454880,
    "text": "just generate all the tokens parallelly."
  },
  {
    "startMs": 4458159,
    "text": "Uh okay. So here so here in this slide"
  },
  {
    "startMs": 4463520,
    "text": ">> yeah the next slide."
  },
  {
    "startMs": 4465199,
    "text": ">> Okay."
  },
  {
    "startMs": 4469440,
    "text": "Okay. Yeah. Okay. Now we come to uh now"
  },
  {
    "startMs": 4473040,
    "text": "we will come to this one prefix caching."
  },
  {
    "startMs": 4475280,
    "text": "So I don't know whether people are using"
  },
  {
    "startMs": 4477120,
    "text": "this one static prefix caching or not"
  },
  {
    "startMs": 4479760,
    "text": "but thing is that main problem with"
  },
  {
    "startMs": 4482880,
    "text": "prefix caching is that sometimes we type"
  },
  {
    "startMs": 4486000,
    "text": "and make a small kind of mistake and"
  },
  {
    "startMs": 4488800,
    "text": "this standard static prefix caching is"
  },
  {
    "startMs": 4491199,
    "text": "basically it takes a prompt do some"
  },
  {
    "startMs": 4493120,
    "text": "hashing and then next time when user"
  },
  {
    "startMs": 4495600,
    "text": "asks similar kind of question it will"
  },
  {
    "startMs": 4497679,
    "text": "try to match the hash. So if hash is uh"
  },
  {
    "startMs": 4501679,
    "text": "if hash is equal then then it will"
  },
  {
    "startMs": 4504400,
    "text": "instead of recomputing all those K and B"
  },
  {
    "startMs": 4506960,
    "text": "it will just uh take it from from the"
  },
  {
    "startMs": 4509520,
    "text": "storage but you know that sometimes we"
  },
  {
    "startMs": 4511760,
    "text": "make a mistake or maybe we can just"
  },
  {
    "startMs": 4513520,
    "text": "change a word or letter something like"
  },
  {
    "startMs": 4515760,
    "text": "that then we have a very higher uh cache"
  },
  {
    "startMs": 4519040,
    "text": "uh hit cache miss hit rate so that's why"
  },
  {
    "startMs": 4523920,
    "text": "uh this one uh radics tree so radics"
  },
  {
    "startMs": 4526239,
    "text": "tree is becoming very popular and also"
  },
  {
    "startMs": 4529040,
    "text": "also because of agent. So I think almost"
  },
  {
    "startMs": 4532320,
    "text": "everyone is doing agent and most of the"
  },
  {
    "startMs": 4534400,
    "text": "computation is going during TT during"
  },
  {
    "startMs": 4537360,
    "text": "test time inference kind of thing where"
  },
  {
    "startMs": 4539840,
    "text": "we keep on asking same kind of questions"
  },
  {
    "startMs": 4542080,
    "text": "and prompt for example you are an expert"
  },
  {
    "startMs": 4545600,
    "text": "software engineer multiply by 200 times."
  },
  {
    "startMs": 4549120,
    "text": "This kind of loop keeps on going inside"
  },
  {
    "startMs": 4552239,
    "text": "this uh agentic agentic kind of things"
  },
  {
    "startMs": 4555920,
    "text": "where it is al necessary to keep uh or"
  },
  {
    "startMs": 4559120,
    "text": "store similar kind of things in a radics"
  },
  {
    "startMs": 4563280,
    "text": "tree. So radics tree is just so so radic"
  },
  {
    "startMs": 4567600,
    "text": "tree is just advanced version of this"
  },
  {
    "startMs": 4569840,
    "text": "prefix tree where where we will just"
  },
  {
    "startMs": 4572960,
    "text": "where we will just collapse a node if it"
  },
  {
    "startMs": 4574960,
    "text": "does not have a does not have any branch"
  },
  {
    "startMs": 4578159,
    "text": "and for this kind of work where keep on"
  },
  {
    "startMs": 4583600,
    "text": "repeating same thing this uh red x tree"
  },
  {
    "startMs": 4588239,
    "text": "helps a lot and st lang uh use this kind"
  },
  {
    "startMs": 4592400,
    "text": "of algorithm"
  },
  {
    "startMs": 4593920,
    "text": "for prefix caching."
  },
  {
    "startMs": 4597600,
    "text": "Okay. Yeah. Then there is another thing."
  },
  {
    "startMs": 4599520,
    "text": "One is tensor RT LLM. This is very"
  },
  {
    "startMs": 4602480,
    "text": "confusing. When I first started, I was I"
  },
  {
    "startMs": 4606080,
    "text": "was just confused. What is tensor RTLM?"
  },
  {
    "startMs": 4609600,
    "text": "So yeah. So tensor RT is just a uh it's"
  },
  {
    "startMs": 4613280,
    "text": "just a standard uh SDK kind of thing."
  },
  {
    "startMs": 4616000,
    "text": "Tensor RTLM is just an inference engine"
  },
  {
    "startMs": 4619679,
    "text": "just like VLM, SG lang. But problem is"
  },
  {
    "startMs": 4623360,
    "text": "that it is related to Nvidia. They"
  },
  {
    "startMs": 4627120,
    "text": "optimized each and every layer and every"
  },
  {
    "startMs": 4630480,
    "text": "problem as I mentioned in our world cup"
  },
  {
    "startMs": 4633360,
    "text": "algorithm. They just break everything"
  },
  {
    "startMs": 4635840,
    "text": "and optimized everything at hardware"
  },
  {
    "startMs": 4638159,
    "text": "level also. So uh yeah. So okay next."
  },
  {
    "startMs": 4647040,
    "text": "Yeah. So for this workshop we also uh"
  },
  {
    "startMs": 4650320,
    "text": "did some benchmarking like which is best"
  },
  {
    "startMs": 4654080,
    "text": "uh so our setup was something similar"
  },
  {
    "startMs": 4657120,
    "text": "was so so we did two kind of testing."
  },
  {
    "startMs": 4659920,
    "text": "First one is without uh without agentic"
  },
  {
    "startMs": 4663040,
    "text": "testing where we just so"
  },
  {
    "startMs": 4666080,
    "text": "we use this shared GPT uh this one data"
  },
  {
    "startMs": 4668719,
    "text": "set and uh just ask those questions uh"
  },
  {
    "startMs": 4673280,
    "text": "using VLM and SG lang."
  },
  {
    "startMs": 4677920,
    "text": "Okay."
  },
  {
    "startMs": 4686000,
    "text": "Yeah. Okay."
  },
  {
    "startMs": 4689120,
    "text": "And let me just zoom it up. Okay, great."
  },
  {
    "startMs": 4694800,
    "text": "Okay. Yeah. So, yeah, for this workshop,"
  },
  {
    "startMs": 4696960,
    "text": "we used H100 and of our first testing"
  },
  {
    "startMs": 4700400,
    "text": "was that uh we just uh we just asked uh"
  },
  {
    "startMs": 4704000,
    "text": "we take questions from shared GPT and"
  },
  {
    "startMs": 4706320,
    "text": "put it into VLM, SG lang and we found"
  },
  {
    "startMs": 4710800,
    "text": "that actually there's no statistical"
  },
  {
    "startMs": 4712880,
    "text": "difference between which one is better."
  },
  {
    "startMs": 4715679,
    "text": "So both have almost similar kind. So"
  },
  {
    "startMs": 4718320,
    "text": "both are fulfilling similar kind of"
  },
  {
    "startMs": 4720159,
    "text": "request per second uh TTFT and latency."
  },
  {
    "startMs": 4724080,
    "text": "So but only difference we have seen"
  },
  {
    "startMs": 4727840,
    "text": "during agentic uh agentic branching. So"
  },
  {
    "startMs": 4732640,
    "text": "uh what we did was that we asked that"
  },
  {
    "startMs": 4734640,
    "text": "similar kind of question that you are"
  },
  {
    "startMs": 4736320,
    "text": "the best this one software engineer in"
  },
  {
    "startMs": 4738640,
    "text": "the world just solve the problem of"
  },
  {
    "startMs": 4742000,
    "text": "traffic congestion in this city kind of"
  },
  {
    "startMs": 4744080,
    "text": "thing. Then we put this into LLM. LLM"
  },
  {
    "startMs": 4749120,
    "text": "generates some output. Then we did"
  },
  {
    "startMs": 4751600,
    "text": "another uh round two also. So once this"
  },
  {
    "startMs": 4755040,
    "text": "LLM generates this output, then in round"
  },
  {
    "startMs": 4759280,
    "text": "two we have specially mentioned that uh"
  },
  {
    "startMs": 4763360,
    "text": "provide uh review the proposal and give"
  },
  {
    "startMs": 4766719,
    "text": "ratings from 1 to 10. So this uh two"
  },
  {
    "startMs": 4770400,
    "text": "turns we did uh and this loop keeps on"
  },
  {
    "startMs": 4774080,
    "text": "uh repeating it. Uh what we found is"
  },
  {
    "startMs": 4776480,
    "text": "that for this kind of"
  },
  {
    "startMs": 4780320,
    "text": "uh workflow where everything is standard"
  },
  {
    "startMs": 4783520,
    "text": "all those prompts and context"
  },
  {
    "startMs": 4785199,
    "text": "engineering comes into the picture. If"
  },
  {
    "startMs": 4787760,
    "text": "we do proper this agentic branching then"
  },
  {
    "startMs": 4790560,
    "text": "I think uh this HG lang is three to four"
  },
  {
    "startMs": 4793840,
    "text": "times better. But again this depends"
  },
  {
    "startMs": 4796719,
    "text": "upon the different setup maybe uh if you"
  },
  {
    "startMs": 4799600,
    "text": "do it uh you may get different results."
  },
  {
    "startMs": 4803280,
    "text": "Okay. Yeah. So I think uh did we"
  },
  {
    "startMs": 4807280,
    "text": "uploaded it on GitHub? Okay."
  },
  {
    "startMs": 4810400,
    "text": ">> Yeah. So the PDF is like also in the"
  },
  {
    "startMs": 4814880,
    "text": "drive. Uh it's the same link as the"
  },
  {
    "startMs": 4817440,
    "text": "slides."
  },
  {
    "startMs": 4819280,
    "text": "So a quick summary here."
  },
  {
    "startMs": 4823280,
    "text": "So on a standard API workload throughput"
  },
  {
    "startMs": 4826719,
    "text": "you would see like a VLM and the SG lang"
  },
  {
    "startMs": 4830000,
    "text": "would behave same. So if you don't have"
  },
  {
    "startMs": 4833280,
    "text": "if you have like a standard workload"
  },
  {
    "startMs": 4835360,
    "text": "definitely go with VLM. It's the"
  },
  {
    "startMs": 4837280,
    "text": "production default anyways. But what"
  },
  {
    "startMs": 4839920,
    "text": "Tanme was also saying is when you try to"
  },
  {
    "startMs": 4843440,
    "text": "like make it like agentic workloads that"
  },
  {
    "startMs": 4846560,
    "text": "is where like your SG link really shines"
  },
  {
    "startMs": 4851120,
    "text": "uh and uh it kind of like provides you"
  },
  {
    "startMs": 4853760,
    "text": "all the benefits."
  },
  {
    "startMs": 4857679,
    "text": "So yeah keep like VLM as a default but"
  },
  {
    "startMs": 4860800,
    "text": "if you have agentic workloads probably"
  },
  {
    "startMs": 4863199,
    "text": "try to move as the towards the SG lang."
  },
  {
    "startMs": 4865920,
    "text": "if you're not happy with DB LLM. Uh but"
  },
  {
    "startMs": 4872480,
    "text": "uh okay. Uh let me"
  },
  {
    "startMs": 4881040,
    "text": "Okay."
  },
  {
    "startMs": 4886000,
    "text": "And then like there is like the like a"
  },
  {
    "startMs": 4889840,
    "text": "comparison that is done at the 120"
  },
  {
    "startMs": 4892480,
    "text": "billion like for the GPTO OSS 120"
  },
  {
    "startMs": 4895440,
    "text": "billion. Um this is a benchmark that was"
  },
  {
    "startMs": 4899440,
    "text": "prepared by clarify. So there is like a"
  },
  {
    "startMs": 4904480,
    "text": "blog link here. Oh nice."
  },
  {
    "startMs": 4908960,
    "text": "Okay. Yeah. So they did the similar"
  },
  {
    "startMs": 4912159,
    "text": "benchmark and they included like a"
  },
  {
    "startMs": 4914080,
    "text": "tensor RT LLM in it."
  },
  {
    "startMs": 4918400,
    "text": "Definitely you can always go through"
  },
  {
    "startMs": 4920480,
    "text": "these benchmarks and try to understand"
  },
  {
    "startMs": 4922719,
    "text": "which basically suits your use case. As"
  },
  {
    "startMs": 4926239,
    "text": "we mentioned like Tensor RT they try to"
  },
  {
    "startMs": 4928880,
    "text": "optimize the hardware side as well"
  },
  {
    "startMs": 4930800,
    "text": "having the peak hardware performance."
  },
  {
    "startMs": 4937520,
    "text": "Wait uh this is"
  },
  {
    "startMs": 4942000,
    "text": "okay."
  },
  {
    "startMs": 4946159,
    "text": "Yeah."
  },
  {
    "startMs": 4947920,
    "text": "And then like in terms of when you want"
  },
  {
    "startMs": 4950560,
    "text": "to dep pick like your engines"
  },
  {
    "startMs": 4954400,
    "text": "once you figure out like between VLM, SG"
  },
  {
    "startMs": 4957280,
    "text": "lang tenserati so that there are some"
  },
  {
    "startMs": 4959840,
    "text": "new engines that are popping up Nvidia"
  },
  {
    "startMs": 4962880,
    "text": "Dynamo for sure. Uh so they are also for"
  },
  {
    "startMs": 4966080,
    "text": "the agentic uh session routing."
  },
  {
    "startMs": 4969760,
    "text": "Uh hugging phase is always there. It's a"
  },
  {
    "startMs": 4972239,
    "text": "simple no server. Then there is like an"
  },
  {
    "startMs": 4975440,
    "text": "MSAR"
  },
  {
    "startMs": 4977440,
    "text": "engine that was recently proposed by"
  },
  {
    "startMs": 4979760,
    "text": "Stanford. They are for like the"
  },
  {
    "startMs": 4982880,
    "text": "multimodel."
  },
  {
    "startMs": 4985280,
    "text": "Uh so definitely you could explore those"
  },
  {
    "startMs": 4988960,
    "text": "and when you try to basically just to"
  },
  {
    "startMs": 4991199,
    "text": "like give a quick summary uh you we"
  },
  {
    "startMs": 4994080,
    "text": "start with like a baseline"
  },
  {
    "startMs": 4997120,
    "text": "we try to find what model could fit our"
  },
  {
    "startMs": 5000639,
    "text": "use cases."
  },
  {
    "startMs": 5002880,
    "text": "Um,"
  },
  {
    "startMs": 5004560,
    "text": "so you could pick like uh Deep Seek, you"
  },
  {
    "startMs": 5007760,
    "text": "could pick like don't pick like a"
  },
  {
    "startMs": 5010080,
    "text": "Mistral 7B. I mean, it's not good. Uh,"
  },
  {
    "startMs": 5014000,
    "text": "but yeah, so you pick your model and you"
  },
  {
    "startMs": 5018960,
    "text": "want to like have a smaller memory and"
  },
  {
    "startMs": 5021040,
    "text": "you want to try to fit that bigger model"
  },
  {
    "startMs": 5023120,
    "text": "into smaller memory so that you could"
  },
  {
    "startMs": 5025600,
    "text": "save cost on the GPU cost. So you can do"
  },
  {
    "startMs": 5029360,
    "text": "like all those com quantization"
  },
  {
    "startMs": 5032480,
    "text": "then you can apply all those serving"
  },
  {
    "startMs": 5034480,
    "text": "optimizations by using the right serving"
  },
  {
    "startMs": 5036960,
    "text": "engine under the hood. So that can"
  },
  {
    "startMs": 5039679,
    "text": "really provide you that"
  },
  {
    "startMs": 5042320,
    "text": "throughut that you really want."
  },
  {
    "startMs": 5047920,
    "text": "And"
  },
  {
    "startMs": 5050639,
    "text": "now something that you can do uh after"
  },
  {
    "startMs": 5053840,
    "text": "going back home pro because we cannot"
  },
  {
    "startMs": 5056719,
    "text": "like actually go over all the material"
  },
  {
    "startMs": 5058719,
    "text": "here uh is definitely reading about some"
  },
  {
    "startMs": 5062080,
    "text": "of the source informations like"
  },
  {
    "startMs": 5064159,
    "text": "different attention mechanisms different"
  },
  {
    "startMs": 5066320,
    "text": "like these engines like try to just read"
  },
  {
    "startMs": 5069280,
    "text": "the different benchmarks which are"
  },
  {
    "startMs": 5071360,
    "text": "present online as well"
  },
  {
    "startMs": 5074000,
    "text": "and then there are a lot of like"
  },
  {
    "startMs": 5077679,
    "text": "in-depth guides or the next phases of it"
  },
  {
    "startMs": 5080960,
    "text": "which is like learning about some KV"
  },
  {
    "startMs": 5083840,
    "text": "eviction strategies. So world is moving"
  },
  {
    "startMs": 5086719,
    "text": "towards having a separate KV cache"
  },
  {
    "startMs": 5088800,
    "text": "engineering domain. So you want to"
  },
  {
    "startMs": 5091440,
    "text": "understand what's going on in there. So"
  },
  {
    "startMs": 5093280,
    "text": "KV cache KV eviction cache compressions"
  },
  {
    "startMs": 5096719,
    "text": "hybrid memories. So there are like lot"
  },
  {
    "startMs": 5099120,
    "text": "of solutions that are happening around"
  },
  {
    "startMs": 5100719,
    "text": "there. So always try to stick to those"
  },
  {
    "startMs": 5104719,
    "text": "foundations or like the fundamentals or"
  },
  {
    "startMs": 5107280,
    "text": "the first principles and try to see like"
  },
  {
    "startMs": 5110800,
    "text": "which solution basically solves what"
  },
  {
    "startMs": 5112880,
    "text": "problem and whether you actually need"
  },
  {
    "startMs": 5114639,
    "text": "that problem to be solved for your use"
  },
  {
    "startMs": 5117040,
    "text": "case"
  },
  {
    "startMs": 5118800,
    "text": "and then there is like distributed LLM"
  },
  {
    "startMs": 5121360,
    "text": "inference which is like a different"
  },
  {
    "startMs": 5124480,
    "text": "painoint altogether. Uh you would"
  },
  {
    "startMs": 5127760,
    "text": "probably need like a two-hour workshop"
  },
  {
    "startMs": 5129600,
    "text": "there as well."
  },
  {
    "startMs": 5131760,
    "text": "uh to like go over like all the"
  },
  {
    "startMs": 5134159,
    "text": "internals do all the hands-on."
  },
  {
    "startMs": 5141280,
    "text": "Yes. And this is something we are trying"
  },
  {
    "startMs": 5143280,
    "text": "to propose for the AI engineer New York"
  },
  {
    "startMs": 5145679,
    "text": "session uh which is to like dive deeper"
  },
  {
    "startMs": 5148639,
    "text": "into the advanced sections of the LLM"
  },
  {
    "startMs": 5150960,
    "text": "inference. So this workshop was more for"
  },
  {
    "startMs": 5153280,
    "text": "the like beginner and the intermediate"
  },
  {
    "startMs": 5155360,
    "text": "level. Um so in this form we do have"
  },
  {
    "startMs": 5158880,
    "text": "like a feedback as well plus also the"
  },
  {
    "startMs": 5161760,
    "text": "interest. Um if uh you think like we"
  },
  {
    "startMs": 5165760,
    "text": "need certain improvements on certain"
  },
  {
    "startMs": 5167440,
    "text": "sections definitely give that feedback"
  },
  {
    "startMs": 5169280,
    "text": "as well and if you want to see this"
  },
  {
    "startMs": 5171120,
    "text": "workshop in like New York uh fair you I"
  },
  {
    "startMs": 5177360,
    "text": "mean definitely feel free to enroll your"
  },
  {
    "startMs": 5179840,
    "text": "interest."
  },
  {
    "startMs": 5182639,
    "text": "Uhhuh."
  },
  {
    "startMs": 5184880,
    "text": "How is it possible?"
  },
  {
    "startMs": 5188000,
    "text": "Well,"
  },
  {
    "startMs": 5194560,
    "text": "let me just check."
  },
  {
    "startMs": 5206800,
    "text": ">> Huh?"
  },
  {
    "startMs": 5208480,
    "text": ">> URL works, right? Not the QR code. Okay."
  },
  {
    "startMs": 5212320,
    "text": "Probably I forgot to link those two"
  },
  {
    "startMs": 5214639,
    "text": "together."
  },
  {
    "startMs": 5220239,
    "text": "Z"
  },
  {
    "startMs": 5222880,
    "text": "G A five."
  },
  {
    "startMs": 5231760,
    "text": "Okay, cool. Yes. So if you can give that"
  },
  {
    "startMs": 5234960,
    "text": "feedback let me just okay"
  },
  {
    "startMs": 5240480,
    "text": "that will be fine um and yeah I think we"
  },
  {
    "startMs": 5244320,
    "text": "would like to wrap this workshop then"
  },
  {
    "startMs": 5248560,
    "text": "I'm sure like lot of you would be having"
  },
  {
    "startMs": 5250639,
    "text": "a lot of questions so we can take all"
  },
  {
    "startMs": 5252960,
    "text": "those like offline uh we can meet uh and"
  },
  {
    "startMs": 5256000,
    "text": "we can uh like talk about those"
  },
  {
    "startMs": 5257920,
    "text": "questions."
  },
  {
    "startMs": 5259440,
    "text": ">> Yeah sure. Uh thank you everyone. Thanks"
  },
  {
    "startMs": 5262719,
    "text": "for joining. Uh I think it was really"
  },
  {
    "startMs": 5266159,
    "text": "meaningful and all of you like came"
  },
  {
    "startMs": 5269440,
    "text": "here. Uh thanks a lot."
  },
  {
    "startMs": 5271360,
    "text": ">> Yeah. Thanks."
  }
]