[
  {
    "startMs": 1309,
    "text": "[music]"
  },
  {
    "startMs": 12800,
    "text": "Hi everyone. Uh, sorry it's a bit loud"
  },
  {
    "startMs": 15040,
    "text": "in here. Was not expecting this. Um, I'm"
  },
  {
    "startMs": 18080,
    "text": "Siman. I'm a principal scientist at"
  },
  {
    "startMs": 20240,
    "text": "Together AI. Um, I previously did my PhD"
  },
  {
    "startMs": 23519,
    "text": "in the Hazy Research Lab with Chris Ray"
  },
  {
    "startMs": 25760,
    "text": "at Stanford and I'm an incoming"
  },
  {
    "startMs": 27760,
    "text": "professor at Caltech. Um I lead the"
  },
  {
    "startMs": 30720,
    "text": "frontier performance research team at"
  },
  {
    "startMs": 33040,
    "text": "together where we develop systems,"
  },
  {
    "startMs": 34960,
    "text": "frameworks and algorithms to extract as"
  },
  {
    "startMs": 37680,
    "text": "much performance as possible out of"
  },
  {
    "startMs": 39360,
    "text": "modern um AI hardware. Today I want to"
  },
  {
    "startMs": 42800,
    "text": "share a little bit about our"
  },
  {
    "startMs": 44559,
    "text": "contributions towards simplifying the"
  },
  {
    "startMs": 46960,
    "text": "development of uh multi-GPU AI kernels."
  },
  {
    "startMs": 51520,
    "text": "A few years ago, um, GPU utilization"
  },
  {
    "startMs": 54559,
    "text": "used to be limited by poor intraGPU"
  },
  {
    "startMs": 57760,
    "text": "memory access and single GPU kernels."
  },
  {
    "startMs": 61440,
    "text": "But with significant investment in"
  },
  {
    "startMs": 63359,
    "text": "better kernels like flash attention, uh,"
  },
  {
    "startMs": 66240,
    "text": "memory efficient architectures like from"
  },
  {
    "startMs": 68320,
    "text": "deepseek, sparse attentions, mambas and"
  },
  {
    "startMs": 71040,
    "text": "so on, um, and better DSLs, we've sort"
  },
  {
    "startMs": 74080,
    "text": "of shifted the bottleneck to multi-GPU"
  },
  {
    "startMs": 77600,
    "text": "communication."
  },
  {
    "startMs": 81200,
    "text": "During this talk, I'll start by telling"
  },
  {
    "startMs": 83600,
    "text": "you a little bit about why now, why GPU"
  },
  {
    "startMs": 86159,
    "text": "networking now. Then I'll tell you about"
  },
  {
    "startMs": 89200,
    "text": "um the sort of problem space. So what"
  },
  {
    "startMs": 91200,
    "text": "are the challenges in maximizing"
  },
  {
    "startMs": 93040,
    "text": "hardware utilization and development"
  },
  {
    "startMs": 95439,
    "text": "simplicity for multiGPU kernels. Um"
  },
  {
    "startMs": 98400,
    "text": "three, we'll talk a little bit about the"
  },
  {
    "startMs": 100079,
    "text": "fundamentals be behind designing"
  },
  {
    "startMs": 102320,
    "text": "effective multiGGPU kernels. Four, we'll"
  },
  {
    "startMs": 105759,
    "text": "look at whether frontier AI models can"
  },
  {
    "startMs": 109680,
    "text": "uh leverage these fundamental"
  },
  {
    "startMs": 111280,
    "text": "principles. Do they understand them? Can"
  },
  {
    "startMs": 113920,
    "text": "they reason about them? Um, you know, in"
  },
  {
    "startMs": 115920,
    "text": "theory, these models are very good at"
  },
  {
    "startMs": 117520,
    "text": "reasoning. Um, and then five, we'll talk"
  },
  {
    "startMs": 120320,
    "text": "through the results of these frontier"
  },
  {
    "startMs": 121920,
    "text": "models on a benchmark that we've"
  },
  {
    "startMs": 123920,
    "text": "developed called um parallel kernel"
  },
  {
    "startMs": 126240,
    "text": "bench for multiGPU kernel generation"
  },
  {
    "startMs": 128319,
    "text": "evaluation."
  },
  {
    "startMs": 132080,
    "text": "Okay, before we dive into those five"
  },
  {
    "startMs": 134560,
    "text": "parts, just basic preliminaries. So,"
  },
  {
    "startMs": 137760,
    "text": "this is an Nvidia GPU, uh, an H100 GPU"
  },
  {
    "startMs": 141599,
    "text": "that you can see on the screen. Um, I"
  },
  {
    "startMs": 144319,
    "text": "always like to help ground people in GPU"
  },
  {
    "startMs": 147200,
    "text": "kernels via looking at the hardware. So,"
  },
  {
    "startMs": 150080,
    "text": "these rainbow colored dots are"
  },
  {
    "startMs": 152239,
    "text": "processors where actual compute is"
  },
  {
    "startMs": 154480,
    "text": "happening. um all of the you know"
  },
  {
    "startMs": 156560,
    "text": "parallel threads are operating within"
  },
  {
    "startMs": 158800,
    "text": "one of those colored dots and there's"
  },
  {
    "startMs": 161120,
    "text": "typically you know 100 200 of them on"
  },
  {
    "startMs": 164080,
    "text": "modern AI GPUs."
  },
  {
    "startMs": 167120,
    "text": "Um around those processors you can see"
  },
  {
    "startMs": 170640,
    "text": "um some of the memory that these"
  },
  {
    "startMs": 172800,
    "text": "processors retrieve data so large"
  },
  {
    "startMs": 175040,
    "text": "weights activations from. So these"
  },
  {
    "startMs": 178640,
    "text": "rectangles between the colored dots are"
  },
  {
    "startMs": 181200,
    "text": "an L2 cache slightly faster memory. um"
  },
  {
    "startMs": 184879,
    "text": "not a ter like crazy large amount of it."
  },
  {
    "startMs": 187599,
    "text": "And then these black boxes are high"
  },
  {
    "startMs": 189760,
    "text": "bandwidth memory. So when you Nvidia Smi"
  },
  {
    "startMs": 192319,
    "text": "and see you know 80 gigabytes, 1008"
  },
  {
    "startMs": 195440,
    "text": "gigabytes, whatever it is on your GPU,"
  },
  {
    "startMs": 197519,
    "text": "that's that memory."
  },
  {
    "startMs": 199680,
    "text": "Um a GPU is operating a highly parallel"
  },
  {
    "startMs": 203519,
    "text": "program. So multiple threads are"
  },
  {
    "startMs": 206720,
    "text": "combined together in uh into larger"
  },
  {
    "startMs": 209760,
    "text": "coarser units. and we schedule these"
  },
  {
    "startMs": 212400,
    "text": "threads and and blocks onto these"
  },
  {
    "startMs": 214400,
    "text": "processors to perform our AI compute."
  },
  {
    "startMs": 218640,
    "text": "Beyond the GPU, we'll have multiple GPUs"
  },
  {
    "startMs": 221440,
    "text": "and we'll also have, you know, CPUs um"
  },
  {
    "startMs": 224159,
    "text": "that have memory as well."
  },
  {
    "startMs": 229040,
    "text": "Um so to perform computation um the"
  },
  {
    "startMs": 233120,
    "text": "memory that these threads use is going"
  },
  {
    "startMs": 235519,
    "text": "to be stored in a really fast register"
  },
  {
    "startMs": 237760,
    "text": "memory that's right next to the"
  },
  {
    "startMs": 239360,
    "text": "computation units. um simple physics if"
  },
  {
    "startMs": 242879,
    "text": "I am pulling data from very very close"
  },
  {
    "startMs": 246080,
    "text": "to my proc my compute unit it's really"
  },
  {
    "startMs": 248799,
    "text": "fast to get to it because it you know"
  },
  {
    "startMs": 250319,
    "text": "that data is right next to me but"
  },
  {
    "startMs": 252159,
    "text": "there's not a large radius and not a"
  },
  {
    "startMs": 254400,
    "text": "large volume of space that's close by to"
  },
  {
    "startMs": 257120,
    "text": "my process uh my my compute units and so"
  },
  {
    "startMs": 260880,
    "text": "I don't have very much of it so you can"
  },
  {
    "startMs": 262720,
    "text": "see that the fastest memory here the"
  },
  {
    "startMs": 264639,
    "text": "registers is sup is you know 130"
  },
  {
    "startMs": 267360,
    "text": "terabytes per second on an H100 but we"
  },
  {
    "startMs": 269680,
    "text": "don't have very much on of it and as we"
  },
  {
    "startMs": 272160,
    "text": "go to the further away memory we have a"
  },
  {
    "startMs": 274639,
    "text": "lot more of it but it takes longer to"
  },
  {
    "startMs": 276960,
    "text": "reach it. Uh again simple physics."
  },
  {
    "startMs": 280720,
    "text": "So in multiGPU systems in particular um"
  },
  {
    "startMs": 284160,
    "text": "there is a hierarchy of interconnects."
  },
  {
    "startMs": 286720,
    "text": "So we will have something called PCIe um"
  },
  {
    "startMs": 289759,
    "text": "as the channel for CPUGGPU"
  },
  {
    "startMs": 291919,
    "text": "communications. We'll have multiGPU or"
  },
  {
    "startMs": 294880,
    "text": "multi- uh node communications over"
  },
  {
    "startMs": 297280,
    "text": "infiniband TCP and then in the we're"
  },
  {
    "startMs": 301120,
    "text": "going to focus mostly on the Nvidia"
  },
  {
    "startMs": 303199,
    "text": "sphere here. Um in uh the intraGPU"
  },
  {
    "startMs": 306479,
    "text": "regime we'll have NVLink um providing"
  },
  {
    "startMs": 309440,
    "text": "point-to-point connections between GPUs"
  },
  {
    "startMs": 312160,
    "text": "and the NV switch. NV switch connects"
  },
  {
    "startMs": 315840,
    "text": "all NVLink in endpoints into a"
  },
  {
    "startMs": 318320,
    "text": "non-blocking fabric for full GPUGGPU"
  },
  {
    "startMs": 321440,
    "text": "communication and NV switch is exciting"
  },
  {
    "startMs": 324320,
    "text": "because it also provides support for in"
  },
  {
    "startMs": 326720,
    "text": "network offdevice acceleration for oper"
  },
  {
    "startMs": 330960,
    "text": "like communication primitives like"
  },
  {
    "startMs": 333199,
    "text": "multiccast and reductions."
  },
  {
    "startMs": 337680,
    "text": "Okay, so diving in with the"
  },
  {
    "startMs": 339840,
    "text": "preliminaries in mind. Why GPU"
  },
  {
    "startMs": 342400,
    "text": "networking now?"
  },
  {
    "startMs": 344960,
    "text": "So as I mentioned at the beginning,"
  },
  {
    "startMs": 346880,
    "text": "we've really put a lot of effort into"
  },
  {
    "startMs": 349280,
    "text": "making AIO uh more efficient over recent"
  },
  {
    "startMs": 352080,
    "text": "years. Um again, we have uh"
  },
  {
    "startMs": 355120,
    "text": "architectures that use less compute,"
  },
  {
    "startMs": 357039,
    "text": "less memory like Mamba or sparse"
  },
  {
    "startMs": 359440,
    "text": "attentions. We have algorithms that make"
  },
  {
    "startMs": 362479,
    "text": "AI more hardware affair aware like flash"
  },
  {
    "startMs": 365520,
    "text": "attention. We have tools to make it easy"
  },
  {
    "startMs": 368000,
    "text": "to map AI algorithms to the hardware"
  },
  {
    "startMs": 370639,
    "text": "like tileang, mojo, triton, gluon,"
  },
  {
    "startMs": 373759,
    "text": "thunder kittens. Um, and we have new"
  },
  {
    "startMs": 376560,
    "text": "techniques to overlap uh execution"
  },
  {
    "startMs": 379199,
    "text": "across many AI operators very tightly"
  },
  {
    "startMs": 381919,
    "text": "like mega kernels. And we also have"
  },
  {
    "startMs": 384960,
    "text": "tools to make it easy to run on multiple"
  },
  {
    "startMs": 387360,
    "text": "vendor and silicon platforms like"
  },
  {
    "startMs": 389840,
    "text": "thunder mittens for Apple silicon or or"
  },
  {
    "startMs": 392160,
    "text": "hipkittens for AMD and so on."
  },
  {
    "startMs": 395520,
    "text": "Um, at this point we really believe that"
  },
  {
    "startMs": 398080,
    "text": "GPU networking offers many new and"
  },
  {
    "startMs": 400880,
    "text": "exciting opportunities for AI"
  },
  {
    "startMs": 402800,
    "text": "efficiency."
  },
  {
    "startMs": 405360,
    "text": "Modern AI workloads are getting very big"
  },
  {
    "startMs": 407600,
    "text": "and require kernels that span multiple"
  },
  {
    "startMs": 410400,
    "text": "um, you know, GPUs. So on many"
  },
  {
    "startMs": 413120,
    "text": "production um distributed training and"
  },
  {
    "startMs": 416560,
    "text": "inference workloads, communication is"
  },
  {
    "startMs": 418880,
    "text": "increasingly consuming the majority of"
  },
  {
    "startMs": 421120,
    "text": "the runtime and yields low model flop"
  },
  {
    "startMs": 424400,
    "text": "utilization at scale."
  },
  {
    "startMs": 428560,
    "text": "So the uh pace of innovation and"
  },
  {
    "startMs": 431680,
    "text": "diversity of approaches that different"
  },
  {
    "startMs": 433520,
    "text": "hardware providers are taking in their"
  },
  {
    "startMs": 435520,
    "text": "networking stacks is another reason why"
  },
  {
    "startMs": 437759,
    "text": "it's an exciting time to study um you"
  },
  {
    "startMs": 440240,
    "text": "know networking and communication. So we"
  },
  {
    "startMs": 442960,
    "text": "can see here um AMD hardware with uh"
  },
  {
    "startMs": 446240,
    "text": "what's called XGMI interconnects um"
  },
  {
    "startMs": 449039,
    "text": "providing pointto-point links between"
  },
  {
    "startMs": 451520,
    "text": "different GPUs in a scaleup domain. We"
  },
  {
    "startMs": 455440,
    "text": "can see here TPU and uh interconnect uh"
  },
  {
    "startMs": 459120,
    "text": "as well. So the TPU will use a 3D Taurus"
  },
  {
    "startMs": 462400,
    "text": "and also have optical wraparound links."
  },
  {
    "startMs": 465280,
    "text": "So yet another diverse form of the"
  },
  {
    "startMs": 468319,
    "text": "topology and links. Um and then again"
  },
  {
    "startMs": 470880,
    "text": "for Nvidia we'll have the INV um switch"
  },
  {
    "startMs": 474560,
    "text": "which integrates um compute capabilities"
  },
  {
    "startMs": 477599,
    "text": "directly into the interconnect fabric"
  },
  {
    "startMs": 480240,
    "text": "like for in network reductions and then"
  },
  {
    "startMs": 483039,
    "text": "we'll have our NV link providing up to"
  },
  {
    "startMs": 485520,
    "text": "you know 900 gigabytes of unidirectional"
  },
  {
    "startMs": 487759,
    "text": "bandwidth between any two remote GPUs"
  },
  {
    "startMs": 490960,
    "text": "high bandwidth memory um on you know"
  },
  {
    "startMs": 494639,
    "text": "particular generation of Nvidia hardware"
  },
  {
    "startMs": 499120,
    "text": "um Beyond the diversity in the"
  },
  {
    "startMs": 500879,
    "text": "networking stacks, there's also a lot of"
  },
  {
    "startMs": 503120,
    "text": "evolution in um how AI workloads are"
  },
  {
    "startMs": 507759,
    "text": "adapting to take advantage of this"
  },
  {
    "startMs": 509680,
    "text": "hardware and the diversity of types of"
  },
  {
    "startMs": 512479,
    "text": "um you know hardware that we're we're"
  },
  {
    "startMs": 514719,
    "text": "using simultaneously for one AI"
  },
  {
    "startMs": 516800,
    "text": "workload. So um KV cache memory in"
  },
  {
    "startMs": 520159,
    "text": "modern inference systems is going to"
  },
  {
    "startMs": 521680,
    "text": "beworked across GPU, CPU, disk and"
  },
  {
    "startMs": 524640,
    "text": "remote machines. um inference systems"
  },
  {
    "startMs": 527519,
    "text": "increasingly disagregate different steps"
  },
  {
    "startMs": 530160,
    "text": "of inference across different hardware"
  },
  {
    "startMs": 532720,
    "text": "backends. So you could run speculative"
  },
  {
    "startMs": 534800,
    "text": "decoding on some hardware, decode on"
  },
  {
    "startMs": 537200,
    "text": "different hardware, prefill on different"
  },
  {
    "startMs": 538959,
    "text": "hardware. Um hardware is also evolving"
  },
  {
    "startMs": 542240,
    "text": "to have larger scaleup domains than ever"
  },
  {
    "startMs": 544720,
    "text": "before with you know 72 GPUs and a"
  },
  {
    "startMs": 547279,
    "text": "scaleup domain in in the coming um chips"
  },
  {
    "startMs": 550480,
    "text": "and Nvidia planning on a single system"
  },
  {
    "startMs": 552880,
    "text": "in 2027 with 576 GPUs."
  },
  {
    "startMs": 557120,
    "text": "Um at the same time to take advantage of"
  },
  {
    "startMs": 559360,
    "text": "these more uh intensive scaleup domains"
  },
  {
    "startMs": 562000,
    "text": "we're getting richer uh primitives for"
  },
  {
    "startMs": 564320,
    "text": "fine grained control and kernel writing"
  },
  {
    "startMs": 567360,
    "text": "over these domains. So we have something"
  },
  {
    "startMs": 570240,
    "text": "called tensor memory acceleration where"
  },
  {
    "startMs": 572720,
    "text": "we can provide uh perform asynchronous"
  },
  {
    "startMs": 575360,
    "text": "network transfers from the device side"
  },
  {
    "startMs": 577839,
    "text": "um on these GPUs."
  },
  {
    "startMs": 580160,
    "text": "So all of these changes are opening up"
  },
  {
    "startMs": 582880,
    "text": "new opportunities and challenges in um"
  },
  {
    "startMs": 585920,
    "text": "both AI you know how do we build models"
  },
  {
    "startMs": 588640,
    "text": "that take advantage of these trends and"
  },
  {
    "startMs": 590800,
    "text": "in systems."
  },
  {
    "startMs": 593839,
    "text": "Okay, so the problems that we're going"
  },
  {
    "startMs": 596160,
    "text": "to go after um how do we get peak"
  },
  {
    "startMs": 598399,
    "text": "hardware utilization and also"
  },
  {
    "startMs": 600640,
    "text": "development simplicity um for these"
  },
  {
    "startMs": 603040,
    "text": "multiGPU kernels."
  },
  {
    "startMs": 606640,
    "text": "So um it's been very difficult to write"
  },
  {
    "startMs": 610160,
    "text": "multiGGPU kernels and there's a lot of"
  },
  {
    "startMs": 612399,
    "text": "there are a lot of papers a lot of"
  },
  {
    "startMs": 614160,
    "text": "systems reports that document you know"
  },
  {
    "startMs": 616320,
    "text": "challenges here. Um, one of the things"
  },
  {
    "startMs": 619200,
    "text": "here is it's compounded by the fact that"
  },
  {
    "startMs": 621440,
    "text": "communication hardware around GPUs has"
  },
  {
    "startMs": 624240,
    "text": "progressed a lot more slowly relative to"
  },
  {
    "startMs": 627519,
    "text": "compute and memory. Um, so comparing"
  },
  {
    "startMs": 630480,
    "text": "NVIDIA A100's in 2020 to B200s in 2024,"
  },
  {
    "startMs": 636399,
    "text": "u BF16 tensor core speeds improved by"
  },
  {
    "startMs": 639120,
    "text": "7.2x."
  },
  {
    "startMs": 640880,
    "text": "um while intra node communication by"
  },
  {
    "startMs": 643519,
    "text": "just 3x and inter node communication by"
  },
  {
    "startMs": 646480,
    "text": "just 2x. Um and coming back to my points"
  },
  {
    "startMs": 649360,
    "text": "about how diverse networking is right"
  },
  {
    "startMs": 651760,
    "text": "now things like tensor cores that run"
  },
  {
    "startMs": 654320,
    "text": "map moles and our memory hierarchies are"
  },
  {
    "startMs": 657120,
    "text": "pretty consistent and resemble one"
  },
  {
    "startMs": 659200,
    "text": "another across diverse AI vendors and"
  },
  {
    "startMs": 661680,
    "text": "multisilicon. Um but again as I I"
  },
  {
    "startMs": 664480,
    "text": "mentioned the networking stack is"
  },
  {
    "startMs": 665920,
    "text": "something that is really different"
  },
  {
    "startMs": 666959,
    "text": "across vendors still"
  },
  {
    "startMs": 670160,
    "text": "um you know a first step as we went"
  },
  {
    "startMs": 672320,
    "text": "about all this work is to just study the"
  },
  {
    "startMs": 674320,
    "text": "baselines um that are out there. So um"
  },
  {
    "startMs": 677839,
    "text": "one of the popular tools for um"
  },
  {
    "startMs": 681200,
    "text": "communications is this nickel library or"
  },
  {
    "startMs": 684160,
    "text": "Rickle on AMD um that both you know"
  },
  {
    "startMs": 687519,
    "text": "companies respectively spend a lot of"
  },
  {
    "startMs": 689600,
    "text": "you know engineering investment into"
  },
  {
    "startMs": 691839,
    "text": "releasing to make it easy for people to"
  },
  {
    "startMs": 693760,
    "text": "do multiGPU work. Um but they're not"
  },
  {
    "startMs": 696480,
    "text": "very flexible. So they're tuned for bulk"
  },
  {
    "startMs": 698880,
    "text": "transfers for large contiguous chunks of"
  },
  {
    "startMs": 701839,
    "text": "data transfers. And the design really"
  },
  {
    "startMs": 704320,
    "text": "breaks down when you care about peak"
  },
  {
    "startMs": 706079,
    "text": "performance, fine grain communication,"
  },
  {
    "startMs": 708959,
    "text": "um, and sort of non-trivial collectives"
  },
  {
    "startMs": 711600,
    "text": "that you want to fuse together."
  },
  {
    "startMs": 714160,
    "text": "So as a result, you can achieve much"
  },
  {
    "startMs": 716240,
    "text": "higher performance by writing custom"
  },
  {
    "startMs": 718720,
    "text": "communication kernels that directly"
  },
  {
    "startMs": 721040,
    "text": "address these needs."
  },
  {
    "startMs": 723200,
    "text": "If we look at a naive baseline that's"
  },
  {
    "startMs": 725760,
    "text": "representative of very popular libraries"
  },
  {
    "startMs": 728000,
    "text": "in machine learning stacking pietorch"
  },
  {
    "startMs": 730639,
    "text": "with nickel um we can we find that"
  },
  {
    "startMs": 733760,
    "text": "across um you know the many uh problems"
  },
  {
    "startMs": 737200,
    "text": "in our parallel kernel bench benchmark"
  },
  {
    "startMs": 739760,
    "text": "that the majority of these um simple"
  },
  {
    "startMs": 742639,
    "text": "baselines will fall below 50% of their"
  },
  {
    "startMs": 746240,
    "text": "communication aware roof line bound. So"
  },
  {
    "startMs": 750320,
    "text": "there's a lot of room for improvement"
  },
  {
    "startMs": 752160,
    "text": "here."
  },
  {
    "startMs": 753760,
    "text": "Um the current frameworks beyond um"
  },
  {
    "startMs": 756800,
    "text": "nickel which is popular in like Megatron"
  },
  {
    "startMs": 759680,
    "text": "LM, Flex Flow, Nanoflow um you know all"
  },
  {
    "startMs": 763200,
    "text": "again these systems are primarily"
  },
  {
    "startMs": 765040,
    "text": "orchestrating bulk collectives via"
  },
  {
    "startMs": 767040,
    "text": "nickel um and require synchronization"
  },
  {
    "startMs": 770079,
    "text": "before and after data transfers. So"
  },
  {
    "startMs": 772560,
    "text": "beyond th those off-the-shelf libraries,"
  },
  {
    "startMs": 774959,
    "text": "we have compilers and DSLs that exist."
  },
  {
    "startMs": 778560,
    "text": "So there's Triton distributed is one of"
  },
  {
    "startMs": 781440,
    "text": "them. Um and uh you know tile link is"
  },
  {
    "startMs": 784560,
    "text": "another one. Um we have found that it's"
  },
  {
    "startMs": 787920,
    "text": "very difficult to support the rapid pace"
  },
  {
    "startMs": 790399,
    "text": "of networking improvements within these"
  },
  {
    "startMs": 792959,
    "text": "frameworks. So our benchmarks and our"
  },
  {
    "startMs": 795680,
    "text": "papers highlight results where Triton"
  },
  {
    "startMs": 797680,
    "text": "distributed originally tuned around 8 uh"
  },
  {
    "startMs": 800560,
    "text": "H800 GPUs fails to adapt efficiently to"
  },
  {
    "startMs": 804160,
    "text": "other architectures like H100s."
  },
  {
    "startMs": 807920,
    "text": "And then the third category of how"
  },
  {
    "startMs": 810079,
    "text": "people can proceed here is to really"
  },
  {
    "startMs": 812880,
    "text": "handtune specific AI operators one by"
  },
  {
    "startMs": 816079,
    "text": "one. So there's a lot of popular work um"
  },
  {
    "startMs": 818800,
    "text": "DPP um comet ring attention um flux"
  },
  {
    "startMs": 823760,
    "text": "flashdoe"
  },
  {
    "startMs": 825440,
    "text": "um and then several distributed gem"
  },
  {
    "startMs": 827680,
    "text": "kernels from cutless and these methods"
  },
  {
    "startMs": 830720,
    "text": "achieve peak performance but often um"
  },
  {
    "startMs": 834480,
    "text": "they do not you know some of these"
  },
  {
    "startMs": 836399,
    "text": "methods have been designed in one"
  },
  {
    "startMs": 838560,
    "text": "precision and it takes five or six"
  },
  {
    "startMs": 840320,
    "text": "months to scale it to another precision"
  },
  {
    "startMs": 842399,
    "text": "and just the scalability of this hand"
  },
  {
    "startMs": 844399,
    "text": "tuning and um fine grain kernel writing"
  },
  {
    "startMs": 847120,
    "text": "is not very um effective."
  },
  {
    "startMs": 850639,
    "text": "So um with this landscape in mind, our"
  },
  {
    "startMs": 853760,
    "text": "research question was really about"
  },
  {
    "startMs": 855920,
    "text": "whether there is a small set of"
  },
  {
    "startMs": 857839,
    "text": "principles and fundamentals that really"
  },
  {
    "startMs": 860079,
    "text": "governs multiGPU kernel writing and"
  },
  {
    "startMs": 863199,
    "text": "whether there are methods that can"
  },
  {
    "startMs": 865120,
    "text": "leverage those principles if they exist"
  },
  {
    "startMs": 867600,
    "text": "to simplify the development of these"
  },
  {
    "startMs": 869600,
    "text": "kernels."
  },
  {
    "startMs": 872000,
    "text": "Um I'll briefly highlight two works here"
  },
  {
    "startMs": 874959,
    "text": "that um govern like that that represent"
  },
  {
    "startMs": 877440,
    "text": "our approach. So first we think it's"
  },
  {
    "startMs": 880160,
    "text": "important to build our own fundamental"
  },
  {
    "startMs": 882399,
    "text": "understanding and to manually do the"
  },
  {
    "startMs": 884480,
    "text": "work to understand it rather than just"
  },
  {
    "startMs": 886720,
    "text": "throwing say an LLM at the problem. So"
  },
  {
    "startMs": 890240,
    "text": "we spent the time to build out parallel"
  },
  {
    "startMs": 892639,
    "text": "kittens which is a small set of minimal"
  },
  {
    "startMs": 895519,
    "text": "primitives and patterns for multiGPU"
  },
  {
    "startMs": 897920,
    "text": "kernels. Um we use this to understand"
  },
  {
    "startMs": 900880,
    "text": "the trade-offs of multiGPU kernels and"
  },
  {
    "startMs": 903040,
    "text": "to one write a large collection of um"
  },
  {
    "startMs": 906959,
    "text": "peak performance kernels for a variety"
  },
  {
    "startMs": 909760,
    "text": "of parallelism schemes. Um and this I"
  },
  {
    "startMs": 912880,
    "text": "will use to hopefully you know educate"
  },
  {
    "startMs": 915279,
    "text": "and bring us all on the same page on"
  },
  {
    "startMs": 916800,
    "text": "what patterns we figured out. Um and"
  },
  {
    "startMs": 919440,
    "text": "then once we found that there is indeed"
  },
  {
    "startMs": 922079,
    "text": "a small set of trade-offs governing this"
  },
  {
    "startMs": 924480,
    "text": "landscape, we were curious whether"
  },
  {
    "startMs": 926800,
    "text": "models, especially these models right"
  },
  {
    "startMs": 928639,
    "text": "now that claim to be very good at kernel"
  },
  {
    "startMs": 931279,
    "text": "writing and also reasoning um could"
  },
  {
    "startMs": 934079,
    "text": "reason about these trade-offs when we"
  },
  {
    "startMs": 936079,
    "text": "provide them in context to actually"
  },
  {
    "startMs": 938399,
    "text": "generate a bunch of net new multiGPU"
  },
  {
    "startMs": 940800,
    "text": "kernels for us. Unfortunately, we found"
  },
  {
    "startMs": 944000,
    "text": "they were not very good, but we'll dive"
  },
  {
    "startMs": 945519,
    "text": "into more of that at the end."
  },
  {
    "startMs": 948000,
    "text": "Um so just the fundamental section this"
  },
  {
    "startMs": 950959,
    "text": "is going to be more you know educational"
  },
  {
    "startMs": 952720,
    "text": "what are the trade-offs that go into"
  },
  {
    "startMs": 954240,
    "text": "these kernels."
  },
  {
    "startMs": 956320,
    "text": "So there are three main ways to do um"
  },
  {
    "startMs": 959199,
    "text": "intraGPU data transfers. Um there's the"
  },
  {
    "startMs": 963360,
    "text": "per GPU what's called copy engine and"
  },
  {
    "startMs": 966079,
    "text": "this is host or CPU initiated work. It's"
  },
  {
    "startMs": 969600,
    "text": "really good for large message transfers."
  },
  {
    "startMs": 972480,
    "text": "So when your message size, the amount of"
  },
  {
    "startMs": 975120,
    "text": "data being transferred is really big um"
  },
  {
    "startMs": 977600,
    "text": "and it can get to sort of like peak"
  },
  {
    "startMs": 979360,
    "text": "bandwidth um on on the communication"
  },
  {
    "startMs": 982160,
    "text": "side. In contrast, you can use device"
  },
  {
    "startMs": 985600,
    "text": "initiated or GPU initiated transfers via"
  },
  {
    "startMs": 988800,
    "text": "that tensor memory accelerator that I"
  },
  {
    "startMs": 990800,
    "text": "mentioned or via register level"
  },
  {
    "startMs": 992959,
    "text": "instructions called uh in sort of their"
  },
  {
    "startMs": 995600,
    "text": "PTX lingo like LDST red multime. Um, and"
  },
  {
    "startMs": 1000399,
    "text": "the uh T TMA is really nice. These"
  },
  {
    "startMs": 1003519,
    "text": "device initiated ones are really nice"
  },
  {
    "startMs": 1005279,
    "text": "because they can saturate our NVLink"
  },
  {
    "startMs": 1007920,
    "text": "bandwidth using relatively small message"
  },
  {
    "startMs": 1011199,
    "text": "sizes. And this means that they can be"
  },
  {
    "startMs": 1013600,
    "text": "really nice when we're trying to do fine"
  },
  {
    "startMs": 1015279,
    "text": "grain communication rather than sending"
  },
  {
    "startMs": 1017680,
    "text": "bulk amounts of data over the links all"
  },
  {
    "startMs": 1019839,
    "text": "at once coarsely."
  },
  {
    "startMs": 1022639,
    "text": "Um, there are some trade-offs here. So"
  },
  {
    "startMs": 1025360,
    "text": "the copy engine is really nice because"
  },
  {
    "startMs": 1027439,
    "text": "it doesn't take away or you know waste a"
  },
  {
    "startMs": 1030079,
    "text": "lot of our precious registers that I"
  },
  {
    "startMs": 1032240,
    "text": "mentioned are important for compute on"
  },
  {
    "startMs": 1034559,
    "text": "the GPU. Um and it doesn't also use any"
  },
  {
    "startMs": 1038799,
    "text": "of those rainbow colored dots the"
  },
  {
    "startMs": 1040640,
    "text": "processors on our GPU allowing us to"
  },
  {
    "startMs": 1042959,
    "text": "repurpose those for memory or um"
  },
  {
    "startMs": 1045280,
    "text": "computation on our uh you know other"
  },
  {
    "startMs": 1047600,
    "text": "parts of the AI pipeline. Um, TMA, the"
  },
  {
    "startMs": 1051120,
    "text": "second option here, consumes very few"
  },
  {
    "startMs": 1053520,
    "text": "registers, which is why it's nice. Um,"
  },
  {
    "startMs": 1056080,
    "text": "and it also can achieve high utilization"
  },
  {
    "startMs": 1058960,
    "text": "using very few of our processors. So,"
  },
  {
    "startMs": 1061600,
    "text": "it's a nice useful tool for fine grain"
  },
  {
    "startMs": 1064640,
    "text": "overlapping. Um, TMA does have"
  },
  {
    "startMs": 1067120,
    "text": "limitations. It can't effectively take"
  },
  {
    "startMs": 1069360,
    "text": "advantage of these in network um,"
  },
  {
    "startMs": 1071600,
    "text": "computations that I mentioned are"
  },
  {
    "startMs": 1073280,
    "text": "feasible with technologies like NV"
  },
  {
    "startMs": 1075679,
    "text": "switch. And the register level"
  },
  {
    "startMs": 1078080,
    "text": "instructions are really nice for being"
  },
  {
    "startMs": 1080400,
    "text": "able to take advantage of um you know"
  },
  {
    "startMs": 1083919,
    "text": "those those sort of in network"
  },
  {
    "startMs": 1085600,
    "text": "reductions that NV switch offers."
  },
  {
    "startMs": 1088880,
    "text": "So again there are different tradeoffs"
  },
  {
    "startMs": 1091520,
    "text": "different functionalities that these"
  },
  {
    "startMs": 1093360,
    "text": "transfer mechanisms offer and they face"
  },
  {
    "startMs": 1095600,
    "text": "different trade-offs."
  },
  {
    "startMs": 1098720,
    "text": "Second um beyond transfer mechanism the"
  },
  {
    "startMs": 1101440,
    "text": "trade-off is around how to overlap"
  },
  {
    "startMs": 1104640,
    "text": "compute memory and uh communication in"
  },
  {
    "startMs": 1108000,
    "text": "GPU kernels. So there's two main"
  },
  {
    "startMs": 1110960,
    "text": "categories of schedules. The first is"
  },
  {
    "startMs": 1114400,
    "text": "intraSM within one of those rainbow dots"
  },
  {
    "startMs": 1118080,
    "text": "um where we'll we'll have different"
  },
  {
    "startMs": 1119760,
    "text": "warps or threads within that processor"
  },
  {
    "startMs": 1122720,
    "text": "specialized to handle either compute or"
  },
  {
    "startMs": 1125280,
    "text": "one specialized for communication"
  },
  {
    "startMs": 1127360,
    "text": "concurrently. Um we can dedicate you"
  },
  {
    "startMs": 1129679,
    "text": "know different warps to each of these."
  },
  {
    "startMs": 1132160,
    "text": "The challenge with this intm overlapping"
  },
  {
    "startMs": 1135120,
    "text": "is that the communication and"
  },
  {
    "startMs": 1137440,
    "text": "computation pattern really need to like"
  },
  {
    "startMs": 1139360,
    "text": "align and jive with one another. they"
  },
  {
    "startMs": 1141360,
    "text": "need to use the same data as inputs for"
  },
  {
    "startMs": 1144080,
    "text": "the computation and communication. When"
  },
  {
    "startMs": 1147200,
    "text": "they don't align, you could use"
  },
  {
    "startMs": 1148960,
    "text": "something like interm um schedules that"
  },
  {
    "startMs": 1152160,
    "text": "are shown on your right here where we'll"
  },
  {
    "startMs": 1155760,
    "text": "now have each of the different rainbow"
  },
  {
    "startMs": 1157600,
    "text": "colored dots on our GPUs, those"
  },
  {
    "startMs": 1159120,
    "text": "different processors specialized to"
  },
  {
    "startMs": 1161360,
    "text": "compute communication and memory. Um and"
  },
  {
    "startMs": 1166559,
    "text": "so this this is nice when the colonel"
  },
  {
    "startMs": 1169120,
    "text": "would others wise need to split across"
  },
  {
    "startMs": 1171840,
    "text": "resources like the register file or"
  },
  {
    "startMs": 1173679,
    "text": "shared memory um across these different"
  },
  {
    "startMs": 1176720,
    "text": "steps in misaligned ways."
  },
  {
    "startMs": 1181200,
    "text": "Um this is also really nice when it's"
  },
  {
    "startMs": 1183360,
    "text": "hard to maximize NVLink traversal with"
  },
  {
    "startMs": 1185919,
    "text": "intram"
  },
  {
    "startMs": 1187520,
    "text": "overlapping."
  },
  {
    "startMs": 1190080,
    "text": "So I just wanted to highlight one quick"
  },
  {
    "startMs": 1192080,
    "text": "example here where each of the patterns"
  },
  {
    "startMs": 1194320,
    "text": "excels in popular you know AI uh uh like"
  },
  {
    "startMs": 1197840,
    "text": "kind of patterns that you'll see. So on"
  },
  {
    "startMs": 1200480,
    "text": "a gem plus uh reduced uh scatter here we"
  },
  {
    "startMs": 1204559,
    "text": "can see that the um intraSM overlapping"
  },
  {
    "startMs": 1208240,
    "text": "scheduleuler schedule is very effective"
  },
  {
    "startMs": 1211360,
    "text": "in the gem plus all reduce we can see"
  },
  {
    "startMs": 1214240,
    "text": "that the interSM which again leverages"
  },
  {
    "startMs": 1217120,
    "text": "the in network reductions of envy switch"
  },
  {
    "startMs": 1219520,
    "text": "is very effective. So we face these"
  },
  {
    "startMs": 1222559,
    "text": "trade-offs and you can read more about"
  },
  {
    "startMs": 1224880,
    "text": "um the design decisions that go into"
  },
  {
    "startMs": 1226480,
    "text": "them in our parallel kittens paper."
  },
  {
    "startMs": 1230159,
    "text": "Um, and then finally, ideally,"
  },
  {
    "startMs": 1231760,
    "text": "abstraction should allow the developer"
  },
  {
    "startMs": 1233840,
    "text": "flexibility to control how they're"
  },
  {
    "startMs": 1235760,
    "text": "buffering and synchronizing between data"
  },
  {
    "startMs": 1239200,
    "text": "senders and receivers."
  },
  {
    "startMs": 1241760,
    "text": "So, we encapsulated these ideas into"
  },
  {
    "startMs": 1244080,
    "text": "parallel kittens. Again, a simple set of"
  },
  {
    "startMs": 1246799,
    "text": "programming primitives and templates for"
  },
  {
    "startMs": 1249280,
    "text": "these multiGPU kernels. Parallel Kittens"
  },
  {
    "startMs": 1252240,
    "text": "is used in production at Together AI as"
  },
  {
    "startMs": 1254960,
    "text": "well as our partner um you know Cursor"
  },
  {
    "startMs": 1257039,
    "text": "and other uh companies in the AI space."
  },
  {
    "startMs": 1261039,
    "text": "Um here's some sample code. I won't"
  },
  {
    "startMs": 1263360,
    "text": "spend too much time here, but we usually"
  },
  {
    "startMs": 1265360,
    "text": "add roughly a dozen lines of code over a"
  },
  {
    "startMs": 1267840,
    "text": "single GPU kernel to insert these"
  },
  {
    "startMs": 1270159,
    "text": "multiGPU primitives."
  },
  {
    "startMs": 1273840,
    "text": "Um, and you can see here across data"
  },
  {
    "startMs": 1276559,
    "text": "sequence and expert parallelism how our"
  },
  {
    "startMs": 1279919,
    "text": "parallel kittens kernels are achieving"
  },
  {
    "startMs": 1282320,
    "text": "state-of-the-art results um, compared to"
  },
  {
    "startMs": 1284640,
    "text": "strong reference baselines. And you can"
  },
  {
    "startMs": 1286960,
    "text": "check out our repo to learn more."
  },
  {
    "startMs": 1290799,
    "text": "Okay, so we understand a little bit"
  },
  {
    "startMs": 1293120,
    "text": "about um the trade-offs that underly"
  },
  {
    "startMs": 1295760,
    "text": "these multiGPU kernels and there's just"
  },
  {
    "startMs": 1297919,
    "text": "a couple main, you know, ones that"
  },
  {
    "startMs": 1299600,
    "text": "exist. So can models reason through them"
  },
  {
    "startMs": 1302880,
    "text": "um and give us these uh you know kernels"
  },
  {
    "startMs": 1305840,
    "text": "models are getting better at reasoning"
  },
  {
    "startMs": 1307919,
    "text": "today. Um do they generalize well to"
  },
  {
    "startMs": 1310000,
    "text": "these problems or are we benchmaxed on"
  },
  {
    "startMs": 1313200,
    "text": "you know benchmarks of the past which"
  },
  {
    "startMs": 1315039,
    "text": "are more single GPU centric"
  },
  {
    "startMs": 1318320,
    "text": "uh models right now are showing really"
  },
  {
    "startMs": 1320240,
    "text": "promising results on single GPU uh"
  },
  {
    "startMs": 1322720,
    "text": "benchmarks. So it's a ripe time to to"
  },
  {
    "startMs": 1325440,
    "text": "extend it."
  },
  {
    "startMs": 1327360,
    "text": "In our benchmark parallel kernel bench,"
  },
  {
    "startMs": 1330000,
    "text": "each task presents the model with an"
  },
  {
    "startMs": 1332080,
    "text": "unoptimized reference implementation"
  },
  {
    "startMs": 1334400,
    "text": "written in PyTorch with torch"
  },
  {
    "startMs": 1336799,
    "text": "distributed um nickel operations and"
  },
  {
    "startMs": 1339760,
    "text": "then a system topology that specifies"
  },
  {
    "startMs": 1342400,
    "text": "the number of ranks and intraode"
  },
  {
    "startMs": 1344320,
    "text": "hardware configuration. And the model"
  },
  {
    "startMs": 1346480,
    "text": "needs to rewrite the reference into a"
  },
  {
    "startMs": 1348640,
    "text": "performance CUDA kernel that uh uses"
  },
  {
    "startMs": 1351679,
    "text": "unified virtual addressing."
  },
  {
    "startMs": 1357360,
    "text": "The um multiGPU problem space expands"
  },
  {
    "startMs": 1360480,
    "text": "combinatorally beyond single GPU um"
  },
  {
    "startMs": 1364080,
    "text": "cases. So a standard transformer layer"
  },
  {
    "startMs": 1366880,
    "text": "can be parallelized across data"
  },
  {
    "startMs": 1369120,
    "text": "sequence, tensor, context, layer,"
  },
  {
    "startMs": 1371200,
    "text": "pipeline and expert dimensions. and each"
  },
  {
    "startMs": 1373919,
    "text": "composition um induces a different"
  },
  {
    "startMs": 1376080,
    "text": "communication pass uh pattern. So to"
  },
  {
    "startMs": 1378720,
    "text": "make sure that our benchmark has high"
  },
  {
    "startMs": 1381120,
    "text": "coverage over the representative types"
  },
  {
    "startMs": 1383280,
    "text": "of multiGPU problems um we created this"
  },
  {
    "startMs": 1386640,
    "text": "taxonomy um that you can read more about"
  },
  {
    "startMs": 1389039,
    "text": "in our paper and then picked"
  },
  {
    "startMs": 1390880,
    "text": "representative problems for each part of"
  },
  {
    "startMs": 1392880,
    "text": "the taxonomy."
  },
  {
    "startMs": 1395280,
    "text": "Um these are all patterns that arise in"
  },
  {
    "startMs": 1398400,
    "text": "real AI workloads from inference to RL"
  },
  {
    "startMs": 1400880,
    "text": "to post-training."
  },
  {
    "startMs": 1403440,
    "text": "There are 87 problems overall um drawn"
  },
  {
    "startMs": 1406400,
    "text": "from GitHub repositories that we found"
  },
  {
    "startMs": 1409440,
    "text": "to be very informative. Um and we uh and"
  },
  {
    "startMs": 1414159,
    "text": "like optimized library implementations"
  },
  {
    "startMs": 1416159,
    "text": "and DSLs that people have written"
  },
  {
    "startMs": 1418000,
    "text": "multiGPU kernels in."
  },
  {
    "startMs": 1420799,
    "text": "Um, we wanted to really make sure that"
  },
  {
    "startMs": 1422720,
    "text": "solving PK this parallel kernel bench"
  },
  {
    "startMs": 1425440,
    "text": "would lead to net new useful production"
  },
  {
    "startMs": 1427760,
    "text": "kernels rather than artificial or"
  },
  {
    "startMs": 1429679,
    "text": "useless kernels."
  },
  {
    "startMs": 1431919,
    "text": "Okay, so that's parallel kernel bench."
  },
  {
    "startMs": 1434400,
    "text": "How do models perform?"
  },
  {
    "startMs": 1438799,
    "text": "So we measured to sorry that this is a"
  },
  {
    "startMs": 1440960,
    "text": "bit small. We measured two main metrics."
  },
  {
    "startMs": 1443200,
    "text": "Um pass at K which is the number of"
  },
  {
    "startMs": 1446000,
    "text": "correct kernels generated after K"
  },
  {
    "startMs": 1448559,
    "text": "attempts and then fast um one at K which"
  },
  {
    "startMs": 1453279,
    "text": "counts solutions that are both correct"
  },
  {
    "startMs": 1456080,
    "text": "and outperform the speed of the pi torch"
  },
  {
    "startMs": 1459200,
    "text": "plus nickel baseline."
  },
  {
    "startMs": 1461600,
    "text": "Um so pass K just correctness fast one"
  },
  {
    "startMs": 1465520,
    "text": "at K is whether you're getting a 1x or"
  },
  {
    "startMs": 1468240,
    "text": "higher speed up over the reference. So"
  },
  {
    "startMs": 1470159,
    "text": "performanceoriented"
  },
  {
    "startMs": 1472320,
    "text": "um we found that in the zeroot setting"
  },
  {
    "startMs": 1475120,
    "text": "the best of the frontier models we tried"
  },
  {
    "startMs": 1477520,
    "text": "solves 28 out of 87 problems and 22 of"
  },
  {
    "startMs": 1481279,
    "text": "those problems are faster than the"
  },
  {
    "startMs": 1482880,
    "text": "pietorch plus nickel baseline."
  },
  {
    "startMs": 1485840,
    "text": "If we make multiple samples, you know,"
  },
  {
    "startMs": 1487840,
    "text": "standard uh scaling up test time"
  },
  {
    "startMs": 1490400,
    "text": "compute, we can uh get that number from"
  },
  {
    "startMs": 1494320,
    "text": "say like to to 36 correct solutions, but"
  },
  {
    "startMs": 1497840,
    "text": "the fast uh one performance still"
  },
  {
    "startMs": 1501120,
    "text": "plateaus out at roughly 31%. So we don't"
  },
  {
    "startMs": 1505840,
    "text": "see much room from continuing to scale"
  },
  {
    "startMs": 1507919,
    "text": "there as we increase the number of"
  },
  {
    "startMs": 1509679,
    "text": "parallel generations."
  },
  {
    "startMs": 1512320,
    "text": "Um we find that the correct once"
  },
  {
    "startMs": 1514159,
    "text": "correctness is established um speedups"
  },
  {
    "startMs": 1516400,
    "text": "naturally come from eliminating nickel"
  },
  {
    "startMs": 1518400,
    "text": "staging overhead in favor of direct"
  },
  {
    "startMs": 1520880,
    "text": "NVLink loads in stores."
  },
  {
    "startMs": 1524240,
    "text": "Um the success patterns here are really"
  },
  {
    "startMs": 1527919,
    "text": "concentrated into familiar patterns. So"
  },
  {
    "startMs": 1530480,
    "text": "collective primitives, tensor parallel"
  },
  {
    "startMs": 1532480,
    "text": "gems and Ulyses style context"
  },
  {
    "startMs": 1534720,
    "text": "parallelism. So in other words, patterns"
  },
  {
    "startMs": 1537360,
    "text": "that we see heavily represented um on"
  },
  {
    "startMs": 1540080,
    "text": "the internet rather than necessarily"
  },
  {
    "startMs": 1543679,
    "text": "patterns that the model has used its"
  },
  {
    "startMs": 1545760,
    "text": "reasoning abilities to think through."
  },
  {
    "startMs": 1550880,
    "text": "Okay, even the best available model that"
  },
  {
    "startMs": 1553600,
    "text": "we benchmarked here GPT 5.5 drops off"
  },
  {
    "startMs": 1556559,
    "text": "very quickly as the speed up threshold"
  },
  {
    "startMs": 1559520,
    "text": "increases. So on the x-axis here, we're"
  },
  {
    "startMs": 1562880,
    "text": "increasing the speed up threshold over"
  },
  {
    "startMs": 1564799,
    "text": "that pietor torch plus nickel baseline."
  },
  {
    "startMs": 1567200,
    "text": "And then we're showing the number of"
  },
  {
    "startMs": 1569279,
    "text": "correct kernels that are faster than"
  },
  {
    "startMs": 1571520,
    "text": "that baseline or this much faster than"
  },
  {
    "startMs": 1573840,
    "text": "the baseline on the y-axis."
  },
  {
    "startMs": 1576880,
    "text": "So GPT 5.5 is this orange line here and"
  },
  {
    "startMs": 1580640,
    "text": "then DeepSeek V4 Pro is the aqua line at"
  },
  {
    "startMs": 1584240,
    "text": "the bottom."
  },
  {
    "startMs": 1588720,
    "text": "um we found that there's deeper issues"
  },
  {
    "startMs": 1591600,
    "text": "than CUDA syntax. So we found that if"
  },
  {
    "startMs": 1594320,
    "text": "you do multiple sampling or have the"
  },
  {
    "startMs": 1596640,
    "text": "model kind of look at its errors and"
  },
  {
    "startMs": 1598960,
    "text": "correct them, it can often compile the"
  },
  {
    "startMs": 1601600,
    "text": "kernels. But the models really struggle"
  },
  {
    "startMs": 1604400,
    "text": "to reason through the tradeoffs that we"
  },
  {
    "startMs": 1606559,
    "text": "talked about in the prior section. um"
  },
  {
    "startMs": 1608960,
    "text": "collective ordering, data partitioning,"
  },
  {
    "startMs": 1610960,
    "text": "thinking about intra versus interm"
  },
  {
    "startMs": 1613120,
    "text": "scheduling or deciding between the"
  },
  {
    "startMs": 1615200,
    "text": "different transfer mechanisms. Um we"
  },
  {
    "startMs": 1618000,
    "text": "find that they often do not use things"
  },
  {
    "startMs": 1620559,
    "text": "like the register transfer instructions"
  },
  {
    "startMs": 1622640,
    "text": "or tensor memory acceleration when"
  },
  {
    "startMs": 1624799,
    "text": "writing the kernels."
  },
  {
    "startMs": 1627760,
    "text": "Um we wanted to try a pretty simple"
  },
  {
    "startMs": 1630400,
    "text": "instantiation of something like a clawed"
  },
  {
    "startMs": 1632880,
    "text": "code um coding agent. So, we took uh the"
  },
  {
    "startMs": 1637039,
    "text": "mini sui agent multi-turn harness and"
  },
  {
    "startMs": 1640080,
    "text": "one of the best performing models,"
  },
  {
    "startMs": 1641679,
    "text": "Gemini 3 Pro, and gave it access to a"
  },
  {
    "startMs": 1644720,
    "text": "local bash environment to sort of mimic"
  },
  {
    "startMs": 1646880,
    "text": "the standard claude code setup. We found"
  },
  {
    "startMs": 1649600,
    "text": "that this could help the agent um go"
  },
  {
    "startMs": 1653039,
    "text": "from solving 24 problems to 35 of 87"
  },
  {
    "startMs": 1656799,
    "text": "problems um with um 26 achieving over a"
  },
  {
    "startMs": 1661600,
    "text": "1x speed up over the reference. But we"
  },
  {
    "startMs": 1664880,
    "text": "found that as we scaled the amount of"
  },
  {
    "startMs": 1666799,
    "text": "time um the performance plateaued as uh"
  },
  {
    "startMs": 1670240,
    "text": "and we find that additional techniques"
  },
  {
    "startMs": 1672559,
    "text": "would be required to continue seeing the"
  },
  {
    "startMs": 1674640,
    "text": "scaling there. Again these results are"
  },
  {
    "startMs": 1677279,
    "text": "discussed in more detail in our paper."
  },
  {
    "startMs": 1680080,
    "text": "Um we think this is a really exciting"
  },
  {
    "startMs": 1682159,
    "text": "you know just to wrap up here we hope"
  },
  {
    "startMs": 1684240,
    "text": "that people out here can use both"
  },
  {
    "startMs": 1685919,
    "text": "parallel kittens and parallel kernel"
  },
  {
    "startMs": 1687919,
    "text": "bench. We think that the kernels"
  },
  {
    "startMs": 1689919,
    "text": "generated from solving parallel kernel"
  },
  {
    "startMs": 1691919,
    "text": "bench will lead to net new production"
  },
  {
    "startMs": 1694320,
    "text": "kernels that are important bottlenecks"
  },
  {
    "startMs": 1696799,
    "text": "for inference in RL right now. um"
  },
  {
    "startMs": 1699440,
    "text": "they're you know we tried our best to"
  },
  {
    "startMs": 1701120,
    "text": "make them you know non-artificial and we"
  },
  {
    "startMs": 1703279,
    "text": "can already see signs of life and"
  },
  {
    "startMs": 1705360,
    "text": "exciting results where people have not"
  },
  {
    "startMs": 1707760,
    "text": "invested a bunch of time to handw write"
  },
  {
    "startMs": 1709360,
    "text": "a multiGPU kernel and we've gotten some"
  },
  {
    "startMs": 1711679,
    "text": "net new interesting ones like this Nemo"
  },
  {
    "startMs": 1714720,
    "text": "vocab parallel um you know filtering"
  },
  {
    "startMs": 1717600,
    "text": "kernel um a hyena architecture context"
  },
  {
    "startMs": 1721919,
    "text": "parallelism kernel"
  },
  {
    "startMs": 1724720,
    "text": "and uh the SAM 3 video segmentation"
  },
  {
    "startMs": 1728000,
    "text": "model um IOU suppression kernel."
  },
  {
    "startMs": 1732320,
    "text": "Just to conclude here um we're really"
  },
  {
    "startMs": 1734399,
    "text": "excited about um how uh well just"
  },
  {
    "startMs": 1739120,
    "text": "talking about the lessons. First off, we"
  },
  {
    "startMs": 1740960,
    "text": "think there aren't that many patterns"
  },
  {
    "startMs": 1742960,
    "text": "that are involved in writing intragpu"
  },
  {
    "startMs": 1745760,
    "text": "effective kernels again encapsulated by"
  },
  {
    "startMs": 1748240,
    "text": "our small set of programming primitives."
  },
  {
    "startMs": 1751200,
    "text": "Um, but unfortunately models do not"
  },
  {
    "startMs": 1753520,
    "text": "currently understand how to reason"
  },
  {
    "startMs": 1755120,
    "text": "through these trade-offs even when we"
  },
  {
    "startMs": 1756720,
    "text": "provide them in context. Um, we're"
  },
  {
    "startMs": 1759200,
    "text": "really excited about methods that can"
  },
  {
    "startMs": 1761200,
    "text": "help attack this benchmark. We're"
  },
  {
    "startMs": 1763440,
    "text": "excited about architectures that can"
  },
  {
    "startMs": 1765520,
    "text": "grow with the trends of how networking"
  },
  {
    "startMs": 1767520,
    "text": "stacks are evolving. Um, you know,"
  },
  {
    "startMs": 1769600,
    "text": "larger scale up domains shift away from"
  },
  {
    "startMs": 1772159,
    "text": "scale out um and massive onchip memory"
  },
  {
    "startMs": 1775039,
    "text": "structures. And we hope that uh you know"
  },
  {
    "startMs": 1778080,
    "text": "we can also extend and you you can feel"
  },
  {
    "startMs": 1780080,
    "text": "free to reach out to me at my email."
  },
  {
    "startMs": 1782480,
    "text": "Thanks."
  }
]
