[
  {
    "startMs": 12400,
    "text": "Hi everyone, I'm Tis. Uh, so I'm going"
  },
  {
    "startMs": 15360,
    "text": "to be explaining how we make models"
  },
  {
    "startMs": 16800,
    "text": "three times faster with auto research."
  },
  {
    "startMs": 19039,
    "text": "Uh, so previous to this, uh, I actually"
  },
  {
    "startMs": 21119,
    "text": "used to do GPU mining in my dorm room"
  },
  {
    "startMs": 22960,
    "text": "with 1080 Ti all the way up to working"
  },
  {
    "startMs": 25039,
    "text": "at Tesla on inference optimization for"
  },
  {
    "startMs": 27039,
    "text": "Tesla AI."
  },
  {
    "startMs": 29920,
    "text": "Uh but first what is auto research? So"
  },
  {
    "startMs": 32000,
    "text": "auto research is this framework from"
  },
  {
    "startMs": 33360,
    "text": "Andre Kapathy where uh you basically set"
  },
  {
    "startMs": 36320,
    "text": "up a framework for an agent to move"
  },
  {
    "startMs": 38160,
    "text": "towards a goal that you define uh and"
  },
  {
    "startMs": 40800,
    "text": "all you have to do basically is say at"
  },
  {
    "startMs": 42480,
    "text": "the high level what you want it to do"
  },
  {
    "startMs": 43680,
    "text": "and it will try things as it goes and"
  },
  {
    "startMs": 45680,
    "text": "move back and forth uh towards that"
  },
  {
    "startMs": 47360,
    "text": "goal."
  },
  {
    "startMs": 49600,
    "text": "In actuality it's really just a while"
  },
  {
    "startMs": 51039,
    "text": "loop. The agent proposes a solution. You"
  },
  {
    "startMs": 53280,
    "text": "have a setup to to define what's correct"
  },
  {
    "startMs": 56079,
    "text": "benchmark it for us. uh and then you"
  },
  {
    "startMs": 58719,
    "text": "keep or revert that and you do this in a"
  },
  {
    "startMs": 60079,
    "text": "loop until your goal is met."
  },
  {
    "startMs": 62719,
    "text": "And so this is very well aligned to GPU"
  },
  {
    "startMs": 64559,
    "text": "kernels. Uh so if you don't know what a"
  },
  {
    "startMs": 66320,
    "text": "GPU kernel is, it's basically a"
  },
  {
    "startMs": 67680,
    "text": "low-level operator. And in a Nvidia GPU,"
  },
  {
    "startMs": 69520,
    "text": "this is a CUDA kernel. Uh and this is um"
  },
  {
    "startMs": 72159,
    "text": "an operator that's used by the GPU to"
  },
  {
    "startMs": 74240,
    "text": "operate like millions of times in"
  },
  {
    "startMs": 75439,
    "text": "parallel, for example, like a matrix"
  },
  {
    "startMs": 76640,
    "text": "multiply or an expert computation."
  },
  {
    "startMs": 80720,
    "text": "And why are GPUs such a good fit for"
  },
  {
    "startMs": 82240,
    "text": "auto research? It's because they're"
  },
  {
    "startMs": 83439,
    "text": "super verifiable. you can verify them"
  },
  {
    "startMs": 85200,
    "text": "for correctness and speed and that's"
  },
  {
    "startMs": 86400,
    "text": "basically all you need for your auto"
  },
  {
    "startMs": 87520,
    "text": "research framework."
  },
  {
    "startMs": 90080,
    "text": "Uh so in actuality there are some"
  },
  {
    "startMs": 91680,
    "text": "caveats here. Um the auto research"
  },
  {
    "startMs": 94079,
    "text": "framework is really good for like"
  },
  {
    "startMs": 95439,
    "text": "picking block sizes and these tiny"
  },
  {
    "startMs": 96960,
    "text": "parameters but they're also still really"
  },
  {
    "startMs": 98640,
    "text": "bad at the high level idea like seeing"
  },
  {
    "startMs": 100240,
    "text": "like I want to use this GPU and I"
  },
  {
    "startMs": 102640,
    "text": "actually want to pipeline it. It's not"
  },
  {
    "startMs": 103840,
    "text": "going to come up with these"
  },
  {
    "startMs": 104560,
    "text": "groundbreaking ideas. So it's still up"
  },
  {
    "startMs": 105680,
    "text": "to the human to do that. But the actual"
  },
  {
    "startMs": 106960,
    "text": "implementation is very straightforward"
  },
  {
    "startMs": 108880,
    "text": "once you once you have the idea laid"
  },
  {
    "startMs": 110560,
    "text": "out. So it is still your job to have"
  },
  {
    "startMs": 113360,
    "text": "good ideas is what I'm saying. Uh and so"
  },
  {
    "startMs": 115600,
    "text": "the actual secret formula here is you"
  },
  {
    "startMs": 117360,
    "text": "have the good ideas auto research picks"
  },
  {
    "startMs": 119360,
    "text": "out the parameters and everything to"
  },
  {
    "startMs": 120719,
    "text": "verify that it actually works uh and go"
  },
  {
    "startMs": 122640,
    "text": "moves toward that verifiable goal of it"
  },
  {
    "startMs": 124240,
    "text": "being x times faster and uh still"
  },
  {
    "startMs": 126560,
    "text": "correct and you mix that with billions"
  },
  {
    "startMs": 128720,
    "text": "of tokens of your favorite model and"
  },
  {
    "startMs": 130479,
    "text": "that results in kernels that beat hand"
  },
  {
    "startMs": 131920,
    "text": "tuning."
  },
  {
    "startMs": 133920,
    "text": "Uh so what are the actual things you"
  },
  {
    "startMs": 135200,
    "text": "care about when you're when you're when"
  },
  {
    "startMs": 136800,
    "text": "you're writing a custom kernel or you're"
  },
  {
    "startMs": 138000,
    "text": "having your agent write a custom kernel?"
  },
  {
    "startMs": 139360,
    "text": "So the three main things you can have"
  },
  {
    "startMs": 140720,
    "text": "are a compute bottleneck uh a memory"
  },
  {
    "startMs": 143040,
    "text": "bottleneck or you just have excessive"
  },
  {
    "startMs": 144640,
    "text": "overhead from uh too many kernels being"
  },
  {
    "startMs": 146879,
    "text": "launched and you can do you can view"
  },
  {
    "startMs": 148480,
    "text": "these things with by profiling with a"
  },
  {
    "startMs": 149920,
    "text": "profiler like NSIS for example which is"
  },
  {
    "startMs": 152480,
    "text": "Nvidia's profiler. Uh, and so this this"
  },
  {
    "startMs": 155519,
    "text": "gra this page looks super daunting, but"
  },
  {
    "startMs": 157599,
    "text": "basically your job as a human is to look"
  },
  {
    "startMs": 159120,
    "text": "at the top here and be like this is"
  },
  {
    "startMs": 161040,
    "text": "dumb. Uh, we are loading 32k chunks into"
  },
  {
    "startMs": 164000,
    "text": "context. Uh, and we don't actually need"
  },
  {
    "startMs": 166480,
    "text": "to for this deepseek attention for"
  },
  {
    "startMs": 168000,
    "text": "example. Uh, and we should only be doing"
  },
  {
    "startMs": 169760,
    "text": "it every 32k instead. And so at a high"
  },
  {
    "startMs": 171760,
    "text": "level, all you have to be telling auto"
  },
  {
    "startMs": 172959,
    "text": "research is this top method is dumb."
  },
  {
    "startMs": 174959,
    "text": "Let's pipeline it instead. And"
  },
  {
    "startMs": 176239,
    "text": "everything else like the sizing, the"
  },
  {
    "startMs": 177519,
    "text": "chunk sizing, the context chunks, that"
  },
  {
    "startMs": 179280,
    "text": "all should just be decided by auto"
  },
  {
    "startMs": 180560,
    "text": "research."
  },
  {
    "startMs": 183040,
    "text": "And so my problem is that I really love"
  },
  {
    "startMs": 184720,
    "text": "cheap GPUs. And so that means like GPUs"
  },
  {
    "startMs": 186959,
    "text": "that don't have NVLink for example uh is"
  },
  {
    "startMs": 189200,
    "text": "example of like GPUs you can get for"
  },
  {
    "startMs": 190400,
    "text": "cheaper. Uh but the problem is you don't"
  },
  {
    "startMs": 192720,
    "text": "actually have kernels off the shelf for"
  },
  {
    "startMs": 193840,
    "text": "those. And so you have to come up with a"
  },
  {
    "startMs": 195200,
    "text": "auto research framework as well as a"
  },
  {
    "startMs": 196480,
    "text": "custom harness. So what goes into the"
  },
  {
    "startMs": 197680,
    "text": "harness to make this really good."
  },
  {
    "startMs": 200239,
    "text": "Uh so one thing you really need to make"
  },
  {
    "startMs": 202159,
    "text": "sure your agent is aware of is the"
  },
  {
    "startMs": 203519,
    "text": "hardware. And so on a B200 for example,"
  },
  {
    "startMs": 205599,
    "text": "you need to make sure it has context of"
  },
  {
    "startMs": 207760,
    "text": "uh the warps. It has T-M TMA. And so if"
  },
  {
    "startMs": 210239,
    "text": "you don't know what these are, these are"
  },
  {
    "startMs": 211360,
    "text": "just uh low-level operators that you"
  },
  {
    "startMs": 213440,
    "text": "have um on a specific hardware. And this"
  },
  {
    "startMs": 216799,
    "text": "changes generation to generation. Like"
  },
  {
    "startMs": 218799,
    "text": "H200 won't have T-M for example. That's"
  },
  {
    "startMs": 221120,
    "text": "a new feature that came out with B200,"
  },
  {
    "startMs": 222640,
    "text": "which is why you need to have this in"
  },
  {
    "startMs": 223760,
    "text": "context. Um and so this this basically"
  },
  {
    "startMs": 226080,
    "text": "is just like bunch of MD files you need"
  },
  {
    "startMs": 227519,
    "text": "to give so it has context."
  },
  {
    "startMs": 230879,
    "text": "Other thing you need to make sure your"
  },
  {
    "startMs": 231840,
    "text": "agent has context of is the model. And"
  },
  {
    "startMs": 233360,
    "text": "so every new model like Deepseek Flash"
  },
  {
    "startMs": 235040,
    "text": "comes out with like new tricks. like"
  },
  {
    "startMs": 236640,
    "text": "deepseek had two new attentions that was"
  },
  {
    "startMs": 238640,
    "text": "released in the deepseek flash for"
  },
  {
    "startMs": 240560,
    "text": "deepseek v4. Uh so compressed sparse"
  },
  {
    "startMs": 243040,
    "text": "attention hierarchal compress and if you"
  },
  {
    "startMs": 244640,
    "text": "don't do this the model will 100%"
  },
  {
    "startMs": 246400,
    "text": "hallucinate uh the actual attention"
  },
  {
    "startMs": 248799,
    "text": "mechanism and you will get useless"
  },
  {
    "startMs": 249920,
    "text": "kernels."
  },
  {
    "startMs": 252560,
    "text": "Uh by far the biggest problem when"
  },
  {
    "startMs": 254239,
    "text": "you're doing this is going to be reward"
  },
  {
    "startMs": 255280,
    "text": "hacking. And so if you were to tell your"
  },
  {
    "startMs": 257680,
    "text": "kernel engineer co-orker I need to make"
  },
  {
    "startMs": 260160,
    "text": "uh the GPU this GPU kernel faster. uh"
  },
  {
    "startMs": 263280,
    "text": "it's obviously not going to your human"
  },
  {
    "startMs": 264639,
    "text": "co-workers is not going to go in and do"
  },
  {
    "startMs": 266000,
    "text": "some stuff that's going to make it slow"
  },
  {
    "startMs": 267280,
    "text": "like the endto-end model inference"
  },
  {
    "startMs": 268720,
    "text": "slower but uh agents are not humans and"
  },
  {
    "startMs": 271759,
    "text": "they will do plenty of things to make it"
  },
  {
    "startMs": 273199,
    "text": "slower like they'll disable CUDA graphs"
  },
  {
    "startMs": 274560,
    "text": "which can make it 20 times slower and"
  },
  {
    "startMs": 276320,
    "text": "they might make that one kernel faster"
  },
  {
    "startMs": 277840,
    "text": "but make the whole like it's not a"
  },
  {
    "startMs": 279680,
    "text": "viable kernel because it's they're"
  },
  {
    "startMs": 280960,
    "text": "disabling a bunch of speed ups like CUDA"
  },
  {
    "startMs": 282560,
    "text": "graphs or only testing on small context"
  },
  {
    "startMs": 284639,
    "text": "windows and so a lot of this is also"
  },
  {
    "startMs": 286400,
    "text": "just defining what not to do which is"
  },
  {
    "startMs": 288080,
    "text": "actually very important when you're"
  },
  {
    "startMs": 289120,
    "text": "doing frontier work that agents can't"
  },
  {
    "startMs": 290960,
    "text": "actually easily with a one shot."
  },
  {
    "startMs": 296479,
    "text": "Uh, another reward hack is that some"
  },
  {
    "startMs": 298320,
    "text": "models just don't actually write the"
  },
  {
    "startMs": 299919,
    "text": "cute DSL you need uh when you're trying"
  },
  {
    "startMs": 301759,
    "text": "to write kernels. And this is a common"
  },
  {
    "startMs": 303600,
    "text": "problem with enthropic models. And so,"
  },
  {
    "startMs": 306479,
    "text": "yeah, I mean, Anthropic says what they"
  },
  {
    "startMs": 308080,
    "text": "say about uh nerfing models. You can"
  },
  {
    "startMs": 310160,
    "text": "it's guess if it's I'm guessing if it's"
  },
  {
    "startMs": 311520,
    "text": "nerfing or not, but I would recommend"
  },
  {
    "startMs": 313280,
    "text": "using a different model. Uh, and it"
  },
  {
    "startMs": 315520,
    "text": "won't always be faster everywhere"
  },
  {
    "startMs": 316880,
    "text": "actually. So sometimes the kernels you"
  },
  {
    "startMs": 318479,
    "text": "come up with might only work well on"
  },
  {
    "startMs": 320400,
    "text": "like zero to 100k and then you need to"
  },
  {
    "startMs": 322560,
    "text": "go back to this the default kernel that"
  },
  {
    "startMs": 324160,
    "text": "you could you get from like a flash"
  },
  {
    "startMs": 325360,
    "text": "infer cutless. Um and so and that's"
  },
  {
    "startMs": 328479,
    "text": "another thing to look out for is that"
  },
  {
    "startMs": 329840,
    "text": "your kernel isn't always just a swap in"
  },
  {
    "startMs": 331600,
    "text": "for all all workloads."
  },
  {
    "startMs": 334400,
    "text": "Uh but one of the great things is is"
  },
  {
    "startMs": 335759,
    "text": "that kernels compound. So like if you"
  },
  {
    "startMs": 337199,
    "text": "make one for your sparse MLA for"
  },
  {
    "startMs": 338639,
    "text": "deepseek for example um you can get"
  },
  {
    "startMs": 340880,
    "text": "speed ups there and you just stack them"
  },
  {
    "startMs": 342160,
    "text": "on like that then plus NVFP4 fore uh you"
  },
  {
    "startMs": 346400,
    "text": "could do for us if we if you don't have"
  },
  {
    "startMs": 348000,
    "text": "NVL link you just keep stacking and"
  },
  {
    "startMs": 349520,
    "text": "stacking and stacking and then"
  },
  {
    "startMs": 350960,
    "text": "eventually you taper off at whatever the"
  },
  {
    "startMs": 352479,
    "text": "hardware limit is uh for your GPU and"
  },
  {
    "startMs": 354639,
    "text": "that's uh some people call this like MFU"
  },
  {
    "startMs": 357199,
    "text": "which is like the actual theoretical max"
  },
  {
    "startMs": 359280,
    "text": "utilization from a GPU."
  },
  {
    "startMs": 361919,
    "text": "Uh and so to go even farther, if you"
  },
  {
    "startMs": 363759,
    "text": "have actually have bare metal access,"
  },
  {
    "startMs": 365199,
    "text": "your auto research framework can uh do"
  },
  {
    "startMs": 367520,
    "text": "very hacky things. So hackers that have"
  },
  {
    "startMs": 368960,
    "text": "hacked with GPUs are probably going to"
  },
  {
    "startMs": 370160,
    "text": "like this. You can uh tweak your BIOS"
  },
  {
    "startMs": 372080,
    "text": "settings, you can overclock the GPU, uh"
  },
  {
    "startMs": 374319,
    "text": "you can force like PCIe relaxing, all"
  },
  {
    "startMs": 376960,
    "text": "these little tweaks of like uh old"
  },
  {
    "startMs": 379440,
    "text": "school hackers used to do, but this can"
  },
  {
    "startMs": 381120,
    "text": "actually help with inference as well."
  },
  {
    "startMs": 382240,
    "text": "And so net on bare metal optimizations,"
  },
  {
    "startMs": 385199,
    "text": "you can get roughly 25% over like a"
  },
  {
    "startMs": 387199,
    "text": "virtualized setup you get from using a"
  },
  {
    "startMs": 388960,
    "text": "cloud provider."
  },
  {
    "startMs": 391520,
    "text": "Uh so once you've done that, you can"
  },
  {
    "startMs": 393120,
    "text": "combine all of the kernels you did as"
  },
  {
    "startMs": 394400,
    "text": "well as all of the hardware level hacks"
  },
  {
    "startMs": 395759,
    "text": "you did. Uh you can get a 3x speed up."
  },
  {
    "startMs": 398000,
    "text": "And so I know this this might all sound"
  },
  {
    "startMs": 399759,
    "text": "like roses and flowers, but it's not"
  },
  {
    "startMs": 401280,
    "text": "actually the case. Around 80% of the"
  },
  {
    "startMs": 402880,
    "text": "things that auto research is going to do"
  },
  {
    "startMs": 404160,
    "text": "are going to be bad. Uh so it's"
  },
  {
    "startMs": 405759,
    "text": "important to remember while you're u"
  },
  {
    "startMs": 408160,
    "text": "like working on this that most things"
  },
  {
    "startMs": 409520,
    "text": "are going to be bad. It's going to try"
  },
  {
    "startMs": 410639,
    "text": "to trick you all the time. Uh but at the"
  },
  {
    "startMs": 412960,
    "text": "end you can actually get really good"
  },
  {
    "startMs": 414400,
    "text": "results from this."
  },
  {
    "startMs": 416880,
    "text": "with TLDDR. Uh have better ideas than"
  },
  {
    "startMs": 419759,
    "text": "use auto research. Super simple. Simple,"
  },
  {
    "startMs": 421840,
    "text": "right?"
  },
  {
    "startMs": 423680,
    "text": "Uh so it turns out you can actually get"
  },
  {
    "startMs": 425120,
    "text": "paid to do this. Uh if you think this is"
  },
  {
    "startMs": 426880,
    "text": "cool, consider joining us and you can"
  },
  {
    "startMs": 428880,
    "text": "email me here."
  },
  {
    "startMs": 431039,
    "text": "Thanks, guys."
  }
]