What every AI engineer needs to know about GPUs

Talk transcript

1. 0:00

   [upbeat music] So, um, what I wanted to talk about today was, uh, what every AI engineer needs to know about GPUs.

2. 0:25

   The... Like, so far in the last couple of years, um, most of the things that people have built as AI applications, people who are AI engineers, they've been building on top of model APIs.

3. 0:35

   So they use the OpenAI API, the Anthropic API, the DeepSeek API, and they build an application on top of that. And that goes back to kind of like the initial diagram that Swix put out, the like AI...

4. 0:47

   like rise of the AI engineer thing. Um, and yeah, probably just mirror would be great. Um, and the, um, so that like having that API boundary is like, is like pretty important, right?

5. 1:00

   It's like you can't really build a complex system if everybody has to know how every piece works, and everybody has to know all of it in detail, and there's no like boundaries or breakdowns.

6. 1:09

   Like you're just, yeah. You'll compla- collapse in complexity if you do that. Um, so, um, [guitar strum]

7. 1:19

   oh, that was me, but I'm down with it. Um, so like, so yeah, sorry. It started off by trying to answer the question of like why every AI engineer needs to know about GPUs.

8. 1:30

   Um, and so yeah, so here's our famous diagram, AI engineer on the right of the API boundary, where they're like constrained by the, um, the needs of users rather than like the, like what's possible with research or what infrastructure is capable of providing.

9. 1:46

   Um, and the way that I think about this distinction is that, um, it's kind of similar to the way that very few developers need to actually like write a database.

10. 1:58

    Um, like almost no one writes a database except in their like, you know, like undergrad classes. And then even very few developers like run a database. A lot of them will use either a fully managed service or, um, just like a hosted service like, uh, RDS on, on Amazon.

11. 2:15

    Um, but like almost all developers, despite the fact that they aren't like database engineers, they are users of databases, and they like need to like write... know how to like write good queries.

12. 2:27

    They need to know how to like hold the tool in order to press the like buttons on the side. Uh, so there's a famous educational resource that I really love, um, about databases called Use the Index, Luke, um, that's like basically about how to write SQL queries and not like suck.

13. 2:44

    Um, and the whole point is like there is a thing called an index. There's a couple of data structures that support it. It talks about things like B-trees and log structured merge trees and stuff.

14. 2:56

    And the intent of it isn't that you can then leave and go and like invert a binary tree on a whiteboard so you can get a FAANG job. Like the point of it is to teach you what you need to know so that you can write, like, write queries properly, that use the index and don't not use

15. 3:08

    the index. Primary and secondary indices, all these things. So you don't like, you know, that's like a little bit, um, like an easier like prospect, knowing it well enough to be able to use it rather than like build it or innovate on it.

16. 3:21

    Um, and I think we're reaching this point now with, uh, with language models where, um, where

17. 3:28

    you'll have more ability to like integrate, tightly run your own language models, and so more need to like use the index. Or I guess if you want like the, like one sentence summary of this talk, um, it's, uh, use the tensor cores, Luke.

18. 3:41

    Um, so in building your... There's one, there's basically one part of an NVIDIA GPU, um, and an equivalent in other GPUs that is, uh, fast and good and gets better, and it's the tensor core, and it does matrix-matrix multiplication.

19. 3:56

    And, uh, you should make sure you're using it and, uh, and not, not using it, just like an index on a database. [clears throat]

20. 4:07

    Um, so yeah, so open-- Like I kind of made this point earlier, but open weights models and the open source software to run them, like Dynamo, getting better very quickly, so it finally makes sense to self-host.

21. 4:16

    I'm not gonna belabor this point because I'm giving another talk, twelve forty-five, presenting some like benchmarking results that we did on like running, uh, vLLM, SGLang, TensorRT-LLM on like, uh, you know, ten, twelve different models on ten, twelve different workloads, um, to show like what's, what's economical, what's not.

22. 4:34

    Okay. So, uh, so that's the why. Um, sort of a slight change or adjustment in what AI engineers, I think AI engineers should focus on, know about. Um, so now what is it that you need to know about engineer-- uh, about, uh, these, these, this hardware in detail?

23. 4:51

    So the primary thing is that GPUs embrace high bandwidth, not low latency. That's the like key feature of this hardware. Similar with TPUs, but distinguishes it from pretty much every other piece of hardware that you're used to programming.

24. 5:06

    Um, and then in detail, they optimize for math bandwidth over memory bandwidth. So they do like computing on things. That's what they, where they have the highest throughput. So you, you want to align yourself not to latency, but to throughput, and within throughput, you wanna focus on computational operations.

25. 5:24

    And then within computational operations, what you wanna focus on if you want to like actually use the whole GPU you paid for, it's low precision matrix-matrix multiplications. Sorry, that wasn't a stutter.

26. 5:37

    That was matrix-matrix multiplications, not just matrix vector.

27. 5:42

    Um, okay, so for the first point about latency versus bandwidth. Um, so I, like, regret to inform you that the scaling of latency and the reduction of latency in computing systems died during the Bush administration.

28. 5:54

    It's not coming back. Um, see a talk later today for an alternative perspective, but, um, yeah, GPUs embrace bandwidth scaling. Um, so a little more detail on that. Um, so this is a computer or a piece of a computer, in case you haven't looked inside one in a while.

29. 6:11

    Um, so this is a logic gate from the Zuse 1, um, computer built in Germany in the '30s, kind of first digital computer. Digital, but not electronic. It's mechanical.

30. 6:21

    So there are all these actuator plates in it that, that, um, implemented logical operations. So what you see there on the left is a logical operation a- and. So if two plates are pushed down, then if both of them are present, then when it's, then when the other plate pushes forward, it will push the final plate forward.

31. 6:38

    That's the logical operation and. And the thing that pushes that, like, the, the thing that pushes forward is driven by a clock, like a literal clock. Um, I guess now everybody has Apple Watches, but you know, there was a time when you would have a physical clock for that sort of thing.

32. 6:53

    So this, um, uh, the clock, uh, like dro- like, drives these systems and causes them to cal- like, compute their logical operations, right? So every time the clock ticks, you get a new operation.

33. 7:06

    And so you can just, y- you know, so we, we, we've changed computers a little bit in that we use different physics to drive them, but it's still the same basic, like, abstract system.

34. 7:15

    There's a sort of a motive force that happens on a clock cycle that leads to calculations. Um, and the cool thing about that is that if you just make that faster, literally nobody has to f- like, like, think about anything and the computer gets better.

35. 7:30

    So this was the, like, primary driver of computers getting better in the '90s. No recompiling, no rewriting your software. Everything just got better, 'cause now the clock started going, like, twice as fast, right?

36. 7:44

    And time is very virtual in, in computers and so, like, the program couldn't possibly know the difference. Um, so that was really great during that, like, that, like, mid to late '90s period and then that, like, fell off a cliff in the early 2000s.

37. 7:56

    And this has, like, impacted a lot of computing over the last two decades, but actually its effects are, like, still being felt, like, all this switch from being able to kind of avoid the, like, uh, needing to think about performance.

38. 8:11

    So this is, like, kinda slowly and inevitably changing pretty much, like, everything in software. Um, all kinds of things you've seen around concurrency, uh, GIL-free Python, multi-processing, async coroutines.

39. 8:22

    Um, so there's, like, couple, like, kinda detailed things to dive in here. I wanna make sure that I give enough time to talk about the GPU stuff. But there's kinda two notions of how to make things faster without doing that.

40. 8:34

    One is parallel, so, like, when you have a clock cycle, just do two things instead of one. Sounds like a good idea. Um, the other one is concurrent, which is a little bit trickier, but it's like, so you start doing something, clock cycle hits, you start running a calculation.

41. 8:47

    Maybe that calculation takes five clock cycles to finish. Like, instead of waiting for those clock cycles to finish, try and do five other things with the next couple clock cycles.

42. 8:56

    Makes your programs really ugly 'cause you write, have to write async await everywhere, um, and yeah, uh, if you're writing Rust, it's, uh, it's a world of pin. Um, but yeah, but it helps you keep these, like, these super high bandwidth pipelines busy.

43. 9:11

    Um, and so these, like, concurrent and parallel, these are two strategies to maximize bandwidth that are adopted, like, at the hardware level all the way up to the programming level with GPUs to take this, like, f- bandwidth further than CPUs can.

44. 9:26

    So, uh, GPUs take parallelism further than CPUs, so I'm comparing an AMD EPYC CPU and an NVIDIA H100 SXM GPU here. Uh, the figure of merit here is the number of parallel threads that can operate and the wattage at which they operate.

45. 9:42

    So an H1... like a, like an AMD EPYC, uh, CPU can do two threads per core at about one watt per thread. Um, that's not bad, but an H100 can do over 16,000 parallel threads at five centawatts per thread, which is pretty, um, pretty amazing.

46. 10:00

    Uh, very big difference. And, uh, parallel means, like, literally every clock cycle, all 16,000 threads of execution make progress at the exact same time. So what about concurrency? So it may look like CPUs have an advantage here 'cause effectively concurrent threads are unbounded.

47. 10:16

    Like, you can just make a thread in Linux, like, it's free. Government doesn't want you to know this. Um, uh, and but there's a limit on H100, so it looks like, oh, wow, oh, only 250,000 threads?

48. 10:26

    What am I supposed to do with that? Um, but the difference here is context switching speed. How quickly can you go from executing one thing to another? So if it, like, if our purpose was to take advantage of every clock cycle, and it takes us 1,000 clock cycles, like a microsecond, to context switch, then our concurrency is,

49. 10:43

    like, actually, like, pretty tightly bounded, um, 'cause we can't do, uh, a thing for a whole thousand clock cycles. But in GPUs, context switching happens literally every clock cycle.

50. 10:53

    It's down there at the warp scheduler inside the hardware. Um, if you have to think about it that hard, you're probably having a bad time, but if you, uh, but normally it's just making everything run faster.

51. 11:04

    Um, so there's not really a name for this, uh, the phenomenon that's, that's driving all of this work, um, but David Patterson, who came up with RISC machines, um, and worked on TPUs, uh, wrote it down, so I call it Patterson's Law, latency lags bandwidth.

52. 11:18

    Um, so, like, why are, why are we doing all these things the, to, like, rewriting our programs, rethinking them in order to, to, like, take advantage of increasing bandwidth and, you know, bandwidth is replacing latency scaling?

53. 11:30

    It's 'cause if you look across a variety of different subsystems of computers, networks, uh, memory, disks, the latency improvement is actually the square of... or sorry, the, um, the bandwidth improvement is the square of the latency improvement over time.

54. 11:45

    This is one of those, like, Moore's law style charts, where you're looking at, like, trends in performance over time, and it's like for every 10X that we improve latency, we get 100X improvement in bandwidth.

55. 11:57

    And there's some arguments in the article about where, what it, you know, where this comes from. Basically, with latency, you run into the laws of physics. With bandwidth, you just run into, like, how many things can you do at the same time?

56. 12:08

    And you can always, you can take the same physics and spread it out more easily than you can, like, come up with new physics to take advantage of. Like, you, you cannot bribe the laws of physics, Scotty, in Star Trek, um, and that's, like, one of the limits on, like, network latency is, like, we, uh, we send

57. 12:23

    packets at, like, 70% of the speed of light, so, like, we can't get them 10X faster. Um, yeah. Um, all right, so that's, that's bandwidth. Uh, GPUs embrace bandwidth.

58. 12:34

    Maybe big takeaway from Patterson's law is, like, uh, bandwidth has won out over and over again, so maybe bet on the bandwidth hardware. I don't know if the person who's gonna be talking about, uh, LPUs or, or Etched is here, but we should fight about this later.

59. 12:48

    Um, yeah, so all right. So what kind of bandwidth, though? Um, arithmetic bandwidth over memory bandwidth, so not moving bytes around. That, that they are high, they have high bandwidth memory, the fanciest, finest Hynix f- high bandwidth memory.

60. 13:03

    Um, but they, uh, the thing where they really excel is doing calculations on that memory. And so the takeaway here is that N squared algorithms are usually bad, but if it's N squared operations for N memory loads, the...

61. 13:20

    It actually works out pretty nicely. It's almost like maybe Bill Dally and others were thinking of this when they built the chip. I don't know. Um, so, like, arithmetic intensity is the term for this or, yeah, math intensity, um [audio cuts out].

62. 13:34

    And if you look here at the things highlighted in purple [audio cuts out]

63. 13:38

    second [audio cuts out] nominated in Terra [audio cuts out] that go up into the thousands. Memory bandwidth at the bottom is [audio cuts out]

64. 13:50

    . And that has not changed with Blackwell, it's only gotten worse, um, or better. I don't know. Um, the, the ratio's gone up. Uh, so LLM inference works pretty nicely during prompt processing, where you're moving...

65. 14:03

    You move eight gigabytes, then you... Like, eight billion parameter model, FP8 quantization, you're gonna move eight gigabytes from the memory into the registers for calculation. You're gonna do about 60 billion floating point operations.

66. 14:15

    Um, so that's, that's, uh, you know, doesn't really scale too much with the sequence [audio cuts out] directly. Anyway, you're... The [audio cuts out] that when you then need to do [audio cuts out]

67. 14:27

    . Now you need to move those eight billion parameters again. Um, so this is from the, like, GPU's memory into the place where the compute happens. Like, you can't...

68. 14:35

    You have to... You know, it's von Neumann architecture. You can't, like, keep the things, um, it's... Compute on stuff that is, i- is in place. Um, so LLM inference works great during prompt processing, not so much during decoding.

69. 14:48

    Um, so one way to get around this is to just do more stuff when you're decoding. So one example is to take a small model, so eight billion parameters, and run it, like, 1,000 times on the same prompt.

70. 14:59

    Now you're loading the weights, and you only load the weights one time, but then you generate, like, 10,000 things. Um, and, uh, so there's, like, kind of an inherent advantage there to small models for being more sympathetic to the hardware.

71. 15:11

    You can actually match quality if you do things right, if you have a good verifier. Either, in this case, this is, does it pass a Python test? Um, that allows you to pick, uh, the one of your 10,000 outcomes.

72. 15:23

    And so you can use Llama 3.18B to match GPT-4.0 with, like, 100... Yeah, 100 generations. So that's a... The figure on the left is a reproduction. Um, I read a research paper, I sat down, spent a day coding, and I got the exact same result on different data in a different model. [audio cuts out]

73. 15:40

    any research for that [audio cuts out]. And so that's... So this is, like, this is, this is legit. This is real, this is real science, you know? Um, and it's, uh, so it's, it's a real phenomenon, and it fits with the hardware.

74. 15:52

    Um, so lastly, like, so we wanna do more, like... We wanna do, like, throughput-oriented, like, large scale activities. We wanna do it with, uh, like, computation and mathematics, not with memory movement.

75. 16:06

    And the specific thing we wanna do is low-precision matrix multiplication. Um, and the takeaway here is that some surprising things are gonna turn out to be approximately free. I don't have time to go into details on this, but it turns out Kyle Cranon, also on the Dynamo team, ha- like, came to the exact same conclusion.

76. 16:23

    We were talking, you know, comparing notes last night. So check his talk in the afternoon of the infrastructure track if you want n- less hand-waving and more charts. Um, so things like multi-token prediction, multi-sample query, all this stuff suddenly becomes, like, basically free.

77. 16:38

    Uh, and the reason why is that the latest GPUs, NVIDIAs and others, have this giant chunk in them, the tensor core, that's specialized for low-precision matrix-matrix multiplication. Uh, and so that's, you know, all these things in purple here that have the really big numbers are tensor core output, uh, not the CUDA core output.

78. 16:58

    Um, and tensor cores do exactly one thing, and it's floating point matrix multiplication. Um, bit of a tough world to live in as, like, a... If you're a theoretical programmer to, to discover that there's only one data type you're allowed to work with.

79. 17:12

    But you just get more creative, right? You can do a Fourier transform with this thing if you want. Um, yeah, so the generation phase of li- language models is very heavy on matrix vector operations, if you just, like, write it out at first.

80. 17:24

    Um, so the things that are basically free are things that can upgrade you to a matrix-matrix operation. There's some micro benchmarks from the, um, Thunder kitten's people, I think, Hazy Research, that was basically a tensor core.

81. 17:36

    Looks like it runs at, like, if you give it a matrix and then, like, a mostly empty matrix with one column full, you get, like, one over N of the performance, right?

82. 17:46

    Um, and so, you know, if you just add more stuff there, like, all of a sudden you, like, you're... The, like, performance is scaling to match. Um, so this is sort of...

83. 17:55

    Yeah, this is another phenomenon that pushes you in the direction of generating multiple samples, um, generating multiple tokens, as DeepSeek does the next token prediction, and I think the Llama 4 models do as well.

84. 18:07

    Um, so yeah. So these are the... So as an AI engineer, the things you should be looking at are, like, okay, like maybe I can get away with running a smaller model that fits on a GPU that's under my desk.

85. 18:18

    Um, uh, and then I just, like, scale it out in order to get the, like, sufficient quality to, to satisfy users. There was a bunch of research on this stuff back in, like, around the release of ChatGPT when there was still, like, a thriving academic field on top of language models.

86. 18:34

    Um, and it hasn't... Uh, like, people have kind of forgotten about it a bit, but I think the model, the open models are good enough that this is, uh, back to being a, a good idea.

87. 18:43

    Um, cool. I think I only have about 10 seconds left, so I'll just say if you wanna learn more, uh, I wrote this, uh, GPU glossary, modal.com/gpu-glossary. It's, uh, CUDA docs for humans.

88. 18:55

    Attempt to, like, explain this whole software and hardware stack in one place with lots of links, uh, so that when you're reading about a warp scheduler and you've forgotten what a streaming multiprocessor architecture is and how that's related to the NVIDIA CUDA compiler driver, it's, like, one click away to get all of those things.

89. 19:12

    Um, so if you want to run these, uh, uh, uh, on this hardware, um, no better place than the platform that I work on, uh, Modal, uh, serverless GPUs and more.

90. 19:25

    Um, we sort of, like, ri- ripped out and rewrote the whole container stack to make, um, like, serverless Python for data-intensive and compute-intensive workloads, like language model inference, work well.

91. 19:37

    Um, and so you should definitely check it out. Come find us at the expo hall, and we'll, uh, you know, talk your ear off about it. All right. Thank you very much. [outro music]