AI Engineer World's Fair 2025
What every AI engineer needs to know about GPUs
Read the talk
What every AI engineer needs to know about GPUs
Using GPUs well means understanding why they favor throughput, how decoding wastes arithmetic capacity, and when generating more candidates can make better use of the hardware.
From a talk by Charles Frye
Before you start: Familiarity with model inference, basic Python and matrix multiplication will help; no CUDA programming experience is required.
What should you know beneath the API?
If you build an application on the OpenAI, Anthropic or DeepSeek API, how much should you need to know about the hardware behind it? The API boundary makes that application possible: a complex system becomes unmanageable if every developer must understand every component in detail. In the Rise of the AI Engineer diagram that Charles Frye opens with, AI engineers occupy the product side of that boundary, working backward from user needs rather than forward from research or infrastructure capabilities.
Databases offer a useful precedent. Few developers build a database, and many never operate one themselves; they use a managed service such as Amazon RDS. Yet they still need to write good queries. Use The Index, Luke teaches the underlying machinery insofar as it helps developers use indexes effectively. Frye invokes B-trees, log-structured merge trees, and primary and secondary indexes as examples of the knowledge beneath that abstraction. The objective is practical literacy, not the ability to implement a database from scratch.
Language models are reaching a similar point. As engineers gain more opportunities to run and integrate models themselves, the equivalent advice is “use the tensor cores, Luke.” Know enough about the hardware to direct work toward its strongest execution units. For NVIDIA GPUs, the central operation is matrix-matrix multiplication on tensor cores; other accelerators have corresponding specialized machinery.
Improving open weights and serving software, including NVIDIA Dynamo, make self-hosting more attractive. Whether it is economical remains a workload question. Frye reserves comparisons of vLLM, SGLang and TensorRT-LLM across roughly ten to twelve models and workloads for a separate talk; this recording explains the hardware principles behind those decisions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The GPU’s priorities
The hardware discussion follows three nested priorities:
- Bandwidth over latency. GPUs, like TPUs, are designed to complete large amounts of work, rather than minimize the time of every individual operation.
- Arithmetic throughput over memory throughput. Moving data is necessary, but calculating with that data is where the largest capacity lies.
- Low-precision matrix-matrix multiplication. Even within arithmetic, the choice of operation matters. Matrix-vector multiplication does not expose the same opportunity as multiplying two substantial matrices.
Each step narrows what it means to use the GPU you paid for. Merely moving a program onto a GPU does not guarantee that its workload matches these priorities.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When faster clocks stopped doing the work
Frye frames the shift provocatively: latency scaling died during the Bush administration. To explain what changed, he reaches back to the Zuse 1, a mechanical digital computer built in Germany in the 1930s. Its logic used actuator plates. In the AND-gate example, two input plates must both be in position for a moving plate to push the output forward. A literal clock drives the movement.
Each clock tick advances the computation. Electronic computers changed the physics, but retained the useful abstraction of clocked operations. Increasing the clock frequency therefore offered an unusually convenient performance improvement: existing programs could run faster without being rewritten or recompiled. That was a major source of gains in the 1990s.
When those easy clock-speed gains slowed in the early 2000s, software increasingly had to expose more work at once. The consequences still show up in efforts such as GIL-free Python, multiprocessing and async coroutines. Performance improvement became something programs had to participate in, rather than simply inherit from a faster clock.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep more work in flight
Two strategies can increase completed work without increasing clock speed:
- Parallelism: perform two operations during a clock cycle instead of one.
- Concurrency: when an operation takes five cycles to finish, start other work during the intervening cycles instead of waiting idle.
At the programming level, concurrency can bring the complexity of async/await or Rust’s Pin. Its purpose is to keep pipelines busy. GPUs apply both strategies throughout the hardware and software stack, pushing them further than CPUs.
Frye illustrates the scale difference with an AMD EPYC CPU and an NVIDIA H100 SXM GPU:
| Comparison in the talk | AMD EPYC | NVIDIA H100 SXM |
|---|---|---|
| Parallel execution illustration | Two threads per core | Over 16,000 parallel threads |
| Power divided by thread count | About 1 W per thread | About 0.05 W per thread |
These are architectural illustrations, not measured application efficiency: the EPYC model and workload are unspecified. There is also a distinction between arithmetic cores and threads. NVIDIA’s Hopper architecture description identifies the corresponding H100 figure as an FP32 core count; it does not mean every application thread advances on every cycle.
Concurrency makes the comparison more interesting. A CPU can support a very large number of software threads, while Frye describes the H100 as holding roughly 250,000 concurrent threads. But the useful question is how quickly the machine can switch to ready work. Frye contrasts an illustrative CPU context switch of roughly 1,000 cycles, or a microsecond, with GPU scheduling on each clock cycle. The GPU’s warp scheduler selects ready groups of threads in hardware, avoiding an operating-system context switch each time another group can make progress. This is how concurrency hides waiting rather than merely accumulating threads.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Why bandwidth keeps winning
The larger pattern is David Patterson’s Latency Lags Bandwidth, which Frye calls Patterson’s Law. Across technologies such as networks, memory and disks, bandwidth has historically improved much faster than latency. Frye summarizes the historical trend as 10× improvement in latency accompanying 100× improvement in bandwidth. Patterson’s formulation is a rule of thumb—bandwidth improvement of at least the square of latency improvement—not an exact law governing every device.
Latency encounters physical limits. Bandwidth can often grow by replicating existing machinery and doing more things simultaneously. Frye uses packets traveling at approximately 70% of the speed of light to illustrate why propagation cannot become ten times faster. Spreading work across more channels is a more available path than discovering new physics. That history motivates his preference for bandwidth-oriented hardware, while leaving room for a debate with advocates of LPUs and Etched’s specialized approach.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reuse data before loading it again
Even with advanced Hynix high-bandwidth memory, a GPU’s capacity to calculate outstrips its capacity to move data. Arithmetic intensity describes the amount of computation performed per unit of data movement. An algorithm with N² operations might normally sound undesirable, but N² operations for only N memory loads can be a good fit for this hardware: each loaded value supports more useful work.
The relevant contrast in the hardware specification table is between enormous arithmetic throughput and the smaller rate at which memory supplies bytes. Frye says this compute-to-memory imbalance grows further with Blackwell. His prompt-processing illustration uses an eight-billion-parameter FP8 model: about 8 GB of weights move from GPU memory toward the compute units, with roughly 60 billion floating-point operations in his stated example. The sequence-length conditions are unspecified, so that operation count is not a general formula for processing an arbitrary prompt.
During decoding, the model needs those weights again to produce the next token. Keeping weights in GPU memory is not the same as keeping them at the point where arithmetic happens; they still have to move through the memory hierarchy. This separation of storage and computation is the von Neumann constraint Frye invokes. Prompt processing can apply weights across many token positions, while single-sequence decoding offers much less work per weight load. The result is a phase distinction that matters even though the model and GPU have not changed.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Small models can do more attempts
One response to underused arithmetic capacity is to do more useful work during decoding. Frye proposes taking an eight-billion-parameter model and generating many candidates for the same prompt. He first imagines 1,000 runs, then uses 10,000 outcomes as a larger illustration. The shared mechanism is batching candidates so a weight load serves many calculations, rather than serving only one sequence’s next token. This reuse happens across candidates at a decoding step; it does not eliminate weight movement throughout an entire generation.
More candidates improve the application only if it can identify a good one. For code, executable Python tests can provide a verifier: generate alternatives, test them, and select a passing candidate. This changes the quality question from how often one response succeeds to how often the candidate set contains a success that the verifier can recognize.
Frye reports that Llama 3.1 8B matches GPT-4o with about 100 generations in the presented Python-test verification setting. The comparison is HumanEval pass@100 for the small model against a separately reported GPT-4o zero-shot pass@1 baseline, not equivalence between their individual responses. The displayed chart expresses the same distinction through a declining Llama fail@k curve and a horizontal GPT-4o fail@1 line. Alongside it is Large Language Monkeys, the research motivating the experiment. Frye describes reproducing the core result in a day of coding with a different model and dataset.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn vectors into matrices
The final step is to match the arithmetic itself to the GPU. Frye suggests that multi-token prediction and multi-sample queries can become approximately free when they occupy capacity the original workload leaves unused. He notes that Kyle Kranen on the Dynamo team reached a similar conclusion and points to Kranen’s infrastructure talk for detailed charts. The condition behind the claim is spare capacity, not an absence of additional work.
The largest low-precision throughput figures belong to tensor cores, rather than ordinary CUDA-core arithmetic. Tensor cores specialize in matrix multiply-accumulate operations. Frye focuses on floating-point multiplication; H100 also supports integer tensor operations such as INT8. The practical constraint is the shape of computation, and other tasks can sometimes be reformulated to fit it—he offers a Fourier transform as an example.
A single sequence’s generation step naturally contains matrix-vector work: apply a weight matrix to one activation vector. Multiple candidates provide multiple activation vectors. Stack those vectors as columns, and the same weights can participate in a matrix-matrix multiplication. This Python example makes the shape change explicit, using a shared weight matrix and four candidate columns:
python
import torch
weights = torch.randn(128, 128, device="cuda", dtype=torch.float16)
candidates = torch.randn(128, 4, device="cuda", dtype=torch.float16)
# One candidate: matrix-vector multiplication.
one_output = weights @ candidates[:, 0]
# Four candidates sharing weights: matrix-matrix multiplication.
batch_output = weights @ candidates
print(one_output.shape) # torch.Size([128])
print(batch_output.shape) # torch.Size([128, 4])
The first output corresponds mathematically to the first column of the batched output. The change exposes more work together; actual tensor-core utilization still depends on dimensions and the selected kernel.
Frye describes a microbenchmark, tentatively attributed to ThunderKittens and Hazy Research, in which populating only one column of a wider matrix yields roughly 1/N of the useful performance. Filling more columns increases useful throughput. That is the intuition behind the slide’s description of matrix-vector to matrix-matrix upgrades as “free”: additional work can use arithmetic slots that were already being underused, rather than increasing time in direct proportion to the number of candidates.
Both multiple samples and multiple predicted tokens can move computation in this direction. Frye cites DeepSeek’s multi-token prediction; the DeepSeek-V3 Technical Report establishes that training objective. He also tentatively mentions Llama 4, without establishing its use of the technique. These approaches create opportunities for fuller matrices, but their end-to-end benefit depends on how the serving system generates, verifies and accepts the additional work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A smaller model, used more fully
These hardware properties open a practical deployment option: choose a smaller model that fits on a GPU under your desk, then scale candidate generation until the system meets users’ quality requirements. Where a reliable verifier exists, model size need not carry the entire burden of quality. Frye suggests revisiting research from around ChatGPT’s release, because improving open models make those inference-time strategies worth considering again.
For the terminology underneath those decisions, Frye offers the Modal GPU Glossary. Its purpose is to connect the hardware and software stack: a reader encountering a warp scheduler can follow the relationships to streaming multiprocessors and the NVIDIA CUDA compiler driver, rather than learning each term in isolation.
Running the hardware need not mean maintaining a machine locally. Frye closes with Modal, his employer’s serverless GPU platform. He describes its rewritten container stack as a foundation for serverless Python handling data-intensive and compute-intensive workloads, including language-model inference. The deployment abstraction remains useful; the hardware knowledge helps determine what work to place behind it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Connected explanations of GPU hardware, CUDA software and performance terminology.
A developer's guide to writing SQL queries that use indexes effectively.
Patterson's October 2004 analysis of differing bandwidth and latency trends across computing technologies.
An open-source framework for distributed inference serving.
Further reading
Frye and Halim's HumanEval experiment using repeated Llama 3.1 8B generations and executable tests, with reproduction code linked.
Research on increasing inference compute through repeated sampling and evaluating candidate solutions.
NVIDIA's explanation of H100 streaming multiprocessors, tensor cores and memory architecture, including preliminary launch specifications.
Hazy Research explains Blackwell kernel design and how matrix dimensions affect tensor-core utilization.
The model report describing DeepSeek-V3's architecture and multi-token prediction training objective.
Read the complete timestamped transcript
- 0:00
[upbeat music] So, um, what I wanted to talk about today was, uh, what every AI engineer needs to know about GPUs.
- 0:25
The... Like, so far in the last couple of years, um, most of the things that people have built as AI applications, people who are AI engineers, they've been building on top of model APIs.
- 0:35
So they use the OpenAI API, the Anthropic API, the DeepSeek API, and they build an application on top of that. And that goes back to kind of like the initial diagram that Swix put out, the like AI...
- 0:47
like rise of the AI engineer thing. Um, and yeah, probably just mirror would be great. Um, and the, um, so that like having that API boundary is like, is like pretty important, right?
- 1:00
It's like you can't really build a complex system if everybody has to know how every piece works, and everybody has to know all of it in detail, and there's no like boundaries or breakdowns.
- 1:09
Like you're just, yeah. You'll compla- collapse in complexity if you do that. Um, so, um, [guitar strum]
- 1:19
oh, that was me, but I'm down with it. Um, so like, so yeah, sorry. It started off by trying to answer the question of like why every AI engineer needs to know about GPUs.
- 1:30
Um, and so yeah, so here's our famous diagram, AI engineer on the right of the API boundary, where they're like constrained by the, um, the needs of users rather than like the, like what's possible with research or what infrastructure is capable of providing.
- 1:46
Um, and the way that I think about this distinction is that, um, it's kind of similar to the way that very few developers need to actually like write a database.
- 1:58
Um, like almost no one writes a database except in their like, you know, like undergrad classes. And then even very few developers like run a database. A lot of them will use either a fully managed service or, um, just like a hosted service like, uh, RDS on, on Amazon.
- 2:15
Um, but like almost all developers, despite the fact that they aren't like database engineers, they are users of databases, and they like need to like write... know how to like write good queries.
- 2:27
They need to know how to like hold the tool in order to press the like buttons on the side. Uh, so there's a famous educational resource that I really love, um, about databases called Use the Index, Luke, um, that's like basically about how to write SQL queries and not like suck.
- 2:44
Um, and the whole point is like there is a thing called an index. There's a couple of data structures that support it. It talks about things like B-trees and log structured merge trees and stuff.
- 2:56
And the intent of it isn't that you can then leave and go and like invert a binary tree on a whiteboard so you can get a FAANG job. Like the point of it is to teach you what you need to know so that you can write, like, write queries properly, that use the index and don't not use
- 3:08
the index. Primary and secondary indices, all these things. So you don't like, you know, that's like a little bit, um, like an easier like prospect, knowing it well enough to be able to use it rather than like build it or innovate on it.
- 3:21
Um, and I think we're reaching this point now with, uh, with language models where, um, where
- 3:28
you'll have more ability to like integrate, tightly run your own language models, and so more need to like use the index. Or I guess if you want like the, like one sentence summary of this talk, um, it's, uh, use the tensor cores, Luke.
- 3:41
Um, so in building your... There's one, there's basically one part of an NVIDIA GPU, um, and an equivalent in other GPUs that is, uh, fast and good and gets better, and it's the tensor core, and it does matrix-matrix multiplication.
- 3:56
And, uh, you should make sure you're using it and, uh, and not, not using it, just like an index on a database. [clears throat]
- 4:07
Um, so yeah, so open-- Like I kind of made this point earlier, but open weights models and the open source software to run them, like Dynamo, getting better very quickly, so it finally makes sense to self-host.
- 4:16
I'm not gonna belabor this point because I'm giving another talk, twelve forty-five, presenting some like benchmarking results that we did on like running, uh, vLLM, SGLang, TensorRT-LLM on like, uh, you know, ten, twelve different models on ten, twelve different workloads, um, to show like what's, what's economical, what's not.
- 4:34
Okay. So, uh, so that's the why. Um, sort of a slight change or adjustment in what AI engineers, I think AI engineers should focus on, know about. Um, so now what is it that you need to know about engineer-- uh, about, uh, these, these, this hardware in detail?
- 4:51
So the primary thing is that GPUs embrace high bandwidth, not low latency. That's the like key feature of this hardware. Similar with TPUs, but distinguishes it from pretty much every other piece of hardware that you're used to programming.
- 5:06
Um, and then in detail, they optimize for math bandwidth over memory bandwidth. So they do like computing on things. That's what they, where they have the highest throughput. So you, you want to align yourself not to latency, but to throughput, and within throughput, you wanna focus on computational operations.
- 5:24
And then within computational operations, what you wanna focus on if you want to like actually use the whole GPU you paid for, it's low precision matrix-matrix multiplications. Sorry, that wasn't a stutter.
- 5:37
That was matrix-matrix multiplications, not just matrix vector.
- 5:42
Um, okay, so for the first point about latency versus bandwidth. Um, so I, like, regret to inform you that the scaling of latency and the reduction of latency in computing systems died during the Bush administration.
- 5:54
It's not coming back. Um, see a talk later today for an alternative perspective, but, um, yeah, GPUs embrace bandwidth scaling. Um, so a little more detail on that. Um, so this is a computer or a piece of a computer, in case you haven't looked inside one in a while.
- 6:11
Um, so this is a logic gate from the Zuse 1, um, computer built in Germany in the '30s, kind of first digital computer. Digital, but not electronic. It's mechanical.
- 6:21
So there are all these actuator plates in it that, that, um, implemented logical operations. So what you see there on the left is a logical operation a- and. So if two plates are pushed down, then if both of them are present, then when it's, then when the other plate pushes forward, it will push the final plate forward.
- 6:38
That's the logical operation and. And the thing that pushes that, like, the, the thing that pushes forward is driven by a clock, like a literal clock. Um, I guess now everybody has Apple Watches, but you know, there was a time when you would have a physical clock for that sort of thing.
- 6:53
So this, um, uh, the clock, uh, like dro- like, drives these systems and causes them to cal- like, compute their logical operations, right? So every time the clock ticks, you get a new operation.
- 7:06
And so you can just, y- you know, so we, we, we've changed computers a little bit in that we use different physics to drive them, but it's still the same basic, like, abstract system.
- 7:15
There's a sort of a motive force that happens on a clock cycle that leads to calculations. Um, and the cool thing about that is that if you just make that faster, literally nobody has to f- like, like, think about anything and the computer gets better.
- 7:30
So this was the, like, primary driver of computers getting better in the '90s. No recompiling, no rewriting your software. Everything just got better, 'cause now the clock started going, like, twice as fast, right?
- 7:44
And time is very virtual in, in computers and so, like, the program couldn't possibly know the difference. Um, so that was really great during that, like, that, like, mid to late '90s period and then that, like, fell off a cliff in the early 2000s.
- 7:56
And this has, like, impacted a lot of computing over the last two decades, but actually its effects are, like, still being felt, like, all this switch from being able to kind of avoid the, like, uh, needing to think about performance.
- 8:11
So this is, like, kinda slowly and inevitably changing pretty much, like, everything in software. Um, all kinds of things you've seen around concurrency, uh, GIL-free Python, multi-processing, async coroutines.
- 8:22
Um, so there's, like, couple, like, kinda detailed things to dive in here. I wanna make sure that I give enough time to talk about the GPU stuff. But there's kinda two notions of how to make things faster without doing that.
- 8:34
One is parallel, so, like, when you have a clock cycle, just do two things instead of one. Sounds like a good idea. Um, the other one is concurrent, which is a little bit trickier, but it's like, so you start doing something, clock cycle hits, you start running a calculation.
- 8:47
Maybe that calculation takes five clock cycles to finish. Like, instead of waiting for those clock cycles to finish, try and do five other things with the next couple clock cycles.
- 8:56
Makes your programs really ugly 'cause you write, have to write async await everywhere, um, and yeah, uh, if you're writing Rust, it's, uh, it's a world of pin. Um, but yeah, but it helps you keep these, like, these super high bandwidth pipelines busy.
- 9:11
Um, and so these, like, concurrent and parallel, these are two strategies to maximize bandwidth that are adopted, like, at the hardware level all the way up to the programming level with GPUs to take this, like, f- bandwidth further than CPUs can.
- 9:26
So, uh, GPUs take parallelism further than CPUs, so I'm comparing an AMD EPYC CPU and an NVIDIA H100 SXM GPU here. Uh, the figure of merit here is the number of parallel threads that can operate and the wattage at which they operate.
- 9:42
So an H1... like a, like an AMD EPYC, uh, CPU can do two threads per core at about one watt per thread. Um, that's not bad, but an H100 can do over 16,000 parallel threads at five centawatts per thread, which is pretty, um, pretty amazing.
- 10:00
Uh, very big difference. And, uh, parallel means, like, literally every clock cycle, all 16,000 threads of execution make progress at the exact same time. So what about concurrency? So it may look like CPUs have an advantage here 'cause effectively concurrent threads are unbounded.
- 10:16
Like, you can just make a thread in Linux, like, it's free. Government doesn't want you to know this. Um, uh, and but there's a limit on H100, so it looks like, oh, wow, oh, only 250,000 threads?
- 10:26
What am I supposed to do with that? Um, but the difference here is context switching speed. How quickly can you go from executing one thing to another? So if it, like, if our purpose was to take advantage of every clock cycle, and it takes us 1,000 clock cycles, like a microsecond, to context switch, then our concurrency is,
- 10:43
like, actually, like, pretty tightly bounded, um, 'cause we can't do, uh, a thing for a whole thousand clock cycles. But in GPUs, context switching happens literally every clock cycle.
- 10:53
It's down there at the warp scheduler inside the hardware. Um, if you have to think about it that hard, you're probably having a bad time, but if you, uh, but normally it's just making everything run faster.
- 11:04
Um, so there's not really a name for this, uh, the phenomenon that's, that's driving all of this work, um, but David Patterson, who came up with RISC machines, um, and worked on TPUs, uh, wrote it down, so I call it Patterson's Law, latency lags bandwidth.
- 11:18
Um, so, like, why are, why are we doing all these things the, to, like, rewriting our programs, rethinking them in order to, to, like, take advantage of increasing bandwidth and, you know, bandwidth is replacing latency scaling?
- 11:30
It's 'cause if you look across a variety of different subsystems of computers, networks, uh, memory, disks, the latency improvement is actually the square of... or sorry, the, um, the bandwidth improvement is the square of the latency improvement over time.
- 11:45
This is one of those, like, Moore's law style charts, where you're looking at, like, trends in performance over time, and it's like for every 10X that we improve latency, we get 100X improvement in bandwidth.
- 11:57
And there's some arguments in the article about where, what it, you know, where this comes from. Basically, with latency, you run into the laws of physics. With bandwidth, you just run into, like, how many things can you do at the same time?
- 12:08
And you can always, you can take the same physics and spread it out more easily than you can, like, come up with new physics to take advantage of. Like, you, you cannot bribe the laws of physics, Scotty, in Star Trek, um, and that's, like, one of the limits on, like, network latency is, like, we, uh, we send
- 12:23
packets at, like, 70% of the speed of light, so, like, we can't get them 10X faster. Um, yeah. Um, all right, so that's, that's bandwidth. Uh, GPUs embrace bandwidth.
- 12:34
Maybe big takeaway from Patterson's law is, like, uh, bandwidth has won out over and over again, so maybe bet on the bandwidth hardware. I don't know if the person who's gonna be talking about, uh, LPUs or, or Etched is here, but we should fight about this later.
- 12:48
Um, yeah, so all right. So what kind of bandwidth, though? Um, arithmetic bandwidth over memory bandwidth, so not moving bytes around. That, that they are high, they have high bandwidth memory, the fanciest, finest Hynix f- high bandwidth memory.
- 13:03
Um, but they, uh, the thing where they really excel is doing calculations on that memory. And so the takeaway here is that N squared algorithms are usually bad, but if it's N squared operations for N memory loads, the...
- 13:20
It actually works out pretty nicely. It's almost like maybe Bill Dally and others were thinking of this when they built the chip. I don't know. Um, so, like, arithmetic intensity is the term for this or, yeah, math intensity, um [audio cuts out].
- 13:34
And if you look here at the things highlighted in purple [audio cuts out]
- 13:38
second [audio cuts out] nominated in Terra [audio cuts out] that go up into the thousands. Memory bandwidth at the bottom is [audio cuts out]
- 13:50
. And that has not changed with Blackwell, it's only gotten worse, um, or better. I don't know. Um, the, the ratio's gone up. Uh, so LLM inference works pretty nicely during prompt processing, where you're moving...
- 14:03
You move eight gigabytes, then you... Like, eight billion parameter model, FP8 quantization, you're gonna move eight gigabytes from the memory into the registers for calculation. You're gonna do about 60 billion floating point operations.
- 14:15
Um, so that's, that's, uh, you know, doesn't really scale too much with the sequence [audio cuts out] directly. Anyway, you're... The [audio cuts out] that when you then need to do [audio cuts out]
- 14:27
. Now you need to move those eight billion parameters again. Um, so this is from the, like, GPU's memory into the place where the compute happens. Like, you can't...
- 14:35
You have to... You know, it's von Neumann architecture. You can't, like, keep the things, um, it's... Compute on stuff that is, i- is in place. Um, so LLM inference works great during prompt processing, not so much during decoding.
- 14:48
Um, so one way to get around this is to just do more stuff when you're decoding. So one example is to take a small model, so eight billion parameters, and run it, like, 1,000 times on the same prompt.
- 14:59
Now you're loading the weights, and you only load the weights one time, but then you generate, like, 10,000 things. Um, and, uh, so there's, like, kind of an inherent advantage there to small models for being more sympathetic to the hardware.
- 15:11
You can actually match quality if you do things right, if you have a good verifier. Either, in this case, this is, does it pass a Python test? Um, that allows you to pick, uh, the one of your 10,000 outcomes.
- 15:23
And so you can use Llama 3.18B to match GPT-4.0 with, like, 100... Yeah, 100 generations. So that's a... The figure on the left is a reproduction. Um, I read a research paper, I sat down, spent a day coding, and I got the exact same result on different data in a different model. [audio cuts out]
- 15:40
any research for that [audio cuts out]. And so that's... So this is, like, this is, this is legit. This is real, this is real science, you know? Um, and it's, uh, so it's, it's a real phenomenon, and it fits with the hardware.
- 15:52
Um, so lastly, like, so we wanna do more, like... We wanna do, like, throughput-oriented, like, large scale activities. We wanna do it with, uh, like, computation and mathematics, not with memory movement.
- 16:06
And the specific thing we wanna do is low-precision matrix multiplication. Um, and the takeaway here is that some surprising things are gonna turn out to be approximately free. I don't have time to go into details on this, but it turns out Kyle Cranon, also on the Dynamo team, ha- like, came to the exact same conclusion.
- 16:23
We were talking, you know, comparing notes last night. So check his talk in the afternoon of the infrastructure track if you want n- less hand-waving and more charts. Um, so things like multi-token prediction, multi-sample query, all this stuff suddenly becomes, like, basically free.
- 16:38
Uh, and the reason why is that the latest GPUs, NVIDIAs and others, have this giant chunk in them, the tensor core, that's specialized for low-precision matrix-matrix multiplication. Uh, and so that's, you know, all these things in purple here that have the really big numbers are tensor core output, uh, not the CUDA core output.
- 16:58
Um, and tensor cores do exactly one thing, and it's floating point matrix multiplication. Um, bit of a tough world to live in as, like, a... If you're a theoretical programmer to, to discover that there's only one data type you're allowed to work with.
- 17:12
But you just get more creative, right? You can do a Fourier transform with this thing if you want. Um, yeah, so the generation phase of li- language models is very heavy on matrix vector operations, if you just, like, write it out at first.
- 17:24
Um, so the things that are basically free are things that can upgrade you to a matrix-matrix operation. There's some micro benchmarks from the, um, Thunder kitten's people, I think, Hazy Research, that was basically a tensor core.
- 17:36
Looks like it runs at, like, if you give it a matrix and then, like, a mostly empty matrix with one column full, you get, like, one over N of the performance, right?
- 17:46
Um, and so, you know, if you just add more stuff there, like, all of a sudden you, like, you're... The, like, performance is scaling to match. Um, so this is sort of...
- 17:55
Yeah, this is another phenomenon that pushes you in the direction of generating multiple samples, um, generating multiple tokens, as DeepSeek does the next token prediction, and I think the Llama 4 models do as well.
- 18:07
Um, so yeah. So these are the... So as an AI engineer, the things you should be looking at are, like, okay, like maybe I can get away with running a smaller model that fits on a GPU that's under my desk.
- 18:18
Um, uh, and then I just, like, scale it out in order to get the, like, sufficient quality to, to satisfy users. There was a bunch of research on this stuff back in, like, around the release of ChatGPT when there was still, like, a thriving academic field on top of language models.
- 18:34
Um, and it hasn't... Uh, like, people have kind of forgotten about it a bit, but I think the model, the open models are good enough that this is, uh, back to being a, a good idea.
- 18:43
Um, cool. I think I only have about 10 seconds left, so I'll just say if you wanna learn more, uh, I wrote this, uh, GPU glossary, modal.com/gpu-glossary. It's, uh, CUDA docs for humans.
- 18:55
Attempt to, like, explain this whole software and hardware stack in one place with lots of links, uh, so that when you're reading about a warp scheduler and you've forgotten what a streaming multiprocessor architecture is and how that's related to the NVIDIA CUDA compiler driver, it's, like, one click away to get all of those things.
- 19:12
Um, so if you want to run these, uh, uh, uh, on this hardware, um, no better place than the platform that I work on, uh, Modal, uh, serverless GPUs and more.
- 19:25
Um, we sort of, like, ri- ripped out and rewrote the whole container stack to make, um, like, serverless Python for data-intensive and compute-intensive workloads, like language model inference, work well.
- 19:37
Um, and so you should definitely check it out. Come find us at the expo hall, and we'll, uh, you know, talk your ear off about it. All right. Thank you very much. [outro music]