# What every AI engineer needs to know about GPUs

Charles Frye

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/y-UGrYbJsJk-what-every-ai-engineer-needs-know-about) · [Video](https://www.youtube.com/watch?v=y-UGrYbJsJk)

## What every AI engineer needs to know about GPUs

Using GPUs well means understanding why they favor throughput, how decoding wastes arithmetic capacity, and when generating more candidates can make better use of the hardware.

Familiarity with model inference, basic Python and matrix multiplication will help; no CUDA programming experience is required.

### What should you know beneath the API?

[0:25](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=25s)

If you build an application on the OpenAI, Anthropic or DeepSeek API, how much should you need to know about the hardware behind it? The API boundary makes that application possible: a complex system becomes unmanageable if every developer must understand every component in detail. In the Rise of the AI Engineer diagram that Charles Frye opens with, AI engineers occupy the product side of that boundary, working backward from user needs rather than forward from research or infrastructure capabilities.

![Diagram places research and machine learning roles left of an API boundary, and AI engineers and fullstack engineers on the product/user-constrained right.](https://ai.engineer/api/talk-essay-media/ea019465674097996a10dc4f7f9770fd737adf5f83a64ef1dceee2deb0b7e34e)

AI engineers sit on the product side of the LLM API boundary.

Databases offer a useful precedent. Few developers build a database, and many never operate one themselves; they use a managed service such as Amazon RDS. Yet they still need to write good queries. [Use The Index, Luke](https://use-the-index-luke.com/) teaches the underlying machinery insofar as it helps developers use indexes effectively. Frye invokes B-trees, log-structured merge trees, and primary and secondary indexes as examples of the knowledge beneath that abstraction. The objective is practical literacy, not the ability to implement a database from scratch.

Language models are reaching a similar point. As engineers gain more opportunities to run and integrate models themselves, the equivalent advice is “use the tensor cores, Luke.” **Know enough about the hardware to direct work toward its strongest execution units.** For NVIDIA GPUs, the central operation is matrix-matrix multiplication on tensor cores; other accelerators have corresponding specialized machinery.

Improving open weights and serving software, including [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo), make self-hosting more attractive. Whether it is economical remains a workload question. Frye reserves comparisons of vLLM, SGLang and TensorRT-LLM across roughly ten to twelve models and workloads for a separate talk; this recording explains the hardware principles behind those decisions.

### The GPU’s priorities

[4:34](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=274s)

The hardware discussion follows three nested priorities:

1. **Bandwidth over latency.** GPUs, like TPUs, are designed to complete large amounts of work, rather than minimize the time of every individual operation.
2. **Arithmetic throughput over memory throughput.** Moving data is necessary, but calculating with that data is where the largest capacity lies.
3. **Low-precision matrix-matrix multiplication.** Even within arithmetic, the choice of operation matters. Matrix-vector multiplication does not expose the same opportunity as multiplying two substantial matrices.

Each step narrows what it means to use the GPU you paid for. Merely moving a program onto a GPU does not guarantee that its workload matches these priorities.

### When faster clocks stopped doing the work

[5:42](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=342s)

Frye frames the shift provocatively: latency scaling died during the Bush administration. To explain what changed, he reaches back to the Zuse 1, a mechanical digital computer built in Germany in the 1930s. Its logic used actuator plates. In the AND-gate example, two input plates must both be in position for a moving plate to push the output forward. A literal clock drives the movement.

Each clock tick advances the computation. Electronic computers changed the physics, but retained the useful abstraction of clocked operations. Increasing the clock frequency therefore offered an unusually convenient performance improvement: existing programs could run faster without being rewritten or recompiled. That was a major source of gains in the 1990s.

When those easy clock-speed gains slowed in the early 2000s, software increasingly had to expose more work at once. The consequences still show up in efforts such as GIL-free Python, multiprocessing and async coroutines. Performance improvement became something programs had to participate in, rather than simply inherit from a faster clock.

### Keep more work in flight

[8:22](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=502s)

Two strategies can increase completed work without increasing clock speed:

- **Parallelism:** perform two operations during a clock cycle instead of one.
- **Concurrency:** when an operation takes five cycles to finish, start other work during the intervening cycles instead of waiting idle.

At the programming level, concurrency can bring the complexity of `async`/`await` or Rust’s `Pin`. Its purpose is to keep pipelines busy. GPUs apply both strategies throughout the hardware and software stack, pushing them further than CPUs.

Frye illustrates the scale difference with an AMD EPYC CPU and an NVIDIA H100 SXM GPU:

| Comparison in the talk | AMD EPYC | NVIDIA H100 SXM |
| --- | --- | --- |
| Parallel execution illustration | Two threads per core | Over 16,000 parallel threads |
| Power divided by thread count | About 1 W per thread | About 0.05 W per thread |

These are architectural illustrations, not measured application efficiency: the EPYC model and workload are unspecified. There is also a distinction between arithmetic cores and threads. [NVIDIA’s Hopper architecture description](https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/) identifies the corresponding H100 figure as an FP32 core count; it does not mean every application thread advances on every cycle.

Concurrency makes the comparison more interesting. A CPU can support a very large number of software threads, while Frye describes the H100 as holding roughly 250,000 concurrent threads. But the useful question is how quickly the machine can switch to ready work. Frye contrasts an illustrative CPU context switch of roughly 1,000 cycles, or a microsecond, with GPU scheduling on each clock cycle. The GPU’s **warp scheduler** selects ready groups of threads in hardware, avoiding an operating-system context switch each time another group can make progress. This is how concurrency hides waiting rather than merely accumulating threads.

### Why bandwidth keeps winning

[11:04](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=664s)

The larger pattern is David Patterson’s [Latency Lags Bandwidth](https://pages.cs.wisc.edu/~markhill/restricted/cacm2004_latency.pdf), which Frye calls Patterson’s Law. Across technologies such as networks, memory and disks, bandwidth has historically improved much faster than latency. Frye summarizes the historical trend as 10× improvement in latency accompanying 100× improvement in bandwidth. Patterson’s formulation is a rule of thumb—bandwidth improvement of at least the square of latency improvement—not an exact law governing every device.

Latency encounters physical limits. Bandwidth can often grow by replicating existing machinery and doing more things simultaneously. Frye uses packets traveling at approximately 70% of the speed of light to illustrate why propagation cannot become ten times faster. Spreading work across more channels is a more available path than discovering new physics. That history motivates his preference for bandwidth-oriented hardware, while leaving room for a debate with advocates of LPUs and Etched’s specialized approach.

### Reuse data before loading it again

[12:48](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=768s)

Even with advanced Hynix high-bandwidth memory, a GPU’s capacity to calculate outstrips its capacity to move data. **Arithmetic intensity** describes the amount of computation performed per unit of data movement. An algorithm with N² operations might normally sound undesirable, but N² operations for only N memory loads can be a good fit for this hardware: each loaded value supports more useful work.

The relevant contrast in the hardware specification table is between enormous arithmetic throughput and the smaller rate at which memory supplies bytes. Frye says this compute-to-memory imbalance grows further with Blackwell. His prompt-processing illustration uses an eight-billion-parameter FP8 model: about 8 GB of weights move from GPU memory toward the compute units, with roughly 60 billion floating-point operations in his stated example. The sequence-length conditions are unspecified, so that operation count is not a general formula for processing an arbitrary prompt.

During decoding, the model needs those weights again to produce the next token. Keeping weights in GPU memory is not the same as keeping them at the point where arithmetic happens; they still have to move through the memory hierarchy. This separation of storage and computation is the von Neumann constraint Frye invokes. Prompt processing can apply weights across many token positions, while single-sequence decoding offers much less work per weight load. The result is a phase distinction that matters even though the model and GPU have not changed.

### Small models can do more attempts

[14:48](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=888s)

One response to underused arithmetic capacity is to do more useful work during decoding. Frye proposes taking an eight-billion-parameter model and generating many candidates for the same prompt. He first imagines 1,000 runs, then uses 10,000 outcomes as a larger illustration. The shared mechanism is batching candidates so a weight load serves many calculations, rather than serving only one sequence’s next token. This reuse happens across candidates at a decoding step; it does not eliminate weight movement throughout an entire generation.

More candidates improve the application only if it can identify a good one. For code, executable Python tests can provide a verifier: generate alternatives, test them, and select a passing candidate. This changes the quality question from how often one response succeeds to how often the candidate set contains a success that the verifier can recognize.

Frye reports that **Llama 3.1 8B** matches **GPT-4o** with about 100 generations in the presented Python-test verification setting. The comparison is HumanEval `pass@100` for the small model against a separately reported GPT-4o zero-shot `pass@1` baseline, not equivalence between their individual responses. The displayed chart expresses the same distinction through a declining Llama `fail@k` curve and a horizontal GPT-4o `fail@1` line. Alongside it is [Large Language Monkeys](https://arxiv.org/abs/2407.21787), the research motivating the experiment. Frye describes reproducing the core result in a day of coding with a different model and dataset.

![Chart shows a descending green LLaMA 3.1 8B fail@k curve crossing a dashed GPT-4o fail@1 line, beside the title of the Large Language Monkeys paper.](https://ai.engineer/api/talk-essay-media/d31c3c808e9715bf5be95476113404c52fb9e2207182efc620da2c827e7f4e81)

Repeated sampling lowers the plotted small-model failure rate.

### Turn vectors into matrices

[15:52](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=952s)

The final step is to match the arithmetic itself to the GPU. Frye suggests that multi-token prediction and multi-sample queries can become approximately free when they occupy capacity the original workload leaves unused. He notes that Kyle Kranen on the Dynamo team reached a similar conclusion and points to Kranen’s infrastructure talk for detailed charts. The condition behind the claim is spare capacity, not an absence of additional work.

The largest low-precision throughput figures belong to **tensor cores**, rather than ordinary CUDA-core arithmetic. Tensor cores specialize in matrix multiply-accumulate operations. Frye focuses on floating-point multiplication; H100 also supports integer tensor operations such as INT8. The practical constraint is the shape of computation, and other tasks can sometimes be reformulated to fit it—he offers a Fourier transform as an example.

A single sequence’s generation step naturally contains matrix-vector work: apply a weight matrix to one activation vector. Multiple candidates provide multiple activation vectors. Stack those vectors as columns, and the same weights can participate in a matrix-matrix multiplication. This Python example makes the shape change explicit, using a shared weight matrix and four candidate columns:

```python
import torch

weights = torch.randn(128, 128, device="cuda", dtype=torch.float16)
candidates = torch.randn(128, 4, device="cuda", dtype=torch.float16)

# One candidate: matrix-vector multiplication.
one_output = weights @ candidates[:, 0]

# Four candidates sharing weights: matrix-matrix multiplication.
batch_output = weights @ candidates

print(one_output.shape)    # torch.Size([128])
print(batch_output.shape)  # torch.Size([128, 4])
```

The first output corresponds mathematically to the first column of the batched output. The change exposes more work together; actual tensor-core utilization still depends on dimensions and the selected kernel.

Frye describes a microbenchmark, tentatively attributed to ThunderKittens and Hazy Research, in which populating only one column of a wider matrix yields roughly 1/N of the useful performance. Filling more columns increases useful throughput. That is the intuition behind the slide’s description of matrix-vector to matrix-matrix upgrades as “free”: additional work can use arithmetic slots that were already being underused, rather than increasing time in direct proportion to the number of candidates.

![Slide states that LLM generation is heavy on matrix-vector operations and that upgrades to matrix-matrix operations are “free.”](https://ai.engineer/api/talk-essay-media/c455fd709f4f347c725c02541d019e819b1dca05bd34bf8468d03c2935bda482)

The slide describes upgrades from matrix-vector to matrix-matrix operations as “free.”

Both multiple samples and multiple predicted tokens can move computation in this direction. Frye cites DeepSeek’s multi-token prediction; the [DeepSeek-V3 Technical Report](https://arxiv.org/abs/2412.19437) establishes that training objective. He also tentatively mentions **Llama 4**, without establishing its use of the technique. These approaches create opportunities for fuller matrices, but their end-to-end benefit depends on how the serving system generates, verifies and accepts the additional work.

### A smaller model, used more fully

[18:07](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=1087s)

These hardware properties open a practical deployment option: choose a smaller model that fits on a GPU under your desk, then scale candidate generation until the system meets users’ quality requirements. Where a reliable verifier exists, model size need not carry the entire burden of quality. Frye suggests revisiting research from around ChatGPT’s release, because improving open models make those inference-time strategies worth considering again.

For the terminology underneath those decisions, Frye offers the [Modal GPU Glossary](https://modal.com/gpu-glossary). Its purpose is to connect the hardware and software stack: a reader encountering a warp scheduler can follow the relationships to streaming multiprocessors and the NVIDIA CUDA compiler driver, rather than learning each term in isolation.

Running the hardware need not mean maintaining a machine locally. Frye closes with Modal, his employer’s serverless GPU platform. He describes its rewritten container stack as a foundation for serverless Python handling data-intensive and compute-intensive workloads, including language-model inference. The deployment abstraction remains useful; the hardware knowledge helps determine what work to place behind it.

## Resources

- [Modal GPU Glossary](https://modal.com/gpu-glossary) — Connected explanations of GPU hardware, CUDA software and performance terminology.
- [Use The Index, Luke](https://use-the-index-luke.com/) — A developer's guide to writing SQL queries that use indexes effectively.
- [Beat GPT-4o at Python by searching with 100 dumb LLaMAs](https://modal.com/blog/llama-human-eval) — Frye and Halim's HumanEval experiment using repeated Llama 3.1 8B generations and executable tests, with reproduction code linked.
- [Large Language Monkeys](https://arxiv.org/abs/2407.21787) — Research on increasing inference compute through repeated sampling and evaluating candidate solutions.
- [Latency Lags Bandwidth](https://pages.cs.wisc.edu/~markhill/restricted/cacm2004_latency.pdf) — Patterson's October 2004 analysis of differing bandwidth and latency trends across computing technologies.
- [NVIDIA Hopper Architecture In-Depth](https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/) — NVIDIA's explanation of H100 streaming multiprocessors, tensor cores and memory architecture, including preliminary launch specifications.
- [ThunderKittens Now on Blackwells!](https://hazyresearch.stanford.edu/blog/2025-03-15-tk-blackwell) — Hazy Research explains Blackwell kernel design and how matrix dimensions affect tensor-core utilization.
- [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo) — An open-source framework for distributed inference serving.
- [DeepSeek-V3 Technical Report](https://arxiv.org/abs/2412.19437) — The model report describing DeepSeek-V3's architecture and multi-token prediction training objective.

## Slides

### Innovation Partner and Platinum Sponsors

AWS is the innovation partner; six companies are listed as platinum sponsors.

[0:10](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=10s) [AWS is the innovation partner; six platinum sponsor logos appear below.](https://assets.ai.engineer/sites/aiecode2025/sha256/58324e95b3666977d53711e4776163768ec5b59f9f2afb515851388cb341270e.webp)

INNOVATION PARTNER

aws

PLATINUM SPONSORS

Graphite
Windsurf
MongoDB®
daily™
augment code
WorkOS

### Why Every AI Engineer Needs to Know About GPUs

The talk starts with AI engineers building applications on model APIs and introduces what they need to know about GPUs.

[0:17](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=17s) [Talk title beside a green and turquoise cloud illustration.](https://assets.ai.engineer/sites/aiecode2025/sha256/4fd71de15ae148b4864dfb4ed8b984d8fd1159b2897e4fbc0ccd0f7667d15cbd.webp)

Why Every AI Engineer Needs To Know About GPUs

Modal

2

### AI Engineers and the LLM API Boundary

AI engineers work on the product side of the API, building chains, agents, and tooling around models. The diagram places them between research roles and full-stack engineers, with user needs shaping their work.

[1:25](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=85s) [A diagram places AI engineers on the product and user side of the API boundary, between research roles and full-stack engineers.](https://assets.ai.engineer/sites/aiecode2025/sha256/15a2bf2125656c0c7bb7f1fec0358e444f31039336d19b9776927dd40cd4c52e.webp)

AIEs are on the “product” side of the LLM API boundary.

API

Data/Research constrained

ML Researcher / Research Scientist
- training
- evals

ML Engineer/ Data Scientist/ Research Engineer
- inference
- data

Product/User constrained

AI Engineer
- chains/agents
- tooling & infra

Fullstack Engineer
- product
- platform

latent.space/p/ai-engineer

AIE
aws
World’s Fair

### Developers Need to Write Good Queries

The speaker compares AI engineering to using databases: almost all developers need to write good queries, even though very few write or run a database.

[1:51](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=111s) [Few developers write or run a database, but almost all need to write good queries.](https://assets.ai.engineer/sites/aiecode2025/sha256/6ebeba674dc7e9d5a743c16e5f462091055f8615870178fa384a0e7751cb7f48.webp)

Very few developers write or run a database.

But almost all of them need to write good queries.

AIE
aws
World’s Fair

[1:56](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=116s) [Slide says few developers write or run a database, but almost all need to write good queries.](https://assets.ai.engineer/sites/aiecode2025/sha256/30c285149014c2991d1d9dee22cdc0bebab333f9ec728100fe127d3ac7f86899.webp)

Very few developers write or run a database.

But almost all of them need to write good queries.

AIE
Microsoft
smol ai
World’s Fair

### Use the Index, Luke

The speaker compares database indexes to GPU tensor cores: knowing how to use them matters even if you don't build them. Tensor cores perform matrix-matrix multiplication.

[2:30](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=150s) [A screenshot of the Use the Index, Luke SQL indexing and tuning guide.](https://assets.ai.engineer/sites/aiecode2025/sha256/1613ce48a0e39742f76616f38a892a4fa00e0e13c1219864f0988359e178a229.webp)

USE THE INDEX, LUKE!
A Guide to Database Performance for Developers

Table of Contents
Preface
Anatomy of an Index
The Where Clause
Performance and Scalability

SQL Indexing and Tuning e-Book

A site explaining SQL indexing to developers—no crap about administration.

SQL indexing is the most effective tuning method—yet it is often neglected during development. Use The Index, Luke explains SQL indexing from grounds up and doesn’t stop at ORM tools like Hibernate.

Interface text:
AIE
World’s Fair
Microsoft
smol ai

[3:35](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=215s) [The Use the Index, Luke guide with a Tensor Cores label over its heading.](https://assets.ai.engineer/sites/aiecode2025/sha256/340b84aae7e1dfe0d9ce29245a9e7fbe693e079c90d83dbc1f91176549991a2d.webp)

USE THE [illegible], LUKE!
A Guide to Database [illegible] for Developers

TENSOR CORES

Table of Contents
Preface
Anatomy of an Index
The Where Clause
Performance and Scalability

SQL Indexing and Tuning e-Book

A site explaining SQL indexing to developers—no crap about administration.

SQL indexing is the most effective tuning method—yet it is often neglected during development. Use The Index, Luke explains SQL indexing from grounds up and doesn’t stop at ORM tools like Hibernate.

Interface text:
AIE
World’s Fair
aws

### Open Models and Self-Hosting

Open-weight models and the open-source software that runs them are improving quickly. The speaker cites Dynamo as an example and argues that self-hosting now makes sense.

[4:08](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=248s) [Slide says open-weight models and open-source engines are improving quickly, making self-hosting practical.](https://assets.ai.engineer/sites/aiecode2025/sha256/86d09639bdeb9f1801f6065efe22f75cb83a8b7f6c0631e9b43e01bc67fdc3d1.webp)

Open weights models and open source engines are getting better very quickly.

It finally makes sense to self-host!

AIE
Microsoft
smol ai
World’s Fair

### LLM Almanac: Batch Workload Economics

The LLM Almanac argues that short-context batch workloads can beat API prices because throughput matters more than latency. The speaker previews benchmarking results at 12:45 p.m. in Juniper.

[4:15](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=255s) [An Almanac excerpt says short-context batch workloads can beat language model API prices; two charts compare latency and throughput, and a later talk is announced.](https://assets.ai.engineer/sites/aiecode2025/sha256/55fe238aae6f026d37e35ad32935dca23ef49ddb0595f5137517107bbd112138.webp)

THE LLM ENGINEER’S
20 25
ALMANAC

You can readily beat language model API providers on price if you're running batch workloads on shorter contexts.

Chatting and code completion are the most popular applications of large language models, and they are both interactive. Engineering an interactive, streaming, and latency-sensitive language model application is challenging, just as it is for other computer systems (more on that below).

But LLMs can also perform other tasks that are less latency-sensitive, like extracting data from support chat logs or translating a large corpus of documents. There, throughput is the most important factor — the name of the game is queries per second. This setting is much easier to engineer and optimize, as described below, and so it's easier to beat managed services on price.

modal.com/llm-almanac

Upper chart:
Y-axis: Latency (s, Median Time To First Token)
Lower is better
Y-axis ticks: 0.1, 0.2, 0.3, 0.4, 0.5, 1, 2, 3, 4, 5, 10
X-axis: Throughput (requests/s per replica)
Higher is better
X-axis ticks: 2.0–18.0

How do latency and throughput for LLM engines depend on request load?

Lower chart:
Y-axis: Latency (s, Median Time To Last Token)
Lower is better
Y-axis ticks: 0.1, 0.2, 0.3, 1, 2, 3, 5, 10, 20, 30, 100
X-axis: [illegible]
X-axis ticks: 0–28

More today at 12:45pm in Juniper!

AIE
Microsoft
smol ai
World’s Fair

### What Every AI Engineer Needs to Know About GPUs

The talk shifts from why AI engineers should care to what they need to know about GPU hardware.

[4:44](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=284s) [GPU section title beside an abstract green illustration.](https://assets.ai.engineer/sites/aiecode2025/sha256/4c98416881e031d7d11b0f81ab7161cda82439de0d801befc5f243d7b3c64679.webp)

What Every AI Engineer Needs To Know About GPUs

AIE
aws
9

Interface text:
World’s Fair

### GPUs favor bandwidth over latency

GPUs favor high throughput over low latency, with more bandwidth for math than memory. Their highest bandwidth is for low-precision matrix–matrix multiplication, which the speaker distinguishes from matrix–vector multiplication.

[4:45](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=285s) [Slide says GPUs prioritize math bandwidth and achieve their highest bandwidth on low-precision matrix multiplications.](https://assets.ai.engineer/sites/aiecode2025/sha256/b118cd94d49959b1c943c2b65d40815eb02b571f84c6dafa3194249ff4b3b8f9.webp)

GPUs embrace high bandwidth, not low latency.

GPUs optimize math bandwidth over memory bandwidth.

The highest bandwidth is for low-precision matmuls.

10

Interface text:
AIE
Microsoft
smol ai
World’s Fair

### Latency and bandwidth scaling

The speaker dates the end of latency scaling to the Bush administration and says GPUs embrace bandwidth scaling.

[5:40](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=340s) [Slide says latency scaling died during the Bush administration and GPUs embrace bandwidth scaling.](https://assets.ai.engineer/sites/aiecode2025/sha256/948e4e264a6d7fa861417875200df84f675ea3122cfce075b90e9230a47e2db5.webp)

Latency scaling died during the Bush administration.

GPUs embrace bandwidth scaling.

AIE
Microsoft
smol ai
World's Fair

### This is hardware

A mechanical logic gate from the Zuse 1 uses sliding plates to perform an AND operation: when both input plates are in place, a clock-driven plate can push the output plate forward.

[6:03](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=363s) [A mechanical logic gate is pictured above the caption “This is hardware.”](https://assets.ai.engineer/sites/aiecode2025/sha256/7d97809ad1791a93e9a907b72200308bdb018cbc29b21cd19d03fc774044784e.webp)

This is hardware.

Device labels: 0, 0, L

[7:08](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=428s) [A mechanical computing device with the caption “This is hardware.”](https://assets.ai.engineer/sites/aiecode2025/sha256/4129cf65c961d7831b3a63bf2058626ca4fb6dfc6294c46b54669d0bb372d088.webp)

This is hardware.

Device labels: 0, 0, L

### Changes in software

The need to think about performance is reshaping software, from async coroutines and GIL-free Python to GPU acceleration and vectorization.

[8:13](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=493s) [A statement about changes in software surrounded by examples including vectorization, multiprocessing, async coroutines, GPU acceleration and Rust.](https://assets.ai.engineer/sites/aiecode2025/sha256/96a4b9a961ba3e1f36aa264a2238f4012b8efdae7e77b62b876c806f68b669d6.webp)

This is, slowly and inevitably, changing everything in software.

NUMA
Vectorization
Multi-threading
Analytic databases
Multi-processing
GIL-free Python
Async coroutines
GPU acceleration
io_uring
Rust

AIE
Microsoft
smol ai
World's Fair

### SISD and SIMD

SISD processes one data stream, while SIMD applies one instruction stream to multiple data streams and produces multiple results in parallel.

[8:20](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=500s) [Diagram compares SISD and SIMD using colored blocks for instructions, data and results.](https://assets.ai.engineer/sites/aiecode2025/sha256/1d3ae34a6e60243e84157092370680374c50cfedee321bc39cc559ddf2993b56.webp)

SISD
SIMD

Instructions
Data
Result

### Concurrent programs keep big, long pipes busy

While one operation waits, concurrent work keeps high-bandwidth pipelines busy. The Python example gathers multiple asynchronous JSON requests.

[8:39](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=519s) [A bandwidth and delay diagram alongside asynchronous Python code and the statement that concurrent programs keep big, long pipes busy.](https://assets.ai.engineer/sites/aiecode2025/sha256/e7f5f3de2bd5d76bb6bba4df8c6d957d588bef7632936a826bd40dd52d8a46ab.webp)

Concurrent programs keep big, long pipes busy.

Delay
Bandwidth

async def fetch_json(session, url):
    async with session.get(url) as response:
        return await response.json()

async def fetch_many(urls):
    async with aiohttp.ClientSession() as session:
        tasks = [fetch_json(session, url) for url in urls]
        return await asyncio.gather(*tasks)

Modified from: Computer Networks, A Systems Approach. Section 1.5.
book.systemsapproach.org/foundation/performance.html

AIE
Microsoft
smol ai
World's Fair

[8:45](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=525s) [Bandwidth and delay illustrated as pipes, alongside Python asynchronous requests.](https://assets.ai.engineer/sites/aiecode2025/sha256/e0158bbcc349c9905c51149b6ca62ccd6b6ddd3252a4c92b9bf88bbc8fff1dc3.webp)

Concurrent programs keep big, long pipes busy.

Delay
Bandwidth

async def fetch_json(session, url):
    async with session.get(url) as response:
        return await response.json()

async def fetch_many(urls):
    async with aiohttp.ClientSession() as session:
        tasks = [fetch_json(session, url) for url in urls]
        return await asyncio.gather(*tasks)

Modified from: Computer Networks, A Systems Approach. Section 1.5.
book.systemsapproach.org/foundation/performance.html

AIE
aws
World's Fair

### GPUs, parallelism and concurrency

GPUs use parallelism and concurrency from hardware through programming to keep high-bandwidth pipelines busy.

[9:12](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=552s) [Text states that GPUs take parallelism and concurrency further than CPUs to maximize bandwidth.](https://assets.ai.engineer/sites/aiecode2025/sha256/53dfe804c3b6e25d0ee1ef597940a3fb5f6b6b78eb1e30a1a6cfcbea2778abfb.webp)

GPUs take both parallelism and concurrency further than CPUs to maximize bandwidth.

AIE
aws
World’s Fair

### GPU and CPU parallelism

The H100 runs over 16,000 parallel threads at 5 cW per thread, compared with 384 at roughly 1 W per thread for the EPYC CPU.

[9:23](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=563s) [Table compares parallel threads and power per thread for an AMD EPYC CPU and NVIDIA H100 GPU.](https://assets.ai.engineer/sites/aiecode2025/sha256/e547123e817ec30440fd1864039fef07641f346e0416fce98d3ecb38fb0e2308.webp)

GPUs take parallelism further than CPUs.

AMD EPYC 9965 CPU at 500W | 192 cores | 192 × 2 = 384 parallel threads at ~1 W/thread
NVDA H100 SXM GPU at 700W | 132 SMs | 132 × 128 >16,000 parallel threads at 5 cW/thread

phoronix.com/review/amd-epyc-9965-ampereone
[illegible]nvidias-h100-hopper-chip-up-close/

Source: The GPU Glossary
modal.com/gpu-glossary/device-hardware/streaming-multiprocessor

AIE
Microsoft
smol ai
World’s Fair

### GPU and CPU concurrency

The slide contrasts unbounded CPU concurrency with over 250,000 H100 threads: context switching takes 1,000 clocks on the CPU versus one clock on the GPU, helping the GPU keep work moving.

[10:10](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=610s) [Expanded CPU and GPU table compares parallel threads, concurrent threads, and context-switch times.](https://assets.ai.engineer/sites/aiecode2025/sha256/c48490d3e6f22d500a7eae9b74890a28c6cc65acf38c492e8bfb4996d197ba39.webp)

GPUs take concurrency further than CPUs.

AMD EPYC 9965 CPU at 500W
192 cores
192 × 2 = 384 parallel threads at ~1 W/thread
Unbounded concurrent threads
Context switch in 1000 clocks (µs)

NVDA H100 SXM GPU at 700W
132 SMs
132 × 128 >16,000 parallel threads at 5 cW/thread
>250,000 concurrent threads
Context switch in 1 clock (ns)

phoronix.com/review/amd-epyc-9965-ampereone
[illegible]nvidias-h100-hopper-chip-up-close/

Source: The GPU Glossary
modal.com/gpu-glossary/device-hardware/streaming-multiprocessor

AIE
Microsoft
smol ai
World’s Fair

### Patterson’s Law

The speaker describes bandwidth improvements as the square of latency improvements over time, motivating programs that take advantage of growing bandwidth.

[11:01](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=661s) [Slide states that latency lags bandwidth and suggests embracing the quadratic.](https://assets.ai.engineer/sites/aiecode2025/sha256/11e4ba60e8a29d8fc656edecc5d47d3c88821b56a4ccb8d03c87d916badaa285.webp)

Patterson’s Law:
latency lags bandwidth.

Bet on things that embrace the quadratic.

AIE
aws
World’s Fair

### Latency and bandwidth improvements

Across computer subsystems, bandwidth has improved faster than latency over time. The speaker describes a rough square relationship: a 10× latency improvement accompanies a 100× bandwidth improvement.

[11:29](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=689s) [Logarithmic chart comparing relative latency and bandwidth improvements across computer subsystems.](https://assets.ai.engineer/sites/aiecode2025/sha256/1af4f93b9d5f93a81f95bbd3c315a30519187b58e70f4eac0c0f6bdeed424b2f.webp)

Relative bandwidth improvement
Y-axis ticks: 1, 10, 100, 1000, 10,000, 100,000
Relative latency improvement
X-axis ticks: 1, 10, 100

Microprocessor
Network
Memory
Disk
(Latency improvement = bandwidth improvement)

Across systems, latency scales like sqrt(bandwidth).
Source: Computer Architecture, A Quantitative Approach.
acs.pub.ro/~cpop/SMPA/Computer%20Architecture%20A%20Quantitative%20Approach%20(5th%20edition).pdf

Interface text: AIE; World's Fair; Microsoft; smol ai

## Full transcript

[0:00](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=0s) [upbeat music] So, um, what I wanted to talk about today was, uh, what every AI engineer needs to know about GPUs.

[0:25](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=25s) The... Like, so far in the last couple of years, um, most of the things that people have built as AI applications, people who are AI engineers, they've been building on top of model APIs.

[0:35](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=35s) So they use the OpenAI API, the Anthropic API, the DeepSeek API, and they build an application on top of that. And that goes back to kind of like the initial diagram that swyx put out, the like AI...

[0:47](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=47s) like rise of the AI engineer thing. Um, and yeah, probably just mirror would be great. Um, and the, um, so that like having that API boundary is like, is like pretty important, right?

[1:00](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=60s) It's like you can't really build a complex system if everybody has to know how every piece works, and everybody has to know all of it in detail, and there's no like boundaries or breakdowns.

[1:09](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=69s) Like you're just, yeah. You'll compla- collapse in complexity if you do that. Um, so, um, [guitar strum]

[1:19](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=79s) oh, that was me, but I'm down with it. Um, so like, so yeah, sorry. It started off by trying to answer the question of like why every AI engineer needs to know about GPUs.

[1:30](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=90s) Um, and so yeah, so here's our famous diagram, AI engineer on the right of the API boundary, where they're like constrained by the, um, the needs of users rather than like the, like what's possible with research or what infrastructure is capable of providing.

[1:46](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=106s) Um, and the way that I think about this distinction is that, um, it's kind of similar to the way that very few developers need to actually like write a database.

[1:58](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=118s) Um, like almost no one writes a database except in their like, you know, like undergrad classes. And then even very few developers like run a database. A lot of them will use either a fully managed service or, um, just like a hosted service like, uh, RDS on, on Amazon.

[2:15](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=135s) Um, but like almost all developers, despite the fact that they aren't like database engineers, they are users of databases, and they like need to like write... know how to like write good queries.

[2:27](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=147s) They need to know how to like hold the tool in order to press the like buttons on the side. Uh, so there's a famous educational resource that I really love, um, about databases called Use the Index, Luke, um, that's like basically about how to write SQL queries and not like suck.

[2:44](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=164s) Um, and the whole point is like there is a thing called an index. There's a couple of data structures that support it. It talks about things like B-trees and log structured merge trees and stuff.

[2:56](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=176s) And the intent of it isn't that you can then leave and go and like invert a binary tree on a whiteboard so you can get a FAANG job. Like the point of it is to teach you what you need to know so that you can write, like, write queries properly, that use the index and don't not use

[3:08](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=188s) the index. Primary and secondary indices, all these things. So you don't like, you know, that's like a little bit, um, like an easier like prospect, knowing it well enough to be able to use it rather than like build it or innovate on it.

[3:21](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=201s) Um, and I think we're reaching this point now with, uh, with language models where, um, where

[3:28](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=208s) you'll have more ability to like integrate, tightly run your own language models, and so more need to like use the index. Or I guess if you want like the, like one sentence summary of this talk, um, it's, uh, use the tensor cores, Luke.

[3:41](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=221s) Um, so in building your... There's one, there's basically one part of an NVIDIA GPU, um, and an equivalent in other GPUs that is, uh, fast and good and gets better, and it's the tensor core, and it does matrix-matrix multiplication.

[3:56](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=236s) And, uh, you should make sure you're using it and, uh, and not, not using it, just like an index on a database. [clears throat]

[4:07](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=247s) Um, so yeah, so open-- Like I kind of made this point earlier, but open weights models and the open source software to run them, like Dynamo, getting better very quickly, so it finally makes sense to self-host.

[4:16](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=256s) I'm not gonna belabor this point because I'm giving another talk, twelve forty-five, presenting some like benchmarking results that we did on like running, uh, vLLM, SGLang, TensorRT-LLM on like, uh, you know, ten, twelve different models on ten, twelve different workloads, um, to show like what's, what's economical, what's not.

[4:34](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=274s) Okay. So, uh, so that's the why. Um, sort of a slight change or adjustment in what AI engineers, I think AI engineers should focus on, know about. Um, so now what is it that you need to know about engineer-- uh, about, uh, these, these, this hardware in detail?

[4:51](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=291s) So the primary thing is that GPUs embrace high bandwidth, not low latency. That's the like key feature of this hardware. Similar with TPUs, but distinguishes it from pretty much every other piece of hardware that you're used to programming.

[5:06](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=306s) Um, and then in detail, they optimize for math bandwidth over memory bandwidth. So they do like computing on things. That's what they, where they have the highest throughput. So you, you want to align yourself not to latency, but to throughput, and within throughput, you wanna focus on computational operations.

[5:24](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=324s) And then within computational operations, what you wanna focus on if you want to like actually use the whole GPU you paid for, it's low precision matrix-matrix multiplications. Sorry, that wasn't a stutter.

[5:37](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=337s) That was matrix-matrix multiplications, not just matrix vector.

[5:42](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=342s) Um, okay, so for the first point about latency versus bandwidth. Um, so I, like, regret to inform you that the scaling of latency and the reduction of latency in computing systems died during the Bush administration.

[5:54](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=354s) It's not coming back. Um, see a talk later today for an alternative perspective, but, um, yeah, GPUs embrace bandwidth scaling. Um, so a little more detail on that. Um, so this is a computer or a piece of a computer, in case you haven't looked inside one in a while.

[6:11](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=371s) Um, so this is a logic gate from the Zuse 1, um, computer built in Germany in the '30s, kind of first digital computer. Digital, but not electronic. It's mechanical.

[6:21](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=381s) So there are all these actuator plates in it that, that, um, implemented logical operations. So what you see there on the left is a logical operation a- and. So if two plates are pushed down, then if both of them are present, then when it's, then when the other plate pushes forward, it will push the final plate forward.

[6:38](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=398s) That's the logical operation and. And the thing that pushes that, like, the, the thing that pushes forward is driven by a clock, like a literal clock. Um, I guess now everybody has Apple Watches, but you know, there was a time when you would have a physical clock for that sort of thing.

[6:53](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=413s) So this, um, uh, the clock, uh, like dro- like, drives these systems and causes them to cal- like, compute their logical operations, right? So every time the clock ticks, you get a new operation.

[7:06](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=426s) And so you can just, y- you know, so we, we, we've changed computers a little bit in that we use different physics to drive them, but it's still the same basic, like, abstract system.

[7:15](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=435s) There's a sort of a motive force that happens on a clock cycle that leads to calculations. Um, and the cool thing about that is that if you just make that faster, literally nobody has to f- like, like, think about anything and the computer gets better.

[7:30](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=450s) So this was the, like, primary driver of computers getting better in the '90s. No recompiling, no rewriting your software. Everything just got better, 'cause now the clock started going, like, twice as fast, right?

[7:44](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=464s) And time is very virtual in, in computers and so, like, the program couldn't possibly know the difference. Um, so that was really great during that, like, that, like, mid to late '90s period and then that, like, fell off a cliff in the early 2000s.

[7:56](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=476s) And this has, like, impacted a lot of computing over the last two decades, but actually its effects are, like, still being felt, like, all this switch from being able to kind of avoid the, like, uh, needing to think about performance.

[8:11](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=491s) So this is, like, kinda slowly and inevitably changing pretty much, like, everything in software. Um, all kinds of things you've seen around concurrency, uh, GIL-free Python, multi-processing, async coroutines.

[8:22](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=502s) Um, so there's, like, couple, like, kinda detailed things to dive in here. I wanna make sure that I give enough time to talk about the GPU stuff. But there's kinda two notions of how to make things faster without doing that.

[8:34](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=514s) One is parallel, so, like, when you have a clock cycle, just do two things instead of one. Sounds like a good idea. Um, the other one is concurrent, which is a little bit trickier, but it's like, so you start doing something, clock cycle hits, you start running a calculation.

[8:47](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=527s) Maybe that calculation takes five clock cycles to finish. Like, instead of waiting for those clock cycles to finish, try and do five other things with the next couple clock cycles.

[8:56](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=536s) Makes your programs really ugly 'cause you write, have to write async/await everywhere, um, and yeah, uh, if you're writing Rust, it's, uh, it's a world of Pin. Um, but yeah, but it helps you keep these, like, these super high bandwidth pipelines busy.

[9:11](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=551s) Um, and so these, like, concurrent and parallel, these are two strategies to maximize bandwidth that are adopted, like, at the hardware level all the way up to the programming level with GPUs to take this, like, f- bandwidth further than CPUs can.

[9:26](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=566s) So, uh, GPUs take parallelism further than CPUs, so I'm comparing an AMD EPYC CPU and an NVIDIA H100 SXM GPU here. Uh, the figure of merit here is the number of parallel threads that can operate and the wattage at which they operate.

[9:42](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=582s) So an H1... like a, like an AMD EPYC, uh, CPU can do two threads per core at about one watt per thread. Um, that's not bad, but an H100 can do over 16,000 parallel threads at five centiwatts per thread, which is pretty, um, pretty amazing.

[10:00](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=600s) Uh, very big difference. And, uh, parallel means, like, literally every clock cycle, all 16,000 threads of execution make progress at the exact same time. So what about concurrency? So it may look like CPUs have an advantage here 'cause effectively concurrent threads are unbounded.

[10:16](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=616s) Like, you can just make a thread in Linux, like, it's free. Government doesn't want you to know this. Um, uh, and but there's a limit on H100, so it looks like, oh, wow, oh, only 250,000 threads?

[10:26](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=626s) What am I supposed to do with that? Um, but the difference here is context switching speed. How quickly can you go from executing one thing to another? So if it, like, if our purpose was to take advantage of every clock cycle, and it takes us 1,000 clock cycles, like a microsecond, to context switch, then our concurrency is,

[10:43](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=643s) like, actually, like, pretty tightly bounded, um, 'cause we can't do, uh, a thing for a whole thousand clock cycles. But in GPUs, context switching happens literally every clock cycle.

[10:53](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=653s) It's down there at the warp scheduler inside the hardware. Um, if you have to think about it that hard, you're probably having a bad time, but if you, uh, but normally it's just making everything run faster.

[11:04](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=664s) Um, so there's not really a name for this, uh, the phenomenon that's, that's driving all of this work, um, but David Patterson, who came up with RISC machines, um, and worked on TPUs, uh, wrote it down, so I call it Patterson's Law, latency lags bandwidth.

[11:18](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=678s) Um, so, like, why are, why are we doing all these things the, to, like, rewriting our programs, rethinking them in order to, to, like, take advantage of increasing bandwidth and, you know, bandwidth is replacing latency scaling?

[11:30](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=690s) It's 'cause if you look across a variety of different subsystems of computers, networks, uh, memory, disks, the latency improvement is actually the square of... or sorry, the, um, the bandwidth improvement is the square of the latency improvement over time.

[11:45](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=705s) This is one of those, like, Moore's law style charts, where you're looking at, like, trends in performance over time, and it's like for every 10X that we improve latency, we get 100X improvement in bandwidth.

[11:57](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=717s) And there's some arguments in the article about where, what it, you know, where this comes from. Basically, with latency, you run into the laws of physics. With bandwidth, you just run into, like, how many things can you do at the same time?

[12:08](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=728s) And you can always, you can take the same physics and spread it out more easily than you can, like, come up with new physics to take advantage of. Like, you, you cannot bribe the laws of physics, Scotty, in Star Trek, um, and that's, like, one of the limits on, like, network latency is, like, we, uh, we send

[12:23](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=743s) packets at, like, 70% of the speed of light, so, like, we can't get them 10X faster. Um, yeah. Um, all right, so that's, that's bandwidth. Uh, GPUs embrace bandwidth.

[12:34](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=754s) Maybe big takeaway from Patterson's law is, like, uh, bandwidth has won out over and over again, so maybe bet on the bandwidth hardware. I don't know if the person who's gonna be talking about, uh, LPUs or, or Etched is here, but we should fight about this later.

[12:48](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=768s) Um, yeah, so all right. So what kind of bandwidth, though? Um, arithmetic bandwidth over memory bandwidth, so not moving bytes around. That, that they are high, they have high bandwidth memory, the fanciest, finest Hynix f- high bandwidth memory.

[13:03](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=783s) Um, but they, uh, the thing where they really excel is doing calculations on that memory. And so the takeaway here is that N squared algorithms are usually bad, but if it's N squared operations for N memory loads, the...

[13:20](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=800s) It actually works out pretty nicely. It's almost like maybe Bill Dally and others were thinking of this when they built the chip. I don't know. Um, so, like, arithmetic intensity is the term for this or, yeah, math intensity, um [audio cuts out].

[13:34](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=814s) And if you look here at the things highlighted in purple [audio cuts out]

[13:38](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=818s) second [audio cuts out] denominated in tera [audio cuts out] that go up into the thousands. Memory bandwidth at the bottom is [audio cuts out]

[13:50](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=830s) . And that has not changed with Blackwell, it's only gotten worse, um, or better. I don't know. Um, the, the ratio's gone up. Uh, so LLM inference works pretty nicely during prompt processing, where you're moving...

[14:03](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=843s) You move eight gigabytes, then you... Like, eight billion parameter model, FP8 quantization, you're gonna move eight gigabytes from the memory into the registers for calculation. You're gonna do about 60 billion floating point operations.

[14:15](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=855s) Um, so that's, that's, uh, you know, doesn't really scale too much with the sequence [audio cuts out] directly. Anyway, you're... The [audio cuts out] that when you then need to do [audio cuts out]

[14:27](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=867s) . Now you need to move those eight billion parameters again. Um, so this is from the, like, GPU's memory into the place where the compute happens. Like, you can't...

[14:35](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=875s) You have to... You know, it's von Neumann architecture. You can't, like, keep the things, um, it's... Compute on stuff that is, i- is in place. Um, so LLM inference works great during prompt processing, not so much during decoding.

[14:48](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=888s) Um, so one way to get around this is to just do more stuff when you're decoding. So one example is to take a small model, so eight billion parameters, and run it, like, 1,000 times on the same prompt.

[14:59](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=899s) Now you're loading the weights, and you only load the weights one time, but then you generate, like, 10,000 things. Um, and, uh, so there's, like, kind of an inherent advantage there to small models for being more sympathetic to the hardware.

[15:11](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=911s) You can actually match quality if you do things right, if you have a good verifier. Either, in this case, this is, does it pass a Python test? Um, that allows you to pick, uh, the one of your 10,000 outcomes.

[15:23](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=923s) And so you can use Llama 3.1 8B to match GPT-4o with, like, 100... Yeah, 100 generations. So that's a... The figure on the left is a reproduction. Um, I read a research paper, I sat down, spent a day coding, and I got the exact same result on different data in a different model. [audio cuts out]

[15:40](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=940s) any research for that [audio cuts out]. And so that's... So this is, like, this is, this is legit. This is real, this is real science, you know? Um, and it's, uh, so it's, it's a real phenomenon, and it fits with the hardware.

[15:52](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=952s) Um, so lastly, like, so we wanna do more, like... We wanna do, like, throughput-oriented, like, large scale activities. We wanna do it with, uh, like, computation and mathematics, not with memory movement.

[16:06](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=966s) And the specific thing we wanna do is low-precision matrix multiplication. Um, and the takeaway here is that some surprising things are gonna turn out to be approximately free. I don't have time to go into details on this, but it turns out Kyle Kranen, also on the Dynamo team, ha- like, came to the exact same conclusion.

[16:23](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=983s) We were talking, you know, comparing notes last night. So check his talk in the afternoon of the infrastructure track if you want n- less hand-waving and more charts. Um, so things like multi-token prediction, multi-sample query, all this stuff suddenly becomes, like, basically free.

[16:38](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=998s) Uh, and the reason why is that the latest GPUs, NVIDIAs and others, have this giant chunk in them, the tensor core, that's specialized for low-precision matrix-matrix multiplication. Uh, and so that's, you know, all these things in purple here that have the really big numbers are tensor core output, uh, not the CUDA core output.

[16:58](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=1018s) Um, and tensor cores do exactly one thing, and it's floating point matrix multiplication. Um, bit of a tough world to live in as, like, a... If you're a theoretical programmer to, to discover that there's only one data type you're allowed to work with.

[17:12](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=1032s) But you just get more creative, right? You can do a Fourier transform with this thing if you want. Um, yeah, so the generation phase of li- language models is very heavy on matrix vector operations, if you just, like, write it out at first.

[17:24](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=1044s) Um, so the things that are basically free are things that can upgrade you to a matrix-matrix operation. There's some micro benchmarks from the, um, ThunderKittens people, I think, Hazy Research, that was basically a tensor core.

[17:36](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=1056s) Looks like it runs at, like, if you give it a matrix and then, like, a mostly empty matrix with one column full, you get, like, one over N of the performance, right?

[17:46](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=1066s) Um, and so, you know, if you just add more stuff there, like, all of a sudden you, like, you're... The, like, performance is scaling to match. Um, so this is sort of...

[17:55](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=1075s) Yeah, this is another phenomenon that pushes you in the direction of generating multiple samples, um, generating multiple tokens, as DeepSeek does multi-token prediction, and I think the Llama 4 models do as well.

[18:07](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=1087s) Um, so yeah. So these are the... So as an AI engineer, the things you should be looking at are, like, okay, like maybe I can get away with running a smaller model that fits on a GPU that's under my desk.

[18:18](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=1098s) Um, uh, and then I just, like, scale it out in order to get the, like, sufficient quality to, to satisfy users. There was a bunch of research on this stuff back in, like, around the release of ChatGPT when there was still, like, a thriving academic field on top of language models.

[18:34](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=1114s) Um, and it hasn't... Uh, like, people have kind of forgotten about it a bit, but I think the model, the open models are good enough that this is, uh, back to being a, a good idea.

[18:43](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=1123s) Um, cool. I think I only have about 10 seconds left, so I'll just say if you wanna learn more, uh, I wrote this, uh, GPU glossary, modal.com/gpu-glossary. It's, uh, CUDA docs for humans.

[18:55](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=1135s) Attempt to, like, explain this whole software and hardware stack in one place with lots of links, uh, so that when you're reading about a warp scheduler and you've forgotten what a streaming multiprocessor architecture is and how that's related to the NVIDIA CUDA compiler driver, it's, like, one click away to get all of those things.

[19:12](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=1152s) Um, so if you want to run these, uh, uh, uh, on this hardware, um, no better place than the platform that I work on, uh, Modal, uh, serverless GPUs and more.

[19:25](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=1165s) Um, we sort of, like, ri- ripped out and rewrote the whole container stack to make, um, like, serverless Python for data-intensive and compute-intensive workloads, like language model inference, work well.

[19:37](https://www.youtube.com/watch?v=y-UGrYbJsJk&t=1177s) Um, and so you should definitely check it out. Come find us at the expo hall, and we'll, uh, you know, talk your ear off about it. All right. Thank you very much. [outro music]
