What every AI engineer needs to know about GPUs

If you build an application on the OpenAI, Anthropic or DeepSeek API, how much should you need to know about the hardware behind it? The API boundary makes that application possible: a complex system becomes unmanageable if every developer must understand every component in detail. In the Rise of the AI Engineer diagram that Charles Frye opens with, AI engineers occupy the product side of that boundary, working backward from user needs rather than forward from research or infrastructure capabilities.

Databases offer a useful precedent. Few developers build a database, and many never operate one themselves; they use a managed service such as Amazon RDS. Yet they still need to write good queries. Use The Index, Luke teaches the underlying machinery insofar as it helps developers use indexes effectively. Frye invokes B-trees, log-structured merge trees, and primary and secondary indexes as examples of the knowledge beneath that abstraction. The objective is practical literacy, not the ability to implement a database from scratch.

Language models are reaching a similar point. As engineers gain more opportunities to run and integrate models themselves, the equivalent advice is “use the tensor cores, Luke.” Know enough about the hardware to direct work toward its strongest execution units. For NVIDIA GPUs, the central operation is matrix-matrix multiplication on tensor cores; other accelerators have corresponding specialized machinery.

Improving open weights and serving software, including NVIDIA Dynamo, make self-hosting more attractive. Whether it is economical remains a workload question. Frye reserves comparisons of vLLM, SGLang and TensorRT-LLM across roughly ten to twelve models and workloads for a separate talk; this recording explains the hardware principles behind those decisions.

The hardware discussion follows three nested priorities:

1. Bandwidth over latency. GPUs, like TPUs, are designed to complete large amounts of work, rather than minimize the time of every individual operation.
2. Arithmetic throughput over memory throughput. Moving data is necessary, but calculating with that data is where the largest capacity lies.
3. Low-precision matrix-matrix multiplication. Even within arithmetic, the choice of operation matters. Matrix-vector multiplication does not expose the same opportunity as multiplying two substantial matrices.

Each step narrows what it means to use the GPU you paid for. Merely moving a program onto a GPU does not guarantee that its workload matches these priorities.

Frye frames the shift provocatively: latency scaling died during the Bush administration. To explain what changed, he reaches back to the Zuse 1, a mechanical digital computer built in Germany in the 1930s. Its logic used actuator plates. In the AND-gate example, two input plates must both be in position for a moving plate to push the output forward. A literal clock drives the movement.

Each clock tick advances the computation. Electronic computers changed the physics, but retained the useful abstraction of clocked operations. Increasing the clock frequency therefore offered an unusually convenient performance improvement: existing programs could run faster without being rewritten or recompiled. That was a major source of gains in the 1990s.

When those easy clock-speed gains slowed in the early 2000s, software increasingly had to expose more work at once. The consequences still show up in efforts such as GIL-free Python, multiprocessing and async coroutines. Performance improvement became something programs had to participate in, rather than simply inherit from a faster clock.

Two strategies can increase completed work without increasing clock speed:

• Parallelism: perform two operations during a clock cycle instead of one.
• Concurrency: when an operation takes five cycles to finish, start other work during the intervening cycles instead of waiting idle.

At the programming level, concurrency can bring the complexity of async/await or Rust’s Pin. Its purpose is to keep pipelines busy. GPUs apply both strategies throughout the hardware and software stack, pushing them further than CPUs.

Frye illustrates the scale difference with an AMD EPYC CPU and an NVIDIA H100 SXM GPU:

These are architectural illustrations, not measured application efficiency: the EPYC model and workload are unspecified. There is also a distinction between arithmetic cores and threads. NVIDIA’s Hopper architecture description identifies the corresponding H100 figure as an FP32 core count; it does not mean every application thread advances on every cycle.

Concurrency makes the comparison more interesting. A CPU can support a very large number of software threads, while Frye describes the H100 as holding roughly 250,000 concurrent threads. But the useful question is how quickly the machine can switch to ready work. Frye contrasts an illustrative CPU context switch of roughly 1,000 cycles, or a microsecond, with GPU scheduling on each clock cycle. The GPU’s warp scheduler selects ready groups of threads in hardware, avoiding an operating-system context switch each time another group can make progress. This is how concurrency hides waiting rather than merely accumulating threads.

The larger pattern is David Patterson’s Latency Lags Bandwidth, which Frye calls Patterson’s Law. Across technologies such as networks, memory and disks, bandwidth has historically improved much faster than latency. Frye summarizes the historical trend as 10× improvement in latency accompanying 100× improvement in bandwidth. Patterson’s formulation is a rule of thumb—bandwidth improvement of at least the square of latency improvement—not an exact law governing every device.

Latency encounters physical limits. Bandwidth can often grow by replicating existing machinery and doing more things simultaneously. Frye uses packets traveling at approximately 70% of the speed of light to illustrate why propagation cannot become ten times faster. Spreading work across more channels is a more available path than discovering new physics. That history motivates his preference for bandwidth-oriented hardware, while leaving room for a debate with advocates of LPUs and Etched’s specialized approach.

Even with advanced Hynix high-bandwidth memory, a GPU’s capacity to calculate outstrips its capacity to move data. Arithmetic intensity describes the amount of computation performed per unit of data movement. An algorithm with N² operations might normally sound undesirable, but N² operations for only N memory loads can be a good fit for this hardware: each loaded value supports more useful work.

The relevant contrast in the hardware specification table is between enormous arithmetic throughput and the smaller rate at which memory supplies bytes. Frye says this compute-to-memory imbalance grows further with Blackwell. His prompt-processing illustration uses an eight-billion-parameter FP8 model: about 8 GB of weights move from GPU memory toward the compute units, with roughly 60 billion floating-point operations in his stated example. The sequence-length conditions are unspecified, so that operation count is not a general formula for processing an arbitrary prompt.

During decoding, the model needs those weights again to produce the next token. Keeping weights in GPU memory is not the same as keeping them at the point where arithmetic happens; they still have to move through the memory hierarchy. This separation of storage and computation is the von Neumann constraint Frye invokes. Prompt processing can apply weights across many token positions, while single-sequence decoding offers much less work per weight load. The result is a phase distinction that matters even though the model and GPU have not changed.

One response to underused arithmetic capacity is to do more useful work during decoding. Frye proposes taking an eight-billion-parameter model and generating many candidates for the same prompt. He first imagines 1,000 runs, then uses 10,000 outcomes as a larger illustration. The shared mechanism is batching candidates so a weight load serves many calculations, rather than serving only one sequence’s next token. This reuse happens across candidates at a decoding step; it does not eliminate weight movement throughout an entire generation.

More candidates improve the application only if it can identify a good one. For code, executable Python tests can provide a verifier: generate alternatives, test them, and select a passing candidate. This changes the quality question from how often one response succeeds to how often the candidate set contains a success that the verifier can recognize.

Frye reports that Llama 3.1 8B matches GPT-4o with about 100 generations in the presented Python-test verification setting. The comparison is HumanEval pass@100 for the small model against a separately reported GPT-4o zero-shot pass@1 baseline, not equivalence between their individual responses. The displayed chart expresses the same distinction through a declining Llama fail@k curve and a horizontal GPT-4o fail@1 line. Alongside it is Large Language Monkeys, the research motivating the experiment. Frye describes reproducing the core result in a day of coding with a different model and dataset.

The final step is to match the arithmetic itself to the GPU. Frye suggests that multi-token prediction and multi-sample queries can become approximately free when they occupy capacity the original workload leaves unused. He notes that Kyle Kranen on the Dynamo team reached a similar conclusion and points to Kranen’s infrastructure talk for detailed charts. The condition behind the claim is spare capacity, not an absence of additional work.

The largest low-precision throughput figures belong to tensor cores, rather than ordinary CUDA-core arithmetic. Tensor cores specialize in matrix multiply-accumulate operations. Frye focuses on floating-point multiplication; H100 also supports integer tensor operations such as INT8. The practical constraint is the shape of computation, and other tasks can sometimes be reformulated to fit it—he offers a Fourier transform as an example.

A single sequence’s generation step naturally contains matrix-vector work: apply a weight matrix to one activation vector. Multiple candidates provide multiple activation vectors. Stack those vectors as columns, and the same weights can participate in a matrix-matrix multiplication. This Python example makes the shape change explicit, using a shared weight matrix and four candidate columns:

import torch

weights = torch.randn(128, 128, device="cuda", dtype=torch.float16)
candidates = torch.randn(128, 4, device="cuda", dtype=torch.float16)

# One candidate: matrix-vector multiplication.
one_output = weights @ candidates[:, 0]

# Four candidates sharing weights: matrix-matrix multiplication.
batch_output = weights @ candidates

print(one_output.shape)    # torch.Size([128])
print(batch_output.shape)  # torch.Size([128, 4])

The first output corresponds mathematically to the first column of the batched output. The change exposes more work together; actual tensor-core utilization still depends on dimensions and the selected kernel.

Frye describes a microbenchmark, tentatively attributed to ThunderKittens and Hazy Research, in which populating only one column of a wider matrix yields roughly 1/N of the useful performance. Filling more columns increases useful throughput. That is the intuition behind the slide’s description of matrix-vector to matrix-matrix upgrades as “free”: additional work can use arithmetic slots that were already being underused, rather than increasing time in direct proportion to the number of candidates.

Both multiple samples and multiple predicted tokens can move computation in this direction. Frye cites DeepSeek’s multi-token prediction; the DeepSeek-V3 Technical Report establishes that training objective. He also tentatively mentions Llama 4, without establishing its use of the technique. These approaches create opportunities for fuller matrices, but their end-to-end benefit depends on how the serving system generates, verifies and accepts the additional work.

These hardware properties open a practical deployment option: choose a smaller model that fits on a GPU under your desk, then scale candidate generation until the system meets users’ quality requirements. Where a reliable verifier exists, model size need not carry the entire burden of quality. Frye suggests revisiting research from around ChatGPT’s release, because improving open models make those inference-time strategies worth considering again.

For the terminology underneath those decisions, Frye offers the Modal GPU Glossary. Its purpose is to connect the hardware and software stack: a reader encountering a warp scheduler can follow the relationships to streaming multiprocessors and the NVIDIA CUDA compiler driver, rather than learning each term in isolation.

Running the hardware need not mean maintaining a machine locally. Frye closes with Modal, his employer’s serverless GPU platform. He describes its rewritten container stack as a foundation for serverless Python handling data-intensive and compute-intensive workloads, including language-model inference. The deployment abstraction remains useful; the hardware knowledge helps determine what work to place behind it.