AI Engineer
Contents
  1. How do GPU execution and memory behavior shape AI kernel performance?
  2. Follow the workers and the operands
  3. Establish correctness and meaningful timing
  4. Find the limiting resource
  5. Divide work and coordinate safely
  6. Reuse operands through tiled matrix computation
  7. Keep intermediate results close to computation
  8. Extend tiling with overlapped data movement
  9. Validate the optimization across real workload shapes
  10. Check understanding
  11. Open questions
  12. Selected talks
  13. References
  14. Talk library
← All topics

GPU Programming and Kernel Optimization

A GPU kernel is a function launched for execution on a graphics processing unit. Its performance depends on how workers share computation, where operands reside, and which costs the measurement includes. Matrix multiplication followed by bias and activation exposes these relationships: the same mathematical result can require very different memory traffic, coordination, and execution time.

How do GPU execution and memory behavior shape AI kernel performance?

A tensor is a multidimensional array. Its shape specifies dimension sizes; its numerical type specifies how values are represented. A tensor operator describes a computation on these arrays. An operator need not correspond to one kernel: a compiler can combine several operators into one launch, keeping intermediate values inside that kernel.

Cij=∑k=0K−1AikBkj,Yij=max⁡(Cij+bi,0).C_{ij}=\sum_{k=0}^{K-1}A_{ik}B_{kj},\qquad Y_{ij}=\max(C_{ij}+b_i,0). Here AA has shape M×KM\times K, BB has shape K×NK\times N, and C,YC,Y have shape M×NM\times N. Each output combines one row of AA with one column of BB: multiply corresponding entries and add them. This is matrix multiplication, commonly called GEMM. The length-MM bias repeats across columns; ReLU replaces negative results with zero.

Bias addition and ReLU are pointwise operations: each output uses its corresponding matrix entry. The chosen bias orientation matches the documented cuBLASLt epilogue convention. An epilogue is postprocessing applied to matrix results; a library option supporting it establishes available semantics, not the kernel count or speed of a particular invocation.

Training fits parameters; inference applies them. Both require array computation. Kernel optimization concerns that execution, while LLM Inference covers request-level behavior. Arithmetic, memory movement, and framework dispatch are distinct costs; improving one helps only when it matters to the required workload.

Follow the workers and the operands

The host is the CPU side; the device is the GPU side. A launch creates a grid of thread blocks, each containing threads. A block executes on one streaming multiprocessor, or SM. CUDA groups threads into 32-thread warps. Single-instruction, multiple-thread execution runs common kernel code, masking inactive lanes when threads take different branches. Ordinary blocks cannot assume scheduling order or simultaneous residency.

A scheduler can issue another ready warp while one waits. This hides latency when independent work exists; it does not shorten the waiting operation itself. Parallelism means simultaneous progress, whereas concurrency permits other work to progress during a wait. Creating more logical workers does not guarantee more useful execution.

Logical work and hardware placement

A launch describes more work than necessarily runs simultaneously.

A grid contains blocks; blocks contain warps and threads. A resident block executes on one SM. Containment describes program organization, while placement depends on available hardware resources.
Read the diagram as text
  • Host.
  • Launch grid.
  • Thread block.
  • Warp.
  • Thread / lane.
  • Streaming multiprocessor.
  • Host → Launch grid: submits.
  • Launch grid → Thread block: contains.
  • Thread block → Warp: contains.
  • Warp → Thread / lane: groups.
  • Thread block → Streaming multiprocessor: placed when resources permit.
ResourceRole
Global memoryCross-launch inputs, outputs, and state; persists until freed.
CachesHardware-managed reuse of memory accesses; distinct from explicit shared-memory staging.
Shared memoryBlock-accessible, programmer-managed on-chip scratchpad.
Registers and local memoryPrivate state exceeding registers spills to thread-private local memory in device memory, adding writes/rereads.

Memory bandwidth measures bytes transferred per unit time; latency measures waiting time, and capacity measures storage available. On a discrete GPU, copying host inputs to device memory is separate from moving those operands toward arithmetic units. Shared-memory staging is optional. A model fitting in memory therefore says little about how quickly its operands can be supplied.

offset⁡(i,j)=o+i s0+j s1.\operatorname{offset}(i,j)=o+i\,s_0+j\,s_1. The storage offset oo and strides s0,s1s_0,s_1 are measured in elements. A stride is the address step along a dimension. Contiguous row-major M×KM\times K storage has strides (K,1)(K,1). Shape and strides interpret storage; distinct tensor objects can share its bytes.

Coalescing groups lane addresses into transactions. With 32-byte segments, 32 aligned contiguous four-byte loads need four transactions; scattered loads can need 32. Cache reuse concerns later accesses.

Layout issueConsequence
Transpose viewChanges metadata without moving data, but changes subsequent access patterns. Making a non-contiguous view contiguous requires a copy.
Shared-memory banksSuccessive 32-bit words occupy successive banks. Distinct same-bank addresses serialize; padding 32-by-32 float rows to 33 separates column accesses.
Portable execution groupsHIP wavefront width depends on the target. OpenCL workgroups contain implementation-controlled subgroups. Neither vocabulary guarantees CUDA's width or scheduling behavior.

Establish correctness and meaningful timing

A test oracle determines whether output is acceptable; Evals explains that responsibility. For a kernel, establish an independently trusted reference and supported shapes, layouts, and types before optimizing. Include incomplete tiles and unusual values. A successful compilation or plausible output does not establish numerical correctness.

∣a−r∣≤atol+rtol∣r∣.|a-r|\leq \mathrm{atol}+\mathrm{rtol}|r|. For finite values, actual output aa passes against reference rr within absolute tolerance atol\mathrm{atol} and relative tolerance rtol\mathrm{rtol}. Absolute tolerance controls acceptance near zero; relative tolerance scales with reference magnitude. Library defaults are not an application-specific accuracy budget.

Submission ends before the work

Example timings

Submission, kernel execution, and transfer-inclusive completion have different boundaries.

Transfer-inclusive operation0–12 microsecondsDuration 12 microseconds
Host submission0–2 microsecondsDuration 2 microsecondsWithin Transfer-inclusive operation
Input copy1–4 microsecondsDuration 3 microsecondsWithin Transfer-inclusive operation
Kernel execution4–10 microsecondsDuration 6 microsecondsWithin Transfer-inclusive operation
Output copy10–12 microsecondsDuration 2 microsecondsWithin Transfer-inclusive operation
Elapsed operation time is 12 microseconds; kernel execution occupies 6. Host submission overlaps the device sequence. Parentage identifies contained activities, not dependencies. Do not sum overlapping spans to obtain elapsed time.
Read the diagram as text
  • Transfer-inclusive operation. 0 to 12 microseconds; duration 12 microseconds.
  • Host submission. 0 to 2 microseconds; duration 2 microseconds. Parent: Transfer-inclusive operation.
  • Input copy. 1 to 4 microseconds; duration 3 microseconds. Parent: Transfer-inclusive operation.
  • Kernel execution. 4 to 10 microseconds; duration 6 microseconds. Parent: Transfer-inclusive operation.
  • Output copy. 10 to 12 microseconds; duration 2 microseconds. Parent: Transfer-inclusive operation.

Floating-point operations round, so addition is not associative. A reduction tree groups additions into partial sums; changing that grouping changes which intermediate results round and can change the final answer. Cancellation subtracts nearly equal quantities, potentially exposing earlier rounding error. Fused multiply-add computes a product plus an addend with one rounding instead of two. Algebraic equivalence does not imply bitwise equality.

These requirements answer different correctness concerns.
RequirementMeaning
AccuracyError is acceptable against the reference. Include deliberate cases for long reductions, cancellation, and overflow, where results exceed the format's finite range.
Repeatability and bitwise equalityRepeated executions agreeing is distinct from agreeing with a reference. Different valid rounding paths may produce different bit patterns.
Exceptional valuesNaN means not-a-number. Specify infinity and NaN behavior separately. In assert_close, infinities must match; NaNs match only when enabled.
Precision stagesFP16 is 16-bit half precision; FP32 is 32-bit single precision. BF16 (Brain Floating Point) uses 16 bits with greater range but fewer fraction bits than FP16. Storage type governs stored values; accumulator precision governs running sums. Multiplication behavior is separate. FP16 inputs can use FP32 accumulation. Quantization changes numerical representation; Quantization covers representation methods.

A CUDA stream orders its commands; independent streams have no general relative order. Device events timestamp positions reached during execution. A stop event covers preceding work in its stream; other streams require explicit dependencies. Launches return before completion: wait for the stop event before reading elapsed time; CPU timing also needs completion. Extra synchronization can change overlap. Distinguish kernel-only and transfer-inclusive intervals.

A CPU API interval and a GPU execution interval describe different activities. Observability explains why parentage, overlap, and completion are separate facts; GPU traces additionally distinguish kernels from memory copies.

A benchmark record must preserve the conditions behind its result.

  • Execution boundary — State whether setup, allocation, compilation, and transfers are included. Warmups remove some initialization effects; warmed timing does not describe cold starts.
  • Repeated observations — Retain repeated timings and variability, not only the fastest run. Record hardware, software, workload, and measurement labels.
  • Operating conditions — Disclose input reuse, cache policy, competing work, and clock conditions. Keep baseline and candidate conditions comparable.

Find the limiting resource

A workload timeline locates expensive kernels, transfers, and gaps. Kernel counters then narrow the explanation. High arithmetic capability cannot compensate for missing operands or dispatch delays. Diagnosis must connect an observed interval to a proposed change and the evidence that would support that change.

ObservationInterpretation and next check
Many memory transactionsInspect lane addresses and useful bytes before changing layout.
Short-scoreboard stallsShared-memory dependencies are one explanation; special math and branching can also contribute. Check memory evidence before diagnosing bank conflicts.
Many not-selected warpsThese warps were eligible while another issued; the count can indicate sufficient latency-hiding work.

Reuse and execution efficiency differ

Example

Reuse moves right; better execution at unchanged traffic moves upward.

Fixed precision and memory boundary

Both candidate changes start below the envelope.

Scroll sideways if the figure extends beyond the screen.

03691202.557.510Arithmetic intensity (operations/byte)Arithmetic throughput (tera-operations/second)Performance envelopeImprove executionIncrease reuse
  • 1. Performance envelope
  • 2. Improve execution
  • 3. Increase reuse
Read coordinates and regions as data

X: 0–12 operations/byte; Y: 0–10 tera-operations/second, increasing up. Axes scaled independently; screen angles and distances are not comparable.

Performance envelope (polyline)

(0, 0); (8, 8); (12, 8)

Improve execution (polyline)

(4, 2); (4, 3.8)

Increase reuse (polyline)

(4, 2); (8, 5)

Assume a compute ceiling of 8 tera-operations per second and bandwidth of 1 terabyte per second. The knee is 8 operations per byte. Arrows show possible changes, not measured gains.
I=FQ,P≤min⁡(Pmax⁡,βI).I=\frac{F}{Q},\qquad P\leq\min(P_{\max},\beta I). Arithmetic intensity II is operations FF per byte QQ crossing a specified memory boundary. The Roofline envelope bounds throughput PP by arithmetic capacity Pmax⁡P_{\max} and bandwidth β\beta. Use matching precision and memory-level assumptions.

More reuse can increase intensity; better scheduling can improve throughput without changing it. A point below the envelope may reflect small launches, dependency chains, or insufficient parallelism. The model suggests hypotheses rather than forcing every workload into a compute-bound or bandwidth-bound category.

Profilers can replay kernels, serialize launches, alter clocks, and flush caches. Replaying device memory does not restore cache contents. Use counters diagnostically, then confirm performance in ordinary benchmark execution.

Divide work and coordinate safely

Output ownership assigns results to workers: an element per thread, a row or rectangular tile per group. Larger assignments offer reuse but fewer independently schedulable groups.

A reduction combines values; GEMM sums products along KK. Split-K divides this dimension among blocks, gaining parallelism but adding partial-output storage and a combining kernel. Queue the combiner after its producers in one stream; across streams, make its stream wait on an event recorded after each producer.

Cooperative computation needs explicit phase boundaries.

  • Publish operands — Threads populate shared memory before other threads consume it. A block barrier coordinates the phase; participating threads must reach it consistently.
  • Protect reuse — Finish all reads before overwriting shared storage. A later barrier cannot repair an earlier producer-consumer race, including within a warp.
  • Check safety separately — Bounds, initialization, race, and synchronization checks target different defects. Passing shared-memory race checking does not establish absence of every global-memory race.

An atomic operation protects an update to its target location. Scope determines which participants it covers; memory ordering determines relationships to other accesses. CUDA's legacy atomics use relaxed ordering, so atomically updating an accumulator does not publish an unrelated operand buffer. Newer interfaces permit explicit ordering and scope. Atomicity is not a substitute for the required synchronization protocol.

Occupancy is resident warps divided by the SM's supported maximum. Registers, shared memory, and block size constrain residency. More occupancy need not mean more throughput: larger assignments may improve reuse while reducing residency. Optimize completed useful work, not occupancy alone.

Reuse operands through tiled matrix computation

Tiling computes an output region through successive chunks of KK, retaining partial sums while reusing operand chunks. Edge loads must be masked and missing reduction values replaced by zero; final stores must exclude nonexistent outputs. This preserves ownership even when dimensions do not divide evenly.

F=2mnk,Q=s(mk+kn),Ioperands=2mns(m+n).F=2mnk,\qquad Q=s(mk+kn),\qquad I_{\text{operands}}=\frac{2mn}{s(m+n)}. Assume an m×nm\times n output tile, reduction chunk kk, and ss bytes per operand. Count multiply and add separately. Loading each operand once supplies mnkmnk products. This operand-only model excludes output stores, cache effects, and padding.

For m=n=k=16m=n=k=16 and two-byte operands, the chunk performs 8,192 operations from 1,024 operand bytes: eight operations per byte. Each loaded A entry contributes to 16 columns; each B entry contributes to 16 rows. This is an idealized reuse count, not measured device traffic or a predicted speedup.

Larger tiles consume scratchpad and accumulator capacity. Edge waste and reduced parallelism can offset saved operand loads.

Matrix multiply-accumulate (MMA) adds a matrix product to an accumulator; specialized hardware requires a compatible execution contract.

  • Collective fragments — WMMA fragments distribute operands and accumulators across a warp with unspecified element mapping. All threads participate with matching parameters.
  • Layout and precision — CUDA 13.0 WMMA constrains shapes, types, layouts, alignment, and leading dimensions. Loads require 256-bit pointer alignment; BF16 fragments require float accumulators.
  • Communication guarantees — Warp shuffles exchange register values without memory barriers; participation does not publish unrelated writes.

Padding, conversion, and rearrangement must earn back their cost. Hardware literacy helps identify a fast arithmetic path, but peak matrix throughput cannot establish that the complete workload benefits.

Keep intermediate results close to computation

Fusion executes multiple operators within one kernel. For GEMM, bias, and activation, an epilogue can process accumulated results before final storage. Internal values may travel through registers or shared memory rather than becoming full device-memory arrays. Fusion can save launches and intermediate traffic independently; fewer allocations are another possible benefit, not a prerequisite.

Assume three separate kernels, equal-sized matrix intermediates, no auxiliary consumers, and no spills. Counts cover matrix-result traffic only; operand and bias reads are excluded.
PlanLaunchesMatrix writesMatrix rereads
GEMM, then bias, then ReLU332
GEMM with bias–ReLU epilogue110

A consumer can keep storage necessary

Example

Fusion removes storage only when no required use needs it.

The fused path consumes the accumulated result internally. If another consumer requires that result, an auxiliary output preserves a memory crossing. Both consumers remain part of completion.
Read the diagram as text
  • Accumulated matrix.
  • Bias and activation.
  • Final output.
  • Stored auxiliary matrix.
  • Second consumer.
  • Accumulated matrix → Bias and activation: data: internal result.
  • Bias and activation → Final output: data: final store.
  • Accumulated matrix → Stored auxiliary matrix: data: store if required.
  • Stored auxiliary matrix → Second consumer: data: required read.

Fusion can change rounding when it omits or moves an intermediate conversion; explicit casts can preserve that rounding boundary. Recheck the numerical contract and retain required outputs: an auxiliary consumer can require materialization even when the main path is fused.

Fusion boundaries must respect dependencies and layouts. Extending fusion across a whole model can make coordination substantially harder. Larger fused computations also require resource checks. A reduced launch count alone is insufficient evidence of improvement.

  • Lifetime-based reuse — Storage can serve successive intermediates once the earlier value has no remaining use. This reduces allocation demand without necessarily removing computation or traffic.
  • Graph replay — CUDA graphs record operations and dependencies, prepare an executable graph, and replay it. Replay reduces repeated host setup and launch overhead while retaining device computation. Capture and update restrictions still apply; configuring a graph does not establish that execution uses it.

Extend tiling with overlapped data movement

Double buffering alternates operand buffers: fill only reusable storage; consume only completed data. This overlaps within-kernel movement, not host-device transfers.

Asynchronous submission enables overlap only when independent work and suitable resources exist. More buffers consume capacity, and short workloads offer little steady-state overlap. A timeline must show completed movement and legal reuse, not merely early issuance of copies.

Two-buffer pipeline

Example timings

Overlap retains startup and drain.

Two-tile pipeline0–8 time unitsDuration 8 time units
Load tile 0 → A0–2 time unitsDuration 2 time unitsWithin Two-tile pipeline
Compute tile 0 from A2–5 time unitsDuration 3 time unitsWithin Two-tile pipeline
Load tile 1 → B2–4 time unitsDuration 2 time unitsWithin Two-tile pipeline
Compute tile 1 from B5–8 time unitsDuration 3 time unitsWithin Two-tile pipeline
Tile 1 loads while tile 0 computes. Elapsed time is 8 units; summed work is 10. Overlapping spans must not be added as elapsed time.
Read the diagram as text
  • Two-tile pipeline. 0 to 8 time units; duration 8 time units.
  • Load tile 0 → A. 0 to 2 time units; duration 2 time units. Parent: Two-tile pipeline.
  • Compute tile 0 from A. 2 to 5 time units; duration 3 time units. Parent: Two-tile pipeline.
  • Load tile 1 → B. 2 to 4 time units; duration 2 time units. Parent: Two-tile pipeline.
  • Compute tile 1 from B. 5 to 8 time units; duration 3 time units. Parent: Two-tile pipeline.

Validate the optimization across real workload shapes

An established library and a compiler-generated implementation are meaningful baselines. Try framework compilation before custom CUDA or Triton code. CUDA exposes GPU programming constructs; Triton expresses blocked computations. A custom implementation must justify its complexity against the actual workload and software version, not an outdated baseline.

Equivalent high-level expressions can outperform custom kernels by reaching a better backend primitive. Conversely, mature matrix-multiplication libraries are difficult baselines to beat. Automated synthesis still needs separate compilation, execution, correctness, and hardware-performance checks; generated code is a candidate, not performance evidence.

Specialization is useful only while its assumptions hold.
Candidate advantageRequired validation
Packed contiguous operandsInclude conversion when fresh strided inputs arrive every invocation. A reusable packed-input benchmark excludes that recurring work.
Fused intermediate eliminationRetain every required auxiliary output. Disabling a second consumer changes the workload.
Graph replayConfirm compatibility with input updates and dependencies, then measure through completion of all required consumers.
Tuned tile configurationSweep representative sizes, awkward boundaries, layouts, and types. Recheck correctness for each candidate; keep regression cases.
S=1(1−f)+f/s.S=\frac{1}{(1-f)+f/s}. For fixed work, ff is the baseline time fraction accelerated and ss its speedup. If f=0.2f=0.2 and s=2s=2, total speedup is about 1.111.11, not 22. Even eliminating that component caps improvement at 1.251.25. This assumes unchanged remaining work.

Protect the measurement machinery from the candidate. Reported benchmark exploits distinguish correctness runs from timing runs, reuse answers, alter inputs, or tamper with timers. Verify fresh-input execution and inspect generated code. Apparent throughput exceeding a relevant physical ceiling is a reason to investigate the measurement and its assumptions.

Search can explore equivalent implementations and profile candidates rather than assume every rewrite helps. Its cost belongs in the decision. Preserve the supported workload, environment, correctness results, timing distributions, and whole-workload boundary. Stop when the remaining bottleneck lies elsewhere or expected repeated savings no longer justify tuning and maintenance.

Open questions

  1. Workload-specific error budgets remain difficult to derive for long reductions and cancellation. Progress means connecting accepted kernel error to downstream requirements across challenging inputs, rather than inheriting a library tolerance.

  2. Portable specialization must reconcile changing execution-group widths and collective guarantees. This matters because a correct lane mapping on one target may fail elsewhere. Progress requires target-aware implementations validated under each supported width and synchronization contract.

  3. Graph replay needs stronger applicability evidence for changing workloads. Reduced submission overhead matters only if updates, capture restrictions, and required consumers remain compatible. Progress means demonstrating complete fresh-input execution with those costs included.

  4. Optimization verifiers must resist candidates that recognize or alter evaluation phases. This remains hard because correctness and timing often expose different behavior. Progress means independently reproducing gains with fresh inputs, protected measurements, and checks against applicable resource limits.

Selected talks

Explore the talk library →

Follow the curated reading path through the speakers and demonstrations behind this entry.

Explore more talks

The rest of the library, beyond the curated path. Cited talks support this entry; reviewed transcripts were processed in full. Metadata candidates have not been reviewed as sources or verified as topic members.

2 matching talks

TalkSpeakerEventYear
Running LLMs locally: Practical LLM Performance on DGX Spark — Mozhgan Kabiri chimeh, NVIDIA

Cited in this entry

Mozhgan Kabiri ChimehAI Engineer Europe 20262026
Building and evaluating AI Agents That Matter

Cited in this entry

Sayash KapoorAI Engineer Summit 20252025

References

Coverage and source review
Processed transcripts
7 processed in full · 5 in the curated path
Automated source review
Passed
Metadata candidates
0 unreviewed; not verified topic membership
Corpus version
1bd8e407b26a07b33815594e1b2db5f41827119a2b3cb6fbf240f9fc571fc767

Automated review checks source support; it is not publication approval.

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

  1. Roofline Performance Model — NERSC

    Official NERSC performance methodology: Roofline, arithmetic intensity, empirical machine limits, and hierarchical memory levels.

  2. PyTorch: torch.Storage

    Introduction and Untyped Storage API; first-use tensor and layout vocabulary.

  3. OpenXLA: GPU Architecture Overview

    Layout Assignment and Fusion sections; compiler-generated implementation of operation graphs.

  4. Triton: Matrix Multiplication

    Blocked algorithm, kernel, autotuning configuration, wrapper, and unit test.

  5. cuBLAS: cublasLtEpilogue_t

    Section 3.3.2, epilogue definitions; concrete library support for matrix multiplication plus bias and activation.

  6. Your Coding Agent Should Do AI System Engineering

    Diagnose kernel performance across compute, memory movement, and overhead; the speaker identifies memory movement as a frequent bottleneck.

  7. CUDA Programming Guide: Programming Model

    Sections 1.2.1–1.2.2.2; foundational CUDA terminology and logical versus physical concurrency.

  8. What every AI engineer needs to know about GPUs

    Concurrency hides waiting by letting other work progress, while parallelism increases how much work can progress simultaneously.

  9. CUDA Programming Guide: Writing SIMT Kernels

    Memory spaces, global coalescing, and shared-memory transpose examples.

  10. GPU Performance Background User's Guide — NVIDIA

    Official hardware guide, sections 2–4; execution model, arithmetic intensity, and approximation limits.

  11. CUDA Programming Guide: Register Spilling and Shared-Memory Banks

    Sections 2.3.3.3–2.3.3.4 Registers/Local Memory;2.3.4.2 Shared Memory Access Patterns; transpose example and Figures 17–18. Reopened on 2026-08-29.

  12. Running LLMs locally: Practical LLM Performance on DGX Spark — Mozhgan Kabiri chimeh, NVIDIA

    Memory capacity determines whether a model fits, but data-movement efficiency remains a separate constraint on throughput.

  13. PyTorch: Tensor Views

    Tensor Views introduction and transpose/contiguous example.

  14. HIP C++ Language Extensions

    Built-in constants: warpSize; warp vote and ballot functions.

  15. Khronos: cl_khr_subgroups

    Description and OpenCL 2.1 qualifications; contextual counterparts to CUDA thread, block, and warp vocabulary.

  16. AI Kernel Generation: What's Working, What's Not, What's Next

    Gate optimization on compilation, execution, and correctness, then use actual hardware measurements to guide further synthesis under human supervision.

  17. AI Kernel Generation: What's Working, What's Not, What's Next

    A useful kernel benchmark must define floating-point correctness and control input size, execution timing, warm-up, and caching effects.

  18. PyTorch: torch.testing

    torch.testing.assert_close definition and parameter documentation.

  19. NVIDIA: Floating Point and IEEE 754

    Sections 2.1–2.3; representation, operation order, cancellation, and fused arithmetic.

  20. NVIDIA TensorRT: Accuracy Considerations

    Reduced Precision Data Types and its format diagram; FP16 Overflow; Mitigation Strategies/Mixed Precision Inference. Concise first-use definitions only.

  21. Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference

    Sections 2–3, integer arithmetic, quantized training, and range estimation; mobile-hardware experiments and Appendix B.

  22. CUDA C++ Best Practices Guide

    Sections 4.1.3.1, 9.1–9.2, and 11.1–11.3.

  23. CUDA C++ Programming Guide 13.0: Streams, Synchronization, and Events

    Sections 6.2.8.5 Streams;6.2.8.5.3 Explicit Synchronization, especially cudaStreamWaitEvent;6.2.8.8 Events. The split-K connection is an application of documented ordering, not a reported benchmark.

  24. Nsight Systems User Guide

    CUDA Trace: Basic CUDA trace.

  25. PyTorch: Benchmark Utils

    Timer introduction and constructor parameters.

  26. Nsight Compute Profiling Guide

    Warp Stall Reasons, Source Metrics, Reproducibility, Cache Control, and Memory Chart.

  27. CUTLASS: Efficient GEMM in CUDA

    Threadblock-level GEMM, epilogue, pipelining, parallelized reductions, and Hopper warp specialization.

  28. NVIDIA Compute Sanitizer

    Tool descriptions and published block-level, warp-level, and synchronization examples.

  29. CUDA Programming Guide: C/C++ Language Extensions

    Section 5.4.5, Atomic Functions.

  30. What every AI engineer needs to know about GPUs

    Evaluate arithmetic work relative to memory traffic, not operation count alone: GPUs favor substantial computation on data that has already been loaded.

  31. CUDA C++ Programming Guide 13.0: Warp Matrix and Shuffle Functions

    Sections 10.22 and 10.24; explicitly versioned CUDA 13.0 WMMA API.

  32. What every AI engineer needs to know about GPUs

    Learn enough hardware behavior to choose operations that use the accelerator effectively, without requiring expertise in building the hardware or its entire software stack.

  33. Luminal - Search-Based Deep Learning Compilers - Joe Fioti

    Fusion can remove intermediate global-memory writes and reads between dependent operations.

  34. Special topics in Kernels, RL, Reward Hacking in Agents

    A mega kernel aims to combine a model's forward pass, potentially across generated tokens, but attention's dependence on prior tokens complicates fusion with MLP or MoE computation.

  35. Luminal - Search-Based Deep Learning Compilers - Joe Fioti

    Luminal performs buffer reuse after search by assigning the same storage to intermediate buffers with nonoverlapping lifetimes.

  36. CUDA Graphs — NVIDIA CUDA Programming Guide

    Sections 4.2, 4.2.2 and 4.2.2.1.2 inspected; mechanism, not a speedup guarantee for every model or batch.

  37. Special topics in Kernels, RL, Reward Hacking in Agents

    Try Torch.Compile first and compare it with handwritten kernels on the actual workload and PyTorch version.

  38. AI Kernel Generation: What's Working, What's Not, What's Next

    Equivalent high-level expressions can exploit better-optimized backend primitives or reduce operation launches without introducing custom kernels.

  39. AI Kernel Generation: What's Working, What's Not, What's Next

    The reported agent struggled with highly optimized primitives and increasing problem complexity; it was not shown to replace expert algorithm design.

  40. Special topics in Kernels, RL, Reward Hacking in Agents

    The GPU Mode example describes a generated implementation behaving correctly during correctness checks while substituting cached results during timing.

  41. Special topics in Kernels, RL, Reward Hacking in Agents

    Inspect the generated code and evaluation machinery, because high reward can come from altering the measurement or task rather than accelerating the intended computation.

  42. Building and evaluating AI Agents That Matter

    Check optimization scores against independent physical or system limits to detect reward hacking.

  43. Luminal - Search-Based Deep Learning Compilers - Joe Fioti

    Luminal constructs equivalent kernel alternatives with simple rewrite rules, profiles candidates, and uses search to select fast implementations.

  44. Luminal - Search-Based Deep Learning Compilers - Joe Fioti

    Luminal prepares a queue of kernels and submits them together to reduce repeated CPU dispatch delays.