LLM Inference

The request-serving path

Inference applies learned parameters without updating them. For a loaded decoder-only language model, the prompt is the supplied sequence; tokens are its vocabulary units. Autoregressive generation selects a continuation token, then conditions the next prediction on that selection. Vocabulary readout and autoregressive generation explains the underlying prediction mechanism. [2]

Input preparation turns application messages into the model’s expected sequence. A chat template serializes roles, contents, and control tokens. Using the wrong template or duplicating special tokens changes what the model receives, even when the application’s message objects look correct. [3]

An execution engine runs the model; a server accepts requests and coordinates their execution. In the TensorRT-LLM deployment example, a compiled engine supplies model execution while Triton supplies hosting and batching. One HTTP request therefore encloses many model operations rather than one indivisible GPU call. [4]

Model weights are shared learned parameters. Request state depends on the current sequence. Prefill processes the supplied prompt; decode continues it incrementally. Waiting before execution, delivering output, and finishing the request add boundaries around those phases, so execution time and client-visible latency need separate measurements. [5]

Latency and throughput boundaries

Metric names are not interchangeable contracts. AIPerf labels its per-request continuation average ITL. GenAI-Perf instead normalizes successive response intervals by the later response’s token count. Neither convention recovers every internal token-emission interval from bundled responses. [6][7]

A p99 latency is the 99th percentile of a specified population of observations, not an average. Request-latency percentiles and response-interval percentiles use different observation units. Record the population, duration, tokenizer, and load alongside the number. [7]

A service-level objective, or SLO, states a target and its required attainment. Serving objectives can constrain both response onset and continuation pace. Task metrics, latency and usage accounting covers their broader design. Meeting a latency objective does not establish answer correctness. [8]

Workload shape and service objectives

A workload has a joint distribution of input and output lengths: how often each combination occurs. Independent averages hide whether long prompts also produce long answers. Those requests can combine expensive prompt processing with prolonged memory residency, reducing the room available for other requests. [9]

Arrival rate counts incoming requests per second. Active concurrency counts requests occupying execution resources; total outstanding requests also includes waiting work. Burstiness means arrivals cluster unevenly over time. A burst can increase outstanding work without changing the long-run average arrival rate. [12]

Measure input length after conversation serialization, including role markers and generation prompts. Complete-request token counts explains why document counts alone are insufficient. Changing formatting can change both total input length and the prefix available for reuse. [3]

Input truncation removes tokens before execution and can discard needed evidence. An output limit instead caps newly generated tokens. A remote service may reject oversized inputs rather than truncate them; the tokenizer setting does not establish the server’s policy. [13]

A supported context window describes admissible sequence length, not reliable use of every position. Controlled long-context studies found position-dependent retrieval performance in several evaluated models. Test relevant-information position and distractors separately instead of treating nominal context capacity as a quality guarantee. [14]

A capacity objective must identify the workload class, arrival pattern, latency percentiles, completion target, and acceptable rejection rate. Throughput comparisons become decision-relevant only after excluding configurations that violate those requirements. A single fast request cannot establish performance under the intended load. [15]

Prefill and iterative decoding

Prompt tokens are already known, so prefill can process many positions together. Causal visibility still restricts each position to itself and earlier positions. Parallel computation does not grant access to future context; Visibility and attention masks explains that distinction. [2]

The final prompt position supplies scores for the first output selection. Processing that selected token supplies the next selection; selection alone does not create its cached representations. [16]

The key-value cache, or KV cache, retains attention keys and values from processed positions. Keeping these representations avoids repeating their computation. It does not eliminate the new computation or retained-state access needed when the next selected token enters the model. [17]

Ordinary continuation remains dependent across selections: choosing a different token changes the prefix and therefore subsequent prediction distributions. Processing a known prompt in parallel and batching independent requests exploit different opportunities; neither removes this within-sequence dependency. [18]

Token selection and stopping

Logits are unnormalized vocabulary scores. Greedy decoding chooses a highest-scoring token; sampling draws from a probability distribution derived from scores. A vocabulary projection produces those scores, but the resulting vector is not itself a token or an estimate of answer correctness. [19]

Suppose three candidates have probabilities 0.60, 0.25, and 0.15 at temperature one. At temperature two, probabilities become proportional to their square roots, approximately 0.47, 0.30, and 0.23 after normalization. The ranking remains unchanged, but lower-ranked candidates become more likely. [20]

Top-k fixes candidate count; nucleus sampling, or top-p, fixes a cumulative-probability threshold. They happen to retain the same set above but need not do so elsewhere. These controls alter selection without retraining the model or validating the selected answer. [22]

A fixed random seed controls only part of reproducibility. Scheduling, numerical execution, hardware, and runtime versions can also matter. For example, vLLM distinguishes seeded randomness from deterministic scheduling or supported batch-invariant execution, and bounds its reproducibility guidance to the same hardware and version. [25]

Constrained decoding masks tokens that would violate a represented structure before selection. It narrows the generation path rather than merely parsing arbitrary text afterward. Constraints during generation covers this mechanism; structural conformity does not establish factual correctness. [26]

Applications must inspect completion status before accepting a generated record. Refusal, output exhaustion, and partial streaming responses require separate handling. A schema-conforming completed result can still contain mistakes; a partial prefix is weaker evidence still. [27]

Streaming delivery and cancellation

Generation decoding selects tokens; tokenizer decoding reconstructs text from their IDs. Incremental reconstruction can buffer output before delivery. Decoding and text preservation explains the representation boundary. A delivered text fragment can contain several tokens or only part of their eventual printable text. [28]

Stop-string handling can intentionally delay text. The inspected vLLM detokenizer withholds a suffix while checking whether it completes a stop string. Generated token IDs and returned text can therefore account for different content, even without network delay. [23]

Backpressure means downstream slowdown constrains upstream production. A slow consumer can instead merely accumulate queued output. Buffering alone establishes neither bounded queues nor producer throttling; those behaviors require evidence from the deployed delivery path. [28]

Cancellation must reach request scheduling. Queued work can be removed; active work must stop receiving further scheduling. Cleanup may wait for an in-flight operation that can still write cache blocks. A disconnected client, terminal request status, and released ownership are therefore distinct observations. [29]

KV state and the resident memory budget

Attention keys provide matching features; values provide the information combined using those matches. The KV cache stores these numerical representations, not conversation text or durable records. Queries, keys, values and information mixing explains their roles. Shared weights describe learned transformations; cached representations depend on processed context. [30]

High-bandwidth memory, or HBM, is accelerator-local memory on hardware that provides it. Capacity measures stored bytes; bandwidth measures transferred bytes per second. Enough capacity to hold a model does not imply enough bandwidth to execute it quickly. [31]

For 32 layers, cached head dimensions totaling 4096 per layer, 4096 retained positions, and two-byte values, one sequence needs 2 × 32 × 4096 × 4096 × 2 = 2,147,483,648 bytes, or 2 GiB. Doubling retained positions doubles these tensor bytes. Cached dimensions need not equal model width in other architectures. [32]

Weights occupy only one allocation. Serving also needs temporary activations, execution workspaces, runtime and communication buffers, and room for transient peaks. Framework-reserved memory and live tensors overlap in accounting; adding them blindly double-counts consumption. [33]

This storage ceiling is not validated concurrency. Representative forward-pass profiling and runtime allocation measurements must establish the actual cache budget; graph storage and transient peaks can reduce it further. [34]

Retained history and allocated slots also differ. A static cache reserves capacity; a dynamic cache grows. Windowed layers can retain less history than the total sequence length. Neither allocating more slots nor changing reservation strategy extends the model’s supported positional range. [24]

Quantization uses lower-precision numerical representations. Weight precision and KV precision affect different allocations. Storage dtype, arithmetic dtype, and device placement are separate choices; reducing weight bytes does not halve all serving memory or latency. Quantization covers numerical methods and quality tradeoffs. Supported kernels and workload-specific validation determine whether a smaller representation helps. [35]

Cache blocks and compatible prefix reuse

Reserving contiguous maximum-length caches wastes space for short sequences and can fragment available memory. PagedAttention maps logical token positions to fixed-size physical KV blocks allocated as needed. Partial blocks can still waste space. This is tensor-storage organization, not automatic disk spill or ordinary CPU paging. [36]

Prefix caching reuses compatible prompt computation across requests; it does not return a cached answer. Matching meaning is insufficient. New continuation tokens still require generation, so long outputs can limit the fraction of latency saved. An HTTP interface alone neither promises nor prevents server-side reuse. [11]

Within a compatible engine and model configuration, reuse identity must preserve preceding context and relevant execution inputs. vLLM’s documented block identity includes the parent hash, exact block tokens, and identifiers such as adapters or multimodal content hashes. Thus an identical suffix does not imply identical state. Cache salts restrict matching but are not independently an authorization mechanism. [37]

Shared blocks require shared ownership accounting. Finishing one request cannot make a block available for overwrite while another request still references it. Once references reach zero, cached contents may remain reusable until eviction. Releasing ownership is different from erasing data. [38]

Dynamic allocation follows growth; static allocation reserves a maximum shape that can support compilation but wastes space on shorter sequences. Offloading moves cache state through host memory, trading accelerator capacity for transfers. Cache quantization reduces precision and storage, yet can add latency when short contexts already fit. [39]

Retaining more cached context also requires fast writes and retrieval. A large storage tier can leave accelerators waiting if it cannot deliver useful state promptly. Capacity beyond the workload’s useful reuse horizon has diminishing benefit. [40]

Compute, memory movement, and execution overhead

A capacity limit prevents required state from fitting. A bandwidth limit delays moving operands. A compute limit delays arithmetic. Arithmetic intensity is operations performed per transferred byte at a specified memory level. Processing more positions together can increase operand reuse, changing which limit matters. [41]

The roofline model expresses the corresponding throughput envelope. Actual execution can fall below it through insufficient parallelism or other inefficiencies. Find the limiting resource connects this model to traces and counters rather than treating peak specifications as predicted request performance. [42]

Prefill often offers more parallel arithmetic; small-batch decode often depends strongly on memory bandwidth. Neither is universal. Batch size, architecture, retained context, and implementation change the balance. Longer histories add retained-state traffic even when the weights remain unchanged. [43][10]

A kernel is an operation executed on the device; a launch is its host-side submission. CPU preparation and dispatch can leave gaps between GPU operations. Compute, operand movement, and host overhead need distinct evidence before choosing an optimization. [44]

Optimizations target different costs. FlashAttention reduces attention data movement through IO-aware tiling while retaining the exact attention algorithm. Exactness here does not promise bit-identical floating-point results across implementations. [45]

CUDA graph replay reduces repeated host setup and launch overhead by preparing a dependency graph for repeated execution. It does not remove GPU computation. Configuring graph support is insufficient evidence that the measured request actually followed that execution path. [46]

Whole-pipeline profiling can overturn a GPU-focused diagnosis. In the Orpheus serving example, token prediction and audio decoding ran on the GPU, yet CPU coordination limited simultaneous streams in some workloads. Device placement alone does not identify the limiting resource. [47]

Batching independent requests

A batch is work executed together, not every outstanding request. Continuous batching changes membership between generation iterations, allowing completed sequences to leave while others continue. New requests still need their own prompt processing. [48]

Shared execution can reuse weights across useful work. Padding fills unused positions; packing organizes useful positions without that filler. Consequently, sequence count alone does not describe the computation or memory traffic of a batch. [41]

A sequence limit bounds participating requests; a token budget bounds positions scheduled in an iteration. A prompt can contribute many positions while an ordinary decode step contributes one per continuing sequence. Changing that mixture changes execution shape and the demand on cache memory. [51]

Aggregate tokens per second can increase while each request progresses more slowly. Batch formation adds initial waiting, and shared execution can lengthen continuation intervals. Sweep batch settings against both user-visible latency and total throughput instead of assuming that higher utilization benefits every request. [52]

Scheduling prompt work and continuing streams

A long prefill can delay a continuing stream when both compete for execution. Chunked prefill divides prompt processing across scheduling opportunities. The benefit is shorter interruptions; the cost can include later prompt completion and repeated state access or scheduling work. [8]

One documented policy schedules decode first and uses remaining token budget for prefill chunks. Smaller budgets can favor continuation latency; larger budgets can favor prompt completion and throughput. These are workload-dependent tendencies, not an ordering that dominates every alternative. [51]

Priority lets some waiting requests bypass others. Starvation occurs when a request keeps losing scheduling opportunities. Continuous batching and priority support do not establish a fairness bound; a service needs an explicit policy for sustained competing traffic. [49]

Unknown output lengths make admission provisional: cache demand grows during generation. Preemption temporarily removes running work from execution; the inspected scheduler can return it to waiting and later recompute discarded state. [29]

Separating prefill and decode across workers changes placement and communication as well as scheduling. Distributed Training and Inference covers that architecture; within-instance scheduling still needs explicit resource and latency priorities. [53]

Queueing and sustainable capacity

Goodput here means completed requests meeting the declared per-request latency criteria per benchmark second. It differs from raw completions and excludes errors or rejections. Timely completions can still be wrong answers. [6]

The following constructed comparison fixes hardware, warm execution, 128 input tokens, and 64 output tokens. Each trial starts empty and observes 60 seconds. Compliance requires TTFT at most 1,000 ms and TPOT at most 50 ms. No cancellation or execution errors occur. [15]

Bounded queues and admission limits make overload explicit. Resource-aware limits matter because requests have unequal costs. Rejection prevents unlimited waiting; degradation reduces work but must expose reduced completeness. Output caps bound generation while potentially truncating useful answers. [1]

Serving experiments and diagnosis

A closed-loop test waits for replies before issuing replacement work, often maintaining fixed concurrency. An open-loop test schedules arrivals independently. As responses slow, a closed-loop client reduces its own offered rate; an open-loop test can expose growing outstanding work and explicit overload. [54]

Coordinated omission hides delay when stalled issuance removes work or starts its latency clock too late. Record scheduled arrival separately from actual send and completion. MLPerf LoadGen preserves scheduled timestamps so client issuance delay remains visible instead of disappearing from the measured residence time. [56]

Worker startup can include moving weights from storage through host memory into accelerator memory. Separate this from steady execution. Warmup is another condition: initialization and compilation effects should not be silently mixed with warmed measurements. [57][58]

A warm worker can still receive a cold prefix. Compare cold-prefix and reused-prefix workloads with unchanged request shapes, documented cache reset or isolation, and observed hits. Merely repeating requests does not establish which retained state the server used. [11]

Inspect execution and memory traces when aggregate profiles leave the cause ambiguous. Redundant launches and idle gaps can explain poor utilization without changing model arithmetic. The useful observation connects a specific interval or allocation to a proposed intervention. [61]

A kernel speedup needs correct execution timing, appropriate input sizes, controlled cache conditions, and an explicit numerical tolerance. Establish correctness and meaningful timing covers those requirements; a faster isolated operation is not yet a serving improvement. [58]

Accept a serving change against matched workload results: latency distributions, completed throughput, errors, rejections, unfinished work, and memory behavior. Evaluate answer quality separately through Controlled offline comparisons. AI Cost and Performance Engineering extends the decision to cost attribution and utilization. [56]

Open questions

1. 01

   Bounded streaming under slow consumers remains deployment-specific. Uncontrolled buffering wastes memory and computation; throttling can instead disrupt execution sharing. Progress requires traces showing delivery-queue bounds, producer response, and disconnect-to-abort behavior under deliberately slow consumption. [28]

2. 02

   Portable KV reuse needs a compatibility contract beyond matching text. Model revisions, positional settings, adapters, and tensor representations can change state meaning. Progress would include explicit compatibility checks and equivalence tests that reject incompatible persisted caches before reuse. [37]

3. 03

   Fair scheduling remains difficult when output lengths are unknown and priority traffic persists. Protecting existing streams can delay new prompts indefinitely. Progress requires a stated waiting or service-share bound, tested under sustained mixed traffic without hiding rejected requests. [49]

4. 04

   Cache retention needs evaluation against changing reuse patterns. Additional capacity can improve hits while slow retrieval erases the benefit. Progress requires controlled cold and reused-prefix trials, measured hits, and joint latency, memory, and completion accounting across a changing working set. [40][55]