Mastering LLM Inference Optimization: From Theory to Cost-Effective Deployment

A language-model deployment can become prohibitively expensive even when the model itself fits comfortably on a GPU. Every incoming prompt creates work and consumes memory, while each generated token adds context that must remain available for subsequent generation. At scale, the limiting question is not simply whether a model loads: it is how many active requests its remaining memory can sustain. Mark Moyou, an NVIDIA solutions architect, approaches inference economics through this distinction between resident model weights and the growing state required to serve users. 0:22

Real deployments often mix a commercial model API with an internally hosted open-source or fine-tuned model. Regardless of where a request goes, the underlying problem remains: the prompt enters the GPU workload, generation proceeds token by token, and previously generated context must remain available to produce a coherent continuation. 1:09

That retained context becomes the key-value cache, or KV cache, an important driver of deployment cost across both hosted APIs and self-managed GPUs. A request first has to be translated into the model’s vocabulary, then its entire prompt must be processed before token-by-token generation can begin. A heavily loaded service must perform that initial work for new users while continuing to generate responses for requests already in flight. Finally, generated model tokens must be converted back into human-readable text. 3:57

Tokenization translates text into units from a model-specific vocabulary. Those units are chosen to represent the training corpus efficiently across natural languages, programming languages, and other text; they are not necessarily whole words. The vocabulary belongs to the model and tokenizer—for example, the Llama tokenizer Moyou uses—not to the application sending the request. 5:36

The inference pipeline has two distinct computational phases. During prefill, the system processes the complete prompt and computes attention over its tokens. During decoding, it generates subsequent tokens one at a time while retaining previously processed context on the GPU. These stages place different demands on hardware, and optimized implementations exploit different levels of GPU memory to accelerate them. 6:18

A useful rough estimate is that one token represents about four characters. Internally, each vocabulary entry has a numerical token ID, so incoming text becomes a sequence of integer lookups. The difference between a prompt ending in smart and one ending in smarter illustrates why word boundaries and token boundaries need not coincide: a small textual change can produce a different sequence of subword units. 6:49

Each token ID maps to an embedding vector: a numerical representation the model can compare and manipulate. Stack the vectors for an entire prompt together and the prompt becomes a matrix. Adding another token changes that matrix’s dimensions. GPUs are effective inference hardware because the workload repeatedly transforms and multiplies these large collections of vectors. 7:57

Attention determines which earlier tokens matter when interpreting context and choosing the next token. The prompt’s embedding matrix is multiplied by learned model-weight matrices to produce query, key, and value representations, conventionally abbreviated Q, K, and V. These projections describe the same token inputs in different mathematical spaces that the attention mechanism uses to relate context to the next generation step. 9:38

The keys and values are the reusable part. Saving them in the KV cache preserves the model’s relevant attention state instead of forcing the server to reconstruct the entire prompt every time another token appears. Making that cache smaller and faster directly affects both serving performance and the amount of context the GPU can hold. 12:09

The distinction becomes clearest with one request. Prefill processes the original prompt as matrix-matrix work. Once the first generated token is available, its embedding is processed against the existing model and cached context as vector-matrix work; its new keys and values extend the cache. Without that cache, each decode step would redo prior prompt processing. Production servers can batch these operations across many requests, but batching does not erase the underlying difference between a single request’s prefill and decode phases. 12:51

Attention is not a single isolated calculation. An attention head contains its own query, key, and value projections, and multiple heads contribute to producing the next token. The particular Llama architecture Moyou uses as an example has 32 attention heads, meaning many sets of attention computations and associated cached state must be managed together; that count should not be generalized to every Llama variant. 14:11

At FP16, model weights require approximately two bytes per parameter. Moyou's simplified budget puts an eight-billion-parameter model on a 20 GB GPU:

Those remaining 4 GB must accommodate request context, KV-cache growth and other serving demands. Fitting the weights is not the same as serving concurrent requests. 14:35

Measure input and output lengths separately, then distinguish the timing questions:

• ISL is input sequence length; OSL is output sequence length. 15:20
• Time to first token: how long until prompt processing yields the first generated token?
• Inter-token latency: how long between subsequent tokens?
• Total generation time: how long from receiving the prompt to completing the answer? Growing memory pressure under concurrent load can change these measurements independently.

GPU utilization also depends on the type of workload. For some non-LLM deployments, colocating several models can increase throughput per hardware unit. LLM serving presents a different memory tradeoff: Moyou describes a typical production configuration as one model with remaining capacity devoted to active requests and their KV caches. More models are not automatically better if their weights displace the state needed for concurrent generation. 16:38

A single benchmark shape cannot represent a production system whose users submit radically different prompts and expect different response lengths. The joint distribution of input and output sizes determines how much work occurs during prefill, how long decoding continues, and how much GPU memory remains occupied across overlapping requests. 17:23

The resulting request shapes expose different bottlenecks:

• Long input, short output: prefill must process substantial context, and the prompt occupies significant memory, but generation completes after relatively few tokens. 17:57
• Long input, long output: substantial prompt state combines with prolonged generation, filling the GPU and reducing the number of requests that can run concurrently.
• Short input, long output: the first token can arrive relatively quickly because prefill has less context to process, while decoding continues across a much longer response.
• Short input, short output completes the four-way classification: both the prompt and continuation are short. Actual traffic mixes all four shapes, so concurrent memory occupancy depends on their distribution.

Shrinking model weights can free space for additional tokens, but weight compression alone does not describe the complete optimization problem. The engine must also accommodate the observed request shapes, their cached context, and the traffic arriving concurrently. A two-dimensional histogram of ISL against OSL reveals which combinations actually occur rather than relying exclusively on advertised maximum context sizes. 19:16

That distribution informs the maximum input and output boundaries used to size and optimize an inference engine. It also provides a basis for understanding when capacity should grow, how requests compete for available memory, and why a deployment tuned for one workload can perform differently when the traffic mix changes. 20:17

Time to first token must be measured under meaningful concurrency. A single-request prefill measurement does not establish how attention processing behaves when the same server is busy admitting new prompts and generating tokens for existing users. The useful comparison is therefore between workloads with stated load conditions, not an isolated query and a saturated deployment treated as equivalent. 20:41

GenAI-Perf is the benchmarking package Moyou identifies for generating plots that expose these serving behaviors. One useful view examines time to completion across requests: a broad distribution invites investigation into whether different prompt shapes, batching decisions, or scheduling behavior explain why otherwise comparable requests finish at different times. 21:02

A second view plots token-to-token latency against token position. As a response grows, the retained context and memory pressure also grow; substantial variation between later token intervals can indicate that concurrency or scheduling is degrading generation. Stable spacing across positions is the operational behavior to investigate, rather than assuming that a fast first token guarantees a smooth response. 21:57

A third view plots time to first token against input length. Longer prompts require more initial attention work, but the slope reveals how sharply prompt-processing latency grows as the input expands. Comparing that relationship under realistic load makes it possible to distinguish an engine that handles growing context efficiently from one whose prefill cost rises too steeply. 22:29

TensorRT-LLM, or TRT-LLM, is the NVIDIA inference and model-compilation framework Moyou introduces after establishing the workload and its measurements. Triton Inference Server provides a separate serving layer that can host models across different frameworks and hardware targets, reducing the need to maintain an independent model server for each deployment stack. NVIDIA Inference Microservice, or NIM, packages an enterprise-oriented path for deploying optimized configurations without reproducing every engine-building step manually. 23:07

One optimization changes numerical precision. Moving weights from FP16 to FP8 reduces their representation from 16 bits to eight bits, potentially freeing approximately half of the weight-storage memory for the same parameter count. Moyou describes accuracy preservation and faster execution as goals that must be evaluated for the model and supported hardware, rather than as unconditional results for every workload. He identifies Hopper and Ada Lovelace hardware when discussing FP8. 24:15

Other serving optimizations target the active workload directly:

• In-flight batching admits a new request as soon as another finishes, without waiting for every request in an existing batch to complete. 24:52
• Quantized KV cache stores cached attention state at reduced precision, shrinking the memory required by active contexts.
• Paged KV cache changes how cache memory is managed across requests, improving control over GPU memory allocation rather than merely compressing model weights.
• Tensor parallelism splits a model across multiple GPUs, typically within one node.
• Pipeline parallelism processes successive model partitions sequentially and can pass work between nodes for larger deployments.

These strategies solve different problems: batching improves utilization as requests arrive and finish, cache techniques change how much active context fits in memory, and parallelism distributes model execution across devices. Moyou describes an NVIDIA 340-billion-parameter model designed for inference on a single H100 node using FP8, illustrating that model placement depends on both hardware topology and numerical representation. 26:05

TensorRT-LLM extends the earlier TensorRT compilation approach with the additional operations required by the autoregressive inference loop. In the workflow Moyou describes, an engine is compiled for its target GPU rather than treated as a generic artifact that can simply be moved unchanged onto different hardware; the optimization depends on the characteristics of the deployment device. 26:36

The compiled engine still needs a process that accepts requests, manages batching, and exposes model execution to clients. Triton Inference Server fills that role: it loads models from a configured repository and describes execution in terms of tensor inputs and tensor outputs. The division of responsibility is straightforward: TensorRT-LLM produces the optimized inference engine, while Triton hosts and schedules requests against it. 27:25

Moyou’s deployment outlook moves from FP16 toward FP8, because lower-precision weights can leave more room for active tokens, and then toward the announced Blackwell generation’s FP4 capability. Each lower-precision deployment still needs its own quality and performance measurements. NVIDIA Inference Microservice rounds out that outlook by packaging model-specific and GPU-specific configurations as an enterprise deployment option. 27:52

An ISL-by-OSL heat map becomes operationally useful when its observed distribution challenges the configuration used to build an engine. An organization might provision for prompts of 4,000 tokens, then discover that actual requests over the previous week were closer to 1,300 tokens. Those observations can justify rebuilding around the measured workload instead of optimizing exclusively for a theoretical maximum. 29:06

Moyou calls the possibility of varying configurations over time seasonal engines: daytime and nighttime traffic, or other recurring demand changes, might warrant differently sized engines and different scaling behavior. The heat map supplies the input and output bounds across the observed distribution; it does not, by itself, establish that dynamically replacing engines will improve every deployment. 29:55

For mixed request shapes, the typical starting point Moyou describes is one engine configuration capable of handling the full range of incoming traffic. Trying multiple configurations becomes an engineering experiment: check whether a replacement still serves the observed request distribution, then measure its cost and execution behavior. Changing traffic from agent-based systems could make those workload shifts more pronounced. 30:30

Longer context also raises a more fundamental question: can attention scale without making memory requirements prohibitive? Moyou describes approaches that train at shorter context lengths and interpolate to extend usable context, alongside attention implementations that exploit fast GPU memory and hardware interconnects such as NVLink to move data between GPUs. His expectation is that continued architectural and hardware work will address attention bottlenecks, rather than treating ever-larger context as free. 31:38