Quantization

Quantization and the resource constraint

Weights are learned model parameters. Activations are intermediate values computed from inputs. Both are stored in tensors—multidimensional arrays. Quantization maps their values into a smaller set of representable numbers; reconstructing them generally produces approximations. [3][4]

Capacity concerns fitting state; bandwidth concerns moving it; compute concerns arithmetic. Dispatch and conversion introduce additional overhead. These inference bottlenecks explain why a smaller representation can leave latency unchanged. Establish the baseline model, target device, workload, and acceptable quality change before comparing configurations. [7][6]

Distillation trains a student using teacher supervision. Quantization changes numerical representation; combining it with training introduces a separate adaptation process and its data and compute requirements. [8][9]

Bit width, range, and resolution

Bit width counts encoding bits; range describes representable magnitudes; resolution describes spacing between values. Floating point uses a sign, an exponent that controls magnitude, and a significand that supplies significant digits. FP32 and FP16 use 32 and 16 bits. BF16 also uses 16, allocating more exponent bits and fewer fraction bits than FP16. [10]

Numerical format selection therefore trades range against precision. INT8 and INT4 instead identify eight-bit and four-bit integer codes; a quantization scheme determines what real values those codes represent. [10][11]

A cast selects a representable value in another encoding. A quantization scheme can additionally scale values before casting and restore that scale afterward. The FP8 specification separates encoding from conversion policy; neither a format name nor its width establishes hardware support. [12]

Scale, zero point, and reconstruction error

Scale sets reconstructed spacing. Zero point is the code that reconstructs to real zero. Dequantization reconstructs $\hat{x}$; it cannot recover discarded information. These are uniform affine integer equations, not a specification of floating-point quantizers. QuantizeLinear specifies the rounding and saturation rule. [13][11]

Symmetric schemes use zero point zero, often with symmetric bounds. Asymmetric schemes can shift the represented interval using a nonzero zero point. Signed conventions still differ: LiteRT specifies weight codes [−127,127] with zero point zero, but activation codes [−128,127] with potentially nonzero zero points. Bounds belong to the operator contract. [14]

Shared scales and quantization granularity

Granularity identifies which values share quantization parameters. A channel is a feature or output dimension; its tensor axis must be specified. Per-tensor quantization shares one scale, per-channel assigns scales along that axis, and blocked quantization partitions an axis into smaller groups. [11]

Take a weight matrix with output-channel rows [−1, −0.5, 0.5, 1] and [−8, −4, 4, 8]. With codes [−7,7], a shared scale of 8/7 rounds 0.5 to zero. Giving the first row scale 1/7 instead reconstructs 0.5 as 4/7. Separating ranges preserves more local detail. [15]

Finer grouping adds scale metadata and, where used, zero points. It also changes indexing and packing requirements. Smaller groups need not improve every tensor: the two-value blocks above have the same maximum magnitudes as their respective rows. Group size, axis, and layout must match the consuming kernel. [6]

Quantized tensors and arithmetic paths

W4A16 denotes four-bit weights and sixteen-bit activations; W8A8 denotes eight-bit weights and activations. Such labels identify selected representations, not every operator, accumulator, or output. Weights can be prepared ahead of inference, while activation values arise from current inputs. [16][4]

An accumulator holds a running sum of products. Integer operands can accumulate into INT32, then undergo scale restoration and requantization: encoding the result for its next consumer. Storage, multiplication, accumulation, and output precision are separate choices. [17]

Centered operands [2,−1] and [3,4] give $A=6-4=2$. With scales 0.5 and 0.25, reconstruction is 0.25. Output scale 0.125 and zero point zero give code 2. Finite accumulation bounds and output rounding still require implementation checks. [17]

Bias must use compatible units before addition. Under LiteRT's convolution convention, an INT32 bias has zero point zero and scale $s_xs_w$. Adding bias code 2 to the example accumulator gives $A+2=4$, reconstructing 0.5 and yielding output code 4. [14]

Weight-only execution can instead unpack and reconstruct weights before higher-precision multiplication. The documented TensorRT INT4 path does this: four-bit storage does not mean four-bit multiplication. [18]

Language models also retain input-dependent attention keys and values in a KV cache. These are derived numerical representations, not token identities or model weights. Cache precision is a separate configuration choice; weight quantization does not automatically reduce it. [19][20]

Calibration and activation-scale timing

Post-training quantization, or PTQ, prepares a trained model without a quantization-aware training loop. Calibration data supplies representative inputs for estimating numerical distributions or choosing parameters. This numerical calibration differs from probability calibration, which concerns predicted probabilities and observed frequencies. [13]

Representative inputs need not have labels. Some simple weight conversions need no representative inputs at all. Activation calibration instead runs examples through the model to observe input-dependent ranges. [4]

Static activation quantization reuses previously chosen parameters; dynamic activation quantization computes parameters from runtime values. The latter adds range-estimation and conversion work. Both still round and clip. [13]

Timing and sharing are independent dimensions. Per-token scaling assigns a scale to one position's feature vector; it is commonly calculated at runtime. A dynamic range can still be dominated by an outlier among values sharing that scale. [16]

Terminology needs interpretation: some practitioners also call selective layer precision dynamic quantization. Inspect which parameters actually change during inference rather than inferring behavior from that name. [21]

Calibration and development data guide choices; protected assessment tests the selected configuration. Repeated tuning against final results weakens that independence. Preserve independent data boundaries, including rare but important cases rather than relying only on typical inputs. [22][23]

Sensitivity, outliers, and targeted recovery

Outliers lie far from the bulk of a tensor's values. Preserving them can consume a shared range; clipping them can remove useful signal. Weight reconstruction error measures changed weights, layer-output error measures changed computation on inputs, and task error measures consequences. These quantities need not move together. [24][16]

Hold calibration and assessment fixed, quantize one layer at a time, and compare numerical changes with task results. This localizes sensitivity before changing clipping, grouping, or retained precision. [15]

Then test the combined policy. Independently acceptable layer changes can interact, and exhaustive combinations grow rapidly. Selective higher precision spends additional memory to protect sensitive computations; it is not a universal ranking of layer types. [25][21]

Layer reconstruction can improve PTQ beyond independently rounding weights. GPTQ minimizes $\|WX-\hat{W}X\|^2$ using calibration inputs $X$. After rounding weights, it compensates remaining weights using second-order information derived from those inputs. This preparation targets layer outputs, a surrogate for final task quality. [24]

Quantization-aware training and deployment conversion

Quantization-aware training, or QAT, exposes optimization to quantization effects. Fake quantization rounds, clips, and reconstructs values while retaining floating-point tensors. A straight-through estimator supplies approximate gradients through nondifferentiable operations. Parameter updates can then adapt weights to those effects. [26]

PTQ avoids this training loop. QAT adds training data and compute, then conversion into actual low-bit storage. The deployed operators must reproduce the intended rounding, clipping, and scale behavior; successful simulation alone does not establish deployment quality. [26]

Recovery data also matters. Quantization-aware distillation can lose capabilities when its supervision fails to represent the model's earlier specialist training. Adding adaptation is therefore a new capability-preservation experiment, not an automatic repair. [8]

Executable formats and supported kernels

A GPU kernel is device code implementing part of a computation. Fusion combines work inside one kernel. Marlin, for example, rearranges packed weights and group scales offline and schedules dequantization alongside matrix computation. Such layout choices help turn reduced storage into reduced execution time. [6][27]

Quantize/dequantize graph nodes describe conversion semantics; a compiler may fuse them into operators. Unsupported low-precision implementations can instead prevent an engine build. A graph containing low-bit types does not establish which instructions execute. [18]

Conversion can also succeed through permitted floating-point fallback. LiteRT distinguishes that option from requiring integer operators and interfaces. A loadable artifact therefore need not run on integer-only hardware. [4]

Verify the actual path on the target device. Reuse independent numerical references and compare against optimized library or compiler baselines. Code inspection and successful compilation cannot replace hardware correctness and performance tests. [28][29]

Effective storage and runtime memory

NVFP4 illustrates effective storage: sixteen four-bit values require 64 payload bits, plus an eight-bit block scale. That is $(64+8)/16=4.5$ bits per value before its per-tensor FP32 scale. Four-bit names therefore need not mean four total storage bits per value. [1]

Artifact size counts serialized bytes. Resident memory counts allocations present during an identified operating state. Peak memory is the maximum over loading or execution. Profiling determines what remains available for request state; payload arithmetic alone cannot establish sustainable concurrency. [5]

Repacking requires separate inspection. Its existence does not establish simultaneous original and packed buffers or their lifetimes. Record additional loading allocations when observed; otherwise leave peak-memory coverage unmeasured. [6]

Numerical correctness and retained application quality

The numerical reference must implement the declared quantized computation independently. Fixtures should exercise exact zero, ties, saturation, group indexing, bias units, and accumulation bounds. Agreement with this reference establishes implementation behavior; it does not require bitwise agreement with the original higher-precision model. [17][14]

Perplexity is the exponential of average negative log likelihood over a dataset. It measures prediction fit, not task correctness. Aggregate accuracy can also conceal changed cases: newly correct answers can offset newly incorrect ones. Answer-flip counts expose those exchanges but omit changes between two incorrect answers. [31]

Use matched comparisons: preserve model identity, preprocessing, prompt formatting, generation settings, and task judgments. Inspect paired failures and uncertainty, rather than accepting an unchanged mean as behavioral equivalence. [31]

Exercise the actual application harness and relevant context lengths. Reported compression failures emerged only under long-context use. Include consequential slices and safety behavior alongside general capabilities; loading and producing a plausible completion are insufficient acceptance tests. [32][33][34]

Performance under matched workloads

Prefill processes prompt tokens; decode generates continuations incrementally. Their different work shapes can expose different benefits from quantization. Prefill and iterative decoding explains the execution distinction; neither phase has one universal bottleneck. [35]

Time to first token measures response onset; output pace measures continuation delivery; total throughput aggregates work across requests. State the observation boundary and how streaming chunks are handled. Client network time differs from engine time. Latency and throughput boundaries prevents these quantities from being conflated. [36][41]

Warm up, repeat measurements, vary request lengths, and inspect tails. Verify that the load generator can sustain the intended traffic. Larger batches can increase total throughput while slowing individual responses, so spending freed memory on concurrency is a different experiment from reducing latency at fixed load. [42][40][43]

Goodput here counts completed requests meeting declared per-request latency limits per benchmark second, excluding errors and rejections. Timely answers can still be wrong. Preserve a separate application-quality requirement and use serving experiments to distinguish fixed concurrency from independently arriving traffic. [37][36]

Configuration selection and reassessment

Reject unsupported or quality-failing candidates before comparing resource use. Pareto efficiency only means no comparable candidate improves one objective without worsening another; an efficient candidate can still violate requirements. Retaining the baseline is justified when alternatives fail the application's constraints. [7]

Changing the model, calibration distribution, grouping, operators, or device can invalidate earlier conclusions. Reassess affected claims under the new conditions. Release decisions apply acceptance requirements; evaluation records preserve the evidence and conditions behind them. [46][23]

Open questions

1. 01

   Joint sensitivity remains difficult to predict: individually acceptable layer changes can interact. Progress would mean finding harmful combinations with substantially fewer evaluations than exhaustive search while preserving held-out task behavior. [25]

2. 02

   Calibration coverage under changing inputs remains unresolved. Runtime scaling adapts magnitudes but cannot eliminate shared-range limitations. Progress would include controlled tests identifying when changed workloads require new calibration or selective higher precision. [51][21]

3. 03

   Quality recovery can lose specialist capabilities when adaptation data is mismatched. A useful advance would preserve independently tested coding and reasoning behavior without requiring complete reconstruction of the original multi-stage training mixture. [8]

4. 04

   Quality-qualified serving comparisons remain hard to transfer between workloads. Progress requires matched accuracy, latency tails, errors, memory peaks, and preparation costs—not a kernel result alone—to identify where compression improves usable capacity. [37][36]