Are LLM Performance Benchmarks Reliable? — Ashok Chandrasekar & Jason Kramberger, Google

A benchmark usually starts with a small set of inputs: a model, a number of prompts, input and output lengths, and a requested load. It returns input-token throughput, output-token throughput, time to first token, and time per output token. Ashok Chandrasekar and Jason Kramberger, Google engineers working on inference performance and co-maintaining InferencePerf, kept encountering a problem with these reassuring tables: they often could not reproduce other people’s results. The numbers could look plausible even when the harness had failed to run the intended experiment.

The benchmark ecosystem contains tools with different jobs:

• Model-server benchmarks: Frameworks such as vLLM and SGLang include developer-oriented benchmarks, commonly Python scripts, for measuring their servers.
• Competitive comparisons: MLPerf and Artificial Analysis are examples of tools used to compare chips and accelerators.
• HTTP load tests: Locust and Grafana k6 focus on generating web traffic at high scale.
• Production inference benchmarks: The focus here is the complete inference-serving stack, including the many servers and serving mechanisms working together.

A tool that measures one model server successfully may struggle to drive an entire fleet. The benchmark harness—the client that generates requests and collects responses—has its own capacity limits. Those limits affect both the traffic reaching the server and the measurements coming back.

The production example is llm-d, a distributed inference framework. Online serving and batch workloads can run against an inference pool containing many model servers. Prefill/decode disaggregation and workload autoscaling add further complexity. A harness must generate enough traffic to exercise that larger system while preserving the characteristics of the workload customers will actually run.

One queries-per-second figure cannot identify the best operating point. A sweep through several loads lets a team compare a baseline with an optimized configuration and find where the server saturates. The aim is to use capacity efficiently and save costs, with accurate measurements at each load. 3:50

Latency objectives constrain that choice. A P90 time-to-first-token service-level objective asks whether the ninetieth-percentile wait for the first token stays within the target. A configuration may complete plenty of work while failing that objective. The sweep therefore needs to show both throughput and the latency experienced at each load.

Four recurring problems undermine these comparisons: inaccurate metrics, little visibility into the harness itself, runs that cannot be reproduced, and unsuitable datasets. Before interpreting a server’s saturation curve, the team needs to know whether the client maintained the requested load and whether the requests remained comparable across runs.

Chandrasekar’s experiment makes the client problem concrete. A harness was asked to generate 200 queries per second, but on a small machine it delivered only 38 QPS. Moving to a much more powerful machine increased the achievable load, yet some single-process harnesses still capped near 170 QPS. The requested rate remained 200; changing the client machine changed the traffic the server actually received. 5:24

The explanation centers on Python’s global interpreter lock in the CPU-bound, single-process harnesses under discussion. CPU work within that process can be limited by one CPU even when the machine has several available. A larger machine does not necessarily let the process use all that capacity. Distributing the work across processes addresses this limit.

The damaging part is the silent shortfall. The harness can finish and print its results without warning that it missed the requested rate: “I ran it. These are the numbers.” A reader may interpret the table as the server’s behavior under 200 QPS, although the server received much less traffic. The configured load and the measured response no longer describe the same experiment.

The client can also manufacture a latency problem. Collecting many streaming token responses can overwhelm the harness and delay its handling of those streams. One test showed client-induced delay of up to 58 seconds, enough to make a healthy server appear bottlenecked. In a 1,000 QPS test against a simulated server intended to introduce no serving latency, a harness that could scale out instead showed very little latency. The simulated endpoint helps isolate delay introduced by the measurement client. 6:33

Another shared benchmark claimed 20% better throughput. Inspection revealed that its harness had set temperature to zero, producing more deterministic outputs than the workload Chandrasekar was comparing it with, whose temperature was around 0.7. That advantage belonged to the differing test conditions. The example does not establish a general throughput penalty for nonzero temperature or isolate which part of generation caused the difference. 7:19

Even “same dataset” can conceal different requests. Two harnesses using the same ChatGPT dataset produced different input tokens because they sampled and truncated the data differently. Naming the dataset identified the source material, but the preparation rules determined what the model actually processed.

Several workload decisions need to be explicit:

• Generation stopping: Whether the harness forces generation through to the end of the sequence affects the work requested.
• Prefix-cache behavior: Prefix-cache rates help characterize the workload and the reuse it exercises.
• Multi-turn replay: Conversation turns require a replay strategy; a collection of isolated prompts does not fully describe that scenario.
• Sampling and truncation: Rules for selecting and shortening examples must accompany the dataset.

These failures invite the wrong diagnosis. Low throughput may come from insufficient client capacity, high latency from an overwhelmed stream collector, and an apparent improvement from changed generation settings. Server measurements become useful when the experiment also exposes these possible client and workload causes.

Kramberger introduces InferencePerf as a community benchmarking project developed out of Kubernetes Working Group Serving. Its design combines declarative configuration, a multiprocess load generator, and client metrics reported alongside server metrics. The configuration describes the intended experiment; the telemetry helps determine whether the client carried it out. 8:46

The main process queues requests according to their planned execution times. The plan can follow a Poisson process, use a constant request rate, or maintain a constant number of concurrent requests. These options express different demands: a schedule of arriving requests or a target number of requests in flight. Multiple worker processes pull from the queue and execute the requests, spreading client work across processes. 9:52

How does the architecture expose a client that falls behind? The diagram traces two paths: requests move from the plan through the queue to workers, while execution timing joins server metrics in the reporting path. Workers observe when requests actually execute relative to their planned times. That comparison makes scheduling delay visible rather than letting it disappear into an apparently valid server result.

This supplies the missing observation in the 200-QPS example. The requested rate becomes a plan whose execution can be checked, instead of a setting readers must take on trust. In the comparison Kramberger presents, InferencePerf kept up at 5,000 QPS and reported that it kept up. That is a demonstrated result under the comparison’s conditions, rather than a capacity guarantee for every workload and client machine. 10:34

A minimal InferencePerf configuration can run a random dataset against an endpoint. More detailed configurations describe conversation replay, including input and output lengths and their distributions. Those distributions preserve variation in request sizes instead of reducing the workload to one fixed length. The extra controls serve a practical purpose: representing the traffic the inference stack is supposed to handle. 11:00

The published workload catalog extends this idea beyond a single configuration file. Its examples include multi-turn generation, tree of thought, agentic generation, and batch summarization. Each has a natural-language scenario definition and detailed configuration information. Generic descriptions accompany InferencePerf-specific configuration so other tools can express the same scenarios. 11:44

This addresses the earlier dataset mismatch at the level where it occurred. A dataset names the source material; a workload definition explains how that material becomes requests. Cross-tool comparisons need agreement about the scenario, sampling, and preparation. Giving two harnesses the same data does not by itself make them perform the same work.

Prism, part of llm-d, provides a UI for sharing workloads and benchmark results. The closing example uses agentic code generation on TPUs with eight replicas. It compares combined optimizations with a baseline consisting of a simple Kubernetes service in front of multiple model-server replicas. 12:50

The optimized configuration achieves substantially higher throughput in the presented comparison, described as approaching hundreds of thousands of tokens per second. The explanation does not specify the optimization bundle or precise curve values, so the result illustrates a production-scale comparison rather than a recipe for reproducing that gain. Its role is to complete the progression: define the workload, generate observable load, compare serving configurations, and share the result.

The closing principles turn the earlier failures into requirements for a valid benchmark:

• Client concurrency: The harness needs enough parallel execution capacity to generate production-scale load.
• Client observability: Measurements must show whether it met the configured demand, including planned-versus-actual execution behavior.
• Metric fidelity: Client and server measurements together help determine whether the scenario ran as intended and where delays occurred.
• Representative randomness and data: Deterministic settings, stochastic variables, and datasets must reflect the workload being tested. Making a run easier to repeat is useful only if it still represents the intended demand. 14:02

InferencePerf supplies the load generation and measurement tool; llm-d supplies the production inference context; Prism shares benchmark results and workload definitions. The practical question behind all three is whether a reported number describes the traffic you intended to measure. Delivered load, client behavior, and workload preparation make that question answerable.