Strategies for LLM Evals (GuideLLM, lm-eval-harness, OpenAI Evals Workshop) — Taylor Jordan Smith

What does it take to turn a convincing generative AI experiment into a scalable, reliable and safe production system? Taylor Jordan Smith, working in developer advocacy and technical marketing in Red Hat’s AI business unit, approaches that question through evaluation and hands-on benchmarking. The opening cartoon captures the gap: a boardroom proposes transforming the business with AI that has just been used to write a poem about a dog.

Enterprises usually cross that gap incrementally. They begin with automation or a chatbot, add retrieval-augmented generation, and then explore agents. Smith describes Red Hat customers as largely working through those early phases, rather than starting with a complex multi-agent framework. Each step introduces behavior that needs testing before the next layer becomes dependable.

The constraints extend beyond answer quality:

• Organizational policy: teams may be restricted in which services they can use. Smith gives the example of Gemini only recently becoming officially available inside Red Hat.
• Legal exposure and harmful advice: an answer can create a customer-facing incident, as the glue-in-pizza example will illustrate.
• Bias and discrimination: Smith points to Eurocentric and US-heavy public internet data as a source of skew that teams need to recognize and mitigate.
• Cost and performance: serving at scale requires acceptable throughput, latency and operating cost.
• Knowledge freshness: a model with an older training cutoff cannot supply newer information from its weights alone. Retrieval and agents can bring current information into the system, while adding components that also need evaluation.

A capable model still needs a serving system that can keep up. Smith’s inference diagram shows green, yellow and orange request dots converging on a bottleneck. A customer-support agent, developer copilot or RAG pipeline can encounter this problem as concurrent requests accumulate. The runtime must handle that traffic efficiently; Smith names TRT, SGLang and vLLM as relevant inference runtimes, with Red Hat focusing on vLLM.

Evaluating that capacity is itself expensive. Realistic runs require multiple parameter configurations, consume compute, and depend on datasets that work with the chosen model. Hardware sizing adds another dimension: the model must fit, but the deployment must also use the GPU investment efficiently. Cost estimation then requires connecting observed inference performance to the token workload the application will actually generate. A benchmark with an unrepresentative workload can therefore mislead both capacity planning and budgeting.

Performance is only one failure surface. Smith first uses Stable Diffusion to illustrate bias, then turns to the recommendation to put glue in pizza. In her account, a joke on Reddit entered an AI overview as advice because the system failed to recognize satire. The failure is especially relevant to retrieval-based applications: retrieving information does not establish that it is appropriate to repeat as an instruction.

A further concern is repeated training on generated material. Smith describes successive generations moving away from human-anchored data, potentially losing output diversity and precision. That motivates accuracy evaluations capable of detecting degradation over time; it should not be read as a claim that every use of synthetic data inevitably damages a model.

The goal is to find these problems before a model or application release. Smith discusses additional safeguards, including satire detection, retrieval-related mitigations and bias guardrails, but explicitly treats the details of closed-source implementations as speculative. The practical requirement is to turn known failure modes into tests and mitigations, rather than wait for a public incident to reveal them.

Benchmarking is a subset of evaluation. Evaluation can assess many components and behaviors across a system; benchmarking uses controlled tasks and datasets to make comparisons possible.

The workshop includes both comparative benchmarks and custom evaluations that do not need to be benchmarks. The distinction matters because a standardized score cannot cover every requirement of a particular application.

The stakes also change between a laptop experiment and a customer-facing deployment. Unexpected output in a private test is an opportunity to learn; the same behavior in production can harm customers and damage the organization’s credibility. Because no initial suite catches everything, evaluation needs a continuous-improvement process: expand and refine tests as the system and its observed failure modes change.

The architecture determines what to measure. A RAG application may use Ragas to assess retrieval and generation, while an agent needs checks for function and tool calling. Trying to evaluate every component and interaction immediately is costly, so Smith recommends starting with a bounded, high-priority surface. In the RAG example, that could be chunk retrieval, followed by a latency and throughput benchmark for the LLM. The highlighted retrieval and LLM components on the slide make those initial boundaries concrete.

From there, extend coverage using the familiar software testing pyramid:

1. Component tests: evaluate an individual retrieval step or model endpoint.
2. Integration tests: check whether the components work correctly together.
3. End-to-end tests: assess the experience exposed through the application’s UI.

This lets the framework grow alongside the system without making exhaustive coverage a prerequisite for the first useful test.

Smith then introduces a complementary pyramid for model evaluation. Its base is system performance: throughput, support for concurrent users and GPU utilization. Above that come formatting checks, such as consistently producing the JSON an application requires; factual-accuracy checks across subjects; and checks against information used in fine-tuning. Higher layers address safety, bias and application-specific behavior. These are different questions, so success at one layer does not answer the others.

The first exercise uses GuideLLM, a project associated with vLLM, to benchmark serving performance. The workflow starts by choosing a model and dataset, then inspecting throughput, time to first token and inter-token latency in its terminal interface. The intended application determines whether the results are acceptable.

Input and output token lengths are central workload parameters. A chatbot and a RAG application can place different demands on the same endpoint: retrieved context changes the input, while the expected answer changes the output. Adjusting those lengths lets the benchmark approximate the use case instead of treating every request as interchangeable.

The hosted workshop required email sign-in and a workshop password. Instructions appeared on the left, with two terminal sessions connected to the same RHEL machine on the right. Each machine had one L4 GPU, and tmux was available for additional terminal organization. Smith initially allotted roughly 15–20 minutes to the first of three activities, with discussion breaks planned between them. The opening page provided background and preparation instructions; the terminals were available through another tab.

The demonstration first prepares the serving endpoint:

1. Install the container toolkit missing from the workshop machine so the inference container can use the required hardware.
2. Obtain a Hugging Face token for model access. Smith suggests trying an incognito window when participants encounter trouble obtaining a token.
3. Launch the containerized vLLM runtime with vllm serve, using an IBM Granite model.
4. Allow time for model loading before proceeding to the benchmark.

Smith mentions local installation as another deployment option, but the exercise itself uses the container. She watches the green INFO startup messages to follow loading progress.

Smith contrasts vLLM’s support for safetensors with the model-format conversion required in the TRT workflow she describes, reporting fewer configuration steps and lower storage needs. The runtime also has its own tuning parameters. She gives reducing the maximum context window as an example of a configuration change that may improve performance in a suitable workload, though runtime tuning is outside this exercise. The benchmark workload and the serving configuration are therefore separate sets of variables to track.

With the endpoint running in the upper terminal, Smith activates the environment already provided by the workshop and installs GuideLLM. The lower terminal runs the client that generates load; the server must remain available throughout. These are the basic installation and option-discovery commands used in that workflow:

pip install guidellm
guidellm --help

The benchmark configuration supplies a target endpoint and selects a sweep. A sweep varies the offered load and measures performance at those loads; latency is an observed metric, not a request rate. Smith uses a setting of five to keep the demonstration short. The recording’s historical rate terminology differs from the current benchmark guide, which uses guidellm run, --profile and a separate sweep-size setting; the historical five should not be interpreted as five requests per second.

GuideLLM places benchmark information above the benchmark statistics, while the workshop instructions explain how to read the output. Smith identifies constant-rate rows of 3.63 and 6.93 requests per second in the demonstration. These describe offered request rates, not latency measurements or independently established production capacity. She explains that changing the workshop setting from five to ten would produce more rows at progressively increasing rates.

Whether the resulting performance is useful depends on the application and serving configuration. Smith corrects the model size to two billion parameters, running on one L4 GPU, and cautions against treating that workshop setup as sufficient for concurrent production traffic. The output includes mean, median and P99 performance. P99 captures the tail that matters when an SLO must account for slow requests rather than just the typical experience. Results can also be exported as JSON for closer inspection.

The next step is to change the input and output token parameters and compare runs—for example, to represent a RAG workload instead of the initial configuration. That comparison makes the exercise useful for planning: the question becomes how a specific workload change affects the distribution of performance, rather than whether a single number looks impressive.

Provisioning constraints change the workshop’s pacing at this point. Smith expects another 10–15 machines, drops the planned discussion breaks, and asks participants to progress independently through the remaining activities. An additional access URL initially offers three machines, with more provisioning and the same password. The initial availability estimate is noon or early afternoon; it is extended later in the session.

The second activity moves into the accuracy layer with MMLU-Pro; the third addresses safety, bias and custom evaluation. MMLU-Pro measures performance on reasoning-focused multiple-choice questions, so its score should not stand in for the factual reliability of an entire application. Participants can move between activities according to their interests.

A question about proprietary datasets leads to a useful distinction between a benchmark’s format and its data. An organization can build an MMLU-style multiple-choice evaluation using its own questions and answers. Smith describes a Red Hat product evaluation for models fine-tuned on proprietary data as a branch of MMLU, and points to modifying open-source evaluation code and changing its data sources as a way to support custom accuracy tests. This makes the evaluation relevant to the knowledge the fine-tuning was meant to add.

The hosted instructions do not include Smith’s latest changes to the third activity, so she supplies a repository link through the public event Slack channel workshop-beyond-benchmarks. The linked material begins at activity two and also provides activity three. Event emails and badge backs explain how to reach Slack, and the channel remains a place for follow-up questions about the tools.

In the wrap-up, Smith extends environment access to the end of the day and points participants to the repository’s reading material. She identifies LM Evaluation Harness as the framework used for MMLU-Pro, with other evaluation tasks available through the same framework. Smith reports that the workshop’s MMLU-Pro exercise took about ten minutes, despite being selected as the quickest option she considered. That duration belongs to this workshop run, not to MMLU-Pro as a fixed runtime.

The safety exercise uses promptfoo. Its custom tests and repository examples provide a starting point for evaluating application-specific behavior beyond standardized accuracy benchmarks. The progression now covers serving performance, multiple-choice accuracy and a safety-focused custom evaluation. Smith returns to the testing pyramid to show how these checks can become layers of a strategy spanning the whole system.

The final audience question connects prompt tests, dataset tests and model swaps to a harder concern: how do you check that performance with real customers matches the expectations established during evaluation? Smith frames the answer as integrating evaluation tests into CI/CD, much as a software project runs unit tests. That gives the evaluation suite a recurring role in delivery instead of leaving it as a one-time experiment. The answer establishes an automation pattern, but does not describe a mechanism for evaluating live customer traffic.

The workshop repository remains the continuing resource, with Slack available for questions after the session. Smith’s final estimate places the temporary environment shutdown around 5 PM that evening, correcting an initial mention of 6 PM. The lasting work is to carry the useful tests out of that temporary environment and into the application’s normal development and delivery cycle.