Engineering Better Evals: Scalable LLM Evaluation Pipelines That Work

What happens when a production application needs many evaluations for every trace? Dat Ngo introduces the problem from his work as an AI architect at Arize AI, observing teams such as Reddit and Duolingo. Dat reports that Duolingo runs about 20 evaluations per trace. In that example, running evaluations, understanding their results and optimizing their cost are substantial operational concerns; he does not specify the workload, sampling policy or evaluator mix.

That scale changes the engineering question. An evaluation pipeline must produce useful signals repeatedly and economically, while giving the people building the application enough visibility to act on them. Dat approaches this as an engineer working on observability and evaluations, rather than as a developer advocate.

Observability answers what the application is actually doing. Engineers may inspect traces and spans; an AI product manager may inspect conversations. Those are different views of the same application, and evaluations can operate at the conversation level as well as on individual calls.

Analytics also reflects organizational ownership. Dat describes a hub-and-spoke structure: a central LLM platform team owns infrastructure, including the model gateway or router, while business teams build applications around it.

The dashboard brings these concerns together, but the people using it need different views and different definitions of success.

Nobody can manually inspect every production trace. Evaluations turn that volume of activity into signals about what is going well and what is failing. An LLM judge is one way to produce those signals, but it is not the whole toolbox.

Consider retrieval-augmented generation: a user asks a question, the application retrieves context, and a model generates an answer. Each relationship offers something to evaluate. Retrieval relevance, for example, compares the retrieved context with the original query: would this material help answer that question? This isolates a retrieval problem before treating the final answer as one undifferentiated success or failure.

Judging an answer is also a different task from generating it. Writing a summary of War and Peace requires producing the summary; assessing an existing summary asks whether that candidate is good or bad. That distinction helps explain why an LLM can be useful as a judge without making its judgments automatically trustworthy.

Autoregressive generation is not required for every evaluation. Dat describes encoder-only BERT-type evaluators as about 10 times cheaper and 10–100 times faster than the larger-model alternative. He supplies no task, comparison model, hardware or quality threshold for those estimates, so they motivate testing a smaller evaluator rather than predicting its performance on a particular workload.

Human feedback supplies another kind of signal. Users can report whether an interaction was helpful, and developers or domain experts can label examples themselves. A golden dataset collects judgments the team trusts. Automated judges supply scale; human-graded examples provide a quality reference.

To connect the two, run the LLM judge on that golden dataset and compare its labels with the trusted labels. The question becomes concrete: can this evaluator approximate the judgments the team expects? Disagreements give you evidence for quantifying and tuning the judge, rather than accepting its first prompt as a reliable measuring instrument.

Some questions need neither a model nor a human. In the code-evaluation demonstration, Dat checks properties such as keyword containment, regex matching and whether an output parses as JSON. These are deterministic requirements, so ordinary code can capture the signal directly. A Python implementation of those checks looks like this:

import json
import re


def evaluate_output(text: str) -> dict[str, bool]:
    try:
        json.loads(text)
        parses_as_json = True
    except json.JSONDecodeError:
        parses_as_json = False

    return {
        "contains_hotel": "hotel" in text.casefold(),
        "has_booking_id": bool(re.search(r"\bBK-\d{4}\b", text)),
        "parses_as_json": parses_as_json,
    }


output = '{"hotel": "Harbor Inn", "booking_id": "BK-1042"}'
scores = evaluate_output(output)

These checks establish syntax and string properties, not whether the hotel exists or the booking is correct. Choose code, human labels or a model according to the question the evaluator must answer.

The application-improvement loop is familiar:

1. Collect traces and other observations.
2. Run evaluations to locate successes and failures.
3. Inspect failures to identify causes—for example, a hallucinated answer caused by a faulty retrieval strategy or agent behavior.
4. Annotate examples to check the evaluation results.
5. Change the prompt, model or agent orchestration, then repeat.

The evaluation is the instrument that tells you where to intervene.

The instrument needs its own improvement loop. A judge may flag a correct answer as a hallucination, or approve an answer that is wrong. Periodically annotate evaluated examples, collect those mistaken judgments, and revise the evaluation prompt. Instructions that initially seemed adequate may turn out to be vague, obscure or insufficiently specific as the application changes.

These loops reinforce each other: better evaluations make application changes easier to assess, and new application behavior exposes weaknesses in the evaluations. Dat emphasizes iteration speed, contrasting four cycles in a month with two. His description of exponential improvement is an argument for faster learning, not a measured growth rate.

Evaluation structure should follow application structure. Dat uses Booking.com’s Trip Planner as a travel-agent example, describing it as revenue-driving and discussing flights, hotels and itineraries. A routed application consists of multiple components, so its evaluation can be as complex as its orchestration.

Start with an individual component, such as one LLM call inside a trace. Then zoom out to the inputs and outputs of a whole workflow. In Dat’s travel example, that workflow combines LLM calls, API calls to find actual flights or hotel vacancies, and heuristics. The accompanying slide illustrates component composition with a separate Chat-to-Purchase Router: product comparison, product search, customer support and package tracking branches combine LLM calls, internal API calls and application code.

Check control flow before spending money evaluating its consequences. If the router sent a request down the wrong branch, downstream quality scores may add little diagnostic value. Conditional evaluation can first judge the routing decision and skip downstream checks when that decision fails. This makes the evaluator’s execution order reflect dependencies in the application.

Zoom out again and a session becomes a series of traces representing a back-and-forth conversation. A session-level evaluator can ask whether the customer became frustrated at any point, even when no single call captures the whole experience. There is no universal evaluation recipe: Dat recommends customizing evaluators heavily and first understanding how the application works.

Arize’s Copilot provides a concrete example of an agent built to observe, troubleshoot and create evaluations for other AI systems. Dat says it had been available for a year at the time of the talk. He frames it as an early step toward a future, five to ten years out, in which AI systems perform more of the work of evaluating other AI systems.

The demonstration moves into agent traces, which grow longer as agent calls become more involved. Dat opens a failed trace and explains how Copilot chooses specialized agents based on the user’s request and their current location in the platform. Each agent has its own set of tools. A trace lets an engineer inspect what happened in one execution, but the operational question is broader: which failure modes recur across the agent’s executions?

For an agent with ten available tools, useful questions include how often it calls each tool and how evaluation results vary by path. The aggregate agent graph is described as framework-agnostic, covering LangGraph, CrewAI and hand-written orchestration. Its purpose is to connect path frequency and path-specific evaluation results across many traces.

Dat’s dependency example makes the value concrete:

The second path eventually reaches component 3, but too late to satisfy component 4’s dependency. Looking across executions exposes an ordering problem that a single overall good-or-bad score would obscure. The target is the distribution of behavior, not just one instantiation.

A trajectory evaluation starts with a particular request and the components the agent should visit. For a Trip Planner hotel request, that might begin with a start agent followed by a tool agent. A golden trajectory dataset can record the expected path, or the three or four components an example must reach.

There are two straightforward ways to use that reference: give an LLM the actual and expected trajectories and ask it to grade the difference, or compare trajectory strings directly. The demonstrated strict comparison shows different actual and reference step sequences and returns False.

Exact sequence equality and required-step coverage answer different questions. For a simplified hotel request, this Python example makes the difference explicit:

request = "Find me hotels."
reference = ["start_agent", "tool_agent"]
actual = ["tool_agent", "start_agent"]

scores = {
    "strict_match": actual == reference,
    "required_steps_present": set(reference).issubset(actual),
}
# {"strict_match": False, "required_steps_present": True}

Both steps are present, but their order is wrong relative to the reference. Choose the comparison that expresses the application’s requirement; a presence check cannot enforce ordering.

An exact ground-truth path is not always necessary. Dat suggests judging whether the visited nodes and their descriptions match the expected process. He also describes an exploratory approach: provide the actual trajectory alongside possible paths, represented as nested key-value pairs, and ask a model to assess it. These alternatives move from literal sequence matching toward judging whether the path fulfills the intended process.

The first audience question turns from after-the-fact evaluation toward checks that can affect the current interaction. Dat prefers in orchestration versus out of orchestration to the less precise distinction between online and offline evaluation. An inline check can decide whether execution may continue; that makes it a guardrail, and puts its latency in the user’s path.

Think of two interacting systems. System one is the application: prompts, orchestration and other behavior. System two is the guardrail that checks it. The guardrail mitigates risk but adds work. Dat describes embedding-based guardrails as roughly 100 times lower-latency, without specifying the comparison system or benchmark conditions. A faster check can reduce the latency cost, but the second system still adds complexity.

When the guardrail repeatedly catches a problem, the first repair should often be in system one. Adjust the prompt or orchestration that produces the failure rather than immediately changing the protective check. Guardrails are fallible and resemble unit tests for known cases; observability and evaluations help discover behavior that was not anticipated before real users arrived.

The placement decision therefore depends on the application. An inline evaluator can prevent a bad continuation, while an evaluator outside orchestration can generate improvement signals without blocking the current response. Neither placement is universally correct; the question is which checks must influence execution now and which can inform the next iteration.

A second question asks how to evaluate a complex, long-running asynchronous system when spans, timeouts and service boundaries limit the view. Dat points to OpenTelemetry context propagation. As LLM applications spread across services, Docker containers and Kubernetes pods, observing one process is no longer enough.

His example follows application A → model router → application A → application B → application A. Propagating trace context lets those service-level observations remain connected, so the engineer can inspect the complete operation instead of disconnected fragments. That requires instrumentation to carry the context, or explicit injection and extraction where automatic propagation is unavailable.

Dat says Arize committed to an OpenTelemetry-first approach two and a half years earlier. The answer addresses cross-service visibility: propagation preserves the relationship between work performed in different places. It does not itself define timeout policy or determine when an asynchronous workflow is complete.

Asked how to obtain a confidence score for an evaluation, Dat starts with autoregressive judges. If the evaluator returns a single-token label and the provider exposes that token’s log probability, the value offers a pseudo-confidence signal. He names OpenAI as one provider with this capability; the OpenAI logprobs example documents the historical approach and is now archived, so availability must be checked for the chosen model and endpoint.

A token’s log probability measures how likely that token is under the model’s context. It is not a calibrated probability that the evaluation is correct. Encoder-only classifiers can instead expose classification probabilities. Dat recommends using these signals together with the rest of the evaluation toolbox to understand where the system is doing well or badly.

The final question asks how to reduce developer intervention between customer feedback and prompt improvement. Dat points to DSPy and its MIPRO optimizer. He illustrates the idea with roughly 30 input-output pairs and describes the goal as less fragile prompts across models from providers such as OpenAI and Google, including Gemini. The example count is illustrative, not a fixed requirement or a guarantee of transfer across models; current MIPROv2 uses configurable datasets and a task metric.

Dat then describes Arize’s meta-prompting approach, with prompt optimization available or being released at the time. The mechanism is to give an LLM the evidence needed to propose a repair:

1. Supply the original prompt.
2. Include a dataset of input-output pairs.
3. Attach evaluation results showing successes and failures.
4. Ask for a revised prompt that addresses the observed failures.

The output is a proposed prompt revision. This automates part of the improvement loop by turning examples and evaluation signals into a concrete change, rather than leaving a developer to translate every failure into new instructions manually.