Ship Real Agents: Hands-On Evals for Agentic Applications

You have already built an agent, and it sometimes goes off the rails. How do you find out what failed, decide what to measure and improve it without breaking behavior that previously worked? Laurie Voss’s workshop starts with that engineering problem: evaluate a supplied agent, rather than build another agent from scratch. The sequence is tracing, reading and categorizing failures, writing evaluations, checking the evaluators themselves, and finally running experiments.

The exercises use the Claude Agent SDK, with Claude powering both the application and its judges. Following along requires an Anthropic API key and a Phoenix Cloud account with its own API key. Phoenix is also open source and can run locally; the hosted version removes installation work. The companion slides provide the learner-notebook link. During setup, distinguish Arize Phoenix from Arize AX: the workshop uses Phoenix, while the main Arize signup flow can lead to the enterprise product. OpenAI and Gemini are alternatives, but their integrations are not demonstrated here.

Evals are tests; traces are execution logs. A trace contains nested spans: an LLM call, a tool call or an agent turn that contains other spans. Each span can record inputs, outputs, timing, token counts and other metadata. That structure lets you inspect both the final response and the operations that produced it.

Informal testing usually covers the vocabulary and questions the developer expects. Real users introduce unfamiliar phrasing, edge cases and adversarial inputs. Meanwhile, repeated runs of the same prompt can produce different answers that are all acceptable. Exact string matching cannot represent that whole space, and manually reviewing a few responses does not give you a repeatable CI check.

A prompt edit intended to improve tone can also make the agent invent product features. A faithfulness check can catch the new failure by comparing the response with its source material. The larger issue is that the model consumes the prompt as a whole: a local wording change can alter behavior across many inputs. Model upgrades create the same regression problem. Voss uses Sonnet 4.5 and Sonnet 4.6 as an example of why a working prompt must be retested against a new model, rather than assumed to transfer unchanged. He cites Descript, Bolt and Claude Code as teams that moved from informal checking to formal evaluations as they scaled.

A deterministic check is the cheapest useful starting point. Python or TypeScript can validate JSON, required fields, forbidden phrases, length limits or required mentions. Semantic questions need a different instrument: an LLM judge receives the response and a rubric defining how to grade it. That enables checks for factual accuracy, source faithfulness and customer-facing tone, at the cost of another model call and another prompt that can be wrong.

Humans provide labeled examples for a golden dataset, then inspect unfamiliar failures and judge disagreements. Voss warns that repetitive annotation causes fatigue; his suggested error percentage is not established here as a general rate. These methods complement one another: code handles mechanical rules, models handle meaning, and humans define and audit what success means.

An agent adds decisions between the request and the answer. Did it select an appropriate tool, pass valid arguments and interpret the result correctly? In a multi-agent system, did the router choose the right sub-agent, communicate the task, return the needed information and stop appropriately? An early error can contaminate otherwise competent downstream work. In Voss’s example, a request for an investment report on Tesla becomes research on Nikola Tesla; the writer then produces an investment case from information about the inventor.

The opposite also happens: the agent finds a valid route you did not anticipate. Requiring tool A, then tool B, then decision C can reject a faster correct solution. Evaluate the required outcome and meaningful constraints without fixing incidental execution order. A capability eval targets something the agent cannot yet do reliably; once the behavior is established, preserve it as a regression eval and move development to the next capability.

An evaluation result typically contains a score and a human-readable label. An explanation adds the diagnostic detail needed to act. LLM judges can generate that explanation; code evaluators can also return one explicitly, as the companion ticker-check example does. Consider the budget-travel example: the response recommends places in Tokyo, but omits a budget breakdown and cost estimates. It answers the destination question while missing the budget requirement. The explanation identifies that distinction.

Across many traces, explanations reveal recurring defects rather than isolated oddities. If their volume becomes unmanageable, another LLM can categorize them for human inspection. The resulting loop is concrete: capture execution, evaluate it, attach annotations, analyze patterns, change the application and run it again. A score becomes useful when it points toward a change you can test.

Phoenix stores traces and evaluation results, exposes them for inspection and supports experiments over datasets. The notebook installs the Claude Agent SDK and its OpenInference instrumentation package. With credentials and the collector configured, registration names the project and enables installed instrumentation:

from phoenix.otel import register

tracer_provider = register(
    project_name="financial-agent",
    auto_instrument=True,
)

Voss loads credentials through Colab Secrets. The demonstration uses a simple span processor; production traffic calls for batching spans. Anthropic is imported separately for judge calls. Similar instrumentation integrations exist for other agent frameworks, so the evaluation workflow does not depend on choosing this particular SDK.

The live setup questions expose two practical details: initialize or select a Phoenix Space and launch it before looking for the relevant settings, and use the collector configuration for that actual Space. The spoken hostname is shorthand; the companion slides expand it to a Phoenix Cloud Space address. It is not a literal endpoint to copy with yourusername unchanged.

The financial agent takes stock tickers and a focus area. Its first turn researches the companies; its second writes a concise report. Both turns use the same Claude SDK client, which carries conversation context from research into writing. OpenTelemetry supplies the tracing foundation, while OpenInference adds LLM-specific attributes such as prompts, completions, model names, token counts and invoked tools.

The application uses Claude Haiku, chosen to make inexpensive, quick runs that still produce instructive mistakes. Its intended research tool is web search. Voss also sets permission_mode="acceptEdits" to avoid permission prompts—a choice that later matters when the agent tries to write files. The async financialReport function encloses both turns in a parent OpenTelemetry span, keeping research and report generation inside one trace instead of presenting them as unrelated agent turns.

The first request asks about Tesla’s financial performance and growth outlook. The agent searches, assesses what it found and searches again before writing. A test that expects exactly one search—or exactly five—would encode an implementation guess. The resulting report contains Q4 2025 highlights, a 2026 outlook, growth drivers and risks. Its polished structure makes it look plausible, but that appearance is precisely what the evaluations must look beyond.

Phoenix’s trace tree exposes the overall request and response, then the operations underneath. The Tesla example includes searches about financial performance, future growth, quarterly revenue and margins, and vehicle production and demand. Opening a span exposes its messages, model, token counts, cost and annotations. These are the facts needed to distinguish an expensive route, a tool failure and an unsupported answer.

The notebook adds twelve queries to the initial Tesla run. They vary both companies and tasks: financial performance, revenue growth, competitive landscape and AI strategy. An Apple–Microsoft comparison requires separate research without mixing the companies; Coca-Cola’s KO ticker broadens the set beyond technology growth stocks. The purpose is representative variation, not a large count of near-duplicate questions. Voss recommends regularly reading the request, the result and the specific failure, rather than treating trace inspection as a one-time setup step.

Before labeling failures, define success. For this agent, a useful report must support an investment decision, including forward-looking analysis and an actionable recommendation. Thoroughness alone is not enough. Domain experts, product managers and actual users should help define these requirements because an engineer’s plausible-looking report may be useless to its intended reader.

When real usage is unavailable, an LLM can generate synthetic requests that exercise the same intent through different wording: formal Tesla financial analysis, a casual stock update or a direct question about whether Tesla is a buy. Include nonexistent tickers, multipart questions and adversarial requests as well. These inputs generate real executions of the agent, but the workshop runs are generated examples, not evidence of real-user production traffic. Aim for production-shaped inputs; use synthetic data when exposing users to an unfinished system would be unsafe.

The pregenerated Apple trace contains the workshop’s clearest failure. The agent researches the company, then interprets the request to write a report as an instruction to create a Markdown file. It invokes Write inside a notebook environment where the operation fails. The prompt specified the deliverable but omitted the destination: the report should have been returned in the response. Without the trace, the poor final output could easily be mistaken for a research or writing failure.

A separate defect is semantic: the Apple and Nvidia reports can summarize financial information without saying whether the reader should buy the stock. Nvidia returns its output in the conversation, avoiding the delivery problem, but still lacks the requested investment decision. Repeated file-write failures across other traces establish a pattern. The fixes differ: one concerns output destination; the other concerns the content that makes a report useful.

The Rivian report raises a source-verification question around a specific vehicle-delivery range. Voss suspects unsupported precision, but suspicion alone does not establish hallucination. His premise that Rivian was not public is incorrect: its shareholder letter records its 2021 listing. The useful diagnostic question remains whether the report’s claim is supported by the retrieved source, and whether it preserves distinctions such as guidance versus completed deliveries.

Categorize root causes rather than collecting a pile of bad answers: possible hallucinations, reasoning gaps, unverifiable data, missing recommendations and delivery failures point toward different changes. Phoenix’s APIs can support LLM-assisted annotation and grouping. Prioritize using severity × frequency, not frequency alone. A rare response that abandons the task and emits Moby-Dick may deserve attention before a common minor defect.

The Swiss cheese model from safety engineering gives evaluation layers a useful role: each has blind spots, and different layers can catch different failures. Voss borrows the illustration from Anthropic’s Demystifying evals for AI agents. A code check catches basic output defects, an LLM judge examines meaning, and human review investigates cases the automated layers miss. Layering improves coverage without making any layer infallible.

The first code eval asks whether the report mentions the requested stock ticker. Select root spans—those without a parent—so the check evaluates whole agent outputs rather than individual tool calls. The notebook wraps its Python function with create_evaluator, gives it a name and sets kind="code". Its regex extracts uppercase ticker candidates from the request and excludes terms that are not tickers. The essential output check can be written directly:

import re

def check_tickers(tickers: list[str], output: str) -> dict:
    missing = [
        ticker
        for ticker in tickers
        if re.search(rf"\b{re.escape(ticker)}\b", output) is None
    ]
    return {
        "score": 0.0 if missing else 1.0,
        "label": "missing" if missing else "present",
        "explanation": (
            f"Missing tickers: {', '.join(missing)}"
            if missing
            else "All requested tickers appear in the report."
        ),
    }

This checks a narrow requirement. Mentioning a ticker does not establish that the surrounding report is correct.

Run the evaluator over the parent-span DataFrame with evaluate_dataframe. The notebook uses suppressed tracing around evaluation work so instrumented judge calls do not become mixed with application traces. The ticker check passes 11 of 13 demonstration runs. Inspection explains the two failures: Tesla is affected by an attempted disk write, while Amazon’s report narrows the subject to AWS and omits the expected company ticker.

Code evals can go beyond strings: parse JSON, enforce length limits, query a pricing database or compare a claimed stock price with an appropriate reference. Reproducibility requires fixing the reference conditions when external data changes. Be flexible about equivalent answers—two hours and 120 minutes can satisfy the same requirement—and avoid requiring an incidental tool sequence merely because that is how you expected the agent to solve the task.

An LLM eval separates three pieces: the judge model, the prompt or rubric, and the examples being graded. Phoenix’s built-in correctness evaluator checks factual accuracy, completeness and logical consistency; its prompt is inspectable. Other built-ins cover tool selection, tool arguments, document relevance and refusal detection. The notebook evaluates the parent spans and sends the resulting annotations back using log_span_annotations_dataframe. Claude Sonnet judges the Haiku outputs, following the heuristic of choosing a more capable judge when one is available.

Every correctness result is zero. Reading the explanations changes the diagnosis: the judge objects to specific 2026 financial information. Voss attributes this to the judge relying on older knowledge and treating the figures as unavailable future facts. The demonstrated evaluator has not been given the evidence needed to assess current financial claims, so its apparent certainty is not a useful correctness signal for this task.

The two-turn architecture now pays off. Extract the first turn’s research output from the child spans and put it in a context column alongside input and output. The faithfulness evaluator can then ask whether the final report follows the supplied research, instead of judging current facts from model memory. Run it over the same parent reports with that additional context.

These are different signals on the same examples, not a before-and-after improvement to the agent. Faithfulness establishes source adherence, not independent truth of the research. Phoenix then lets you sort scores, filter labels and open failing traces; those operations become increasingly useful as the dataset grows.

A built-in evaluator cannot know this product’s definition of an actionable investment report. The custom rubric needs five parts:

1. Role and purpose: explain the financial domain and what the report should help its reader decide.
2. Observable criteria: require specific recommendations and forward-looking interpretation; identify historical summaries without interpretation as failures.
3. Data boundaries: label the user query and report separately, using clear delimiters.
4. Labeled examples: show both a useful investment directive and a merely descriptive company overview.
5. Constrained output: choose explicit categories the application can interpret.

The criteria come from defects already seen in traces. The positive example includes numerical evidence, a concrete risk and a recommendation to accumulate below a price. The negative example describes Nvidia’s industry position without helping the reader decide what to do.

Prefer actionable and not_actionable to an unanchored score from one to ten. A third category can work if its meaning is explicit, but a six-versus-seven distinction adds noise unless the rubric defines it. Voss also recommends asking for an explanation before the final classification; the machine-readable label should remain constrained even when a separate rationale is requested.

The prompt structure translates directly into a Python template. This compact version uses the same distinction between an investment directive and a descriptive summary:

ACTIONABILITY_TEMPLATE = """
You evaluate whether a financial report supports an investment decision.

Label actionable when the report gives a specific investment
recommendation supported by forward-looking analysis and relevant risks.
Label not_actionable when it only summarizes facts or describes the
company without an investment directive.

Example: actionable
Report: Hold rather than add to the position until the next earnings
release clarifies margin pressure. Reconsider if profitability improves.

Example: not_actionable
Report: Nvidia is a major semiconductor company with a strong market position.

BEGIN DATA
User query:
{input}

Financial report:
{output}
END DATA

Return exactly one label: actionable or not_actionable.
"""

def build_actionability_prompt(query: str, report: str) -> str:
    return ACTIONABILITY_TEMPLATE.format(input=query, output=report)

The notebook’s classification evaluator takes a name, this kind of prompt template, an LLM and a label-to-score mapping: actionable becomes 1.0, not actionable becomes 0.0. An audience member notices that the demonstrated prompt omits the earlier recommendation to request reasoning, and Voss acknowledges the omission.

The initial results contain both passing and failing reports, leaving a capability to improve. Logging annotations back to Phoenix makes the failures easy to isolate. Their explanations point to missing buy/sell/hold recommendations, even when the reports contain strong financial data and forward-looking analysis. You can also filter by latency, token count and cost: an agent that eventually answers correctly after excessive searching may still be unsuitable for production.

Treat evaluator prompts like code. Store their versions, compare their behavior on known examples and keep a path back to an earlier version. A wording change can materially alter classifications. Until a judge has been checked against trustworthy labels, running it across more data only scales an uncertain decision rule.

Avoid a single evaluator that simultaneously grades accuracy, tone, completeness, policy compliance and formatting. A failure becomes difficult to diagnose and the rubric difficult to calibrate. Separate meaningful dimensions, then distinguish their operational roles: an invented stock price or reversed investment recommendation can block a release, while suggestions for complementary investments may be an improvement goal rather than a mandatory gate.

Meta-evaluation treats the judge as a classifier whose predictions must be checked. Add human annotations through Phoenix’s UI or API, then compare them with the model’s labels. The human reviewer needs the same concrete criteria and examples as the model. Arbitrary human approval does not become ground truth simply because a person supplied it.

Reference solutions help expose tasks that are ambiguous or impossible with the available information. An evaluator that always fails warrants inspection of the task as well as the agent. Include cases where a behavior should occur and where it should not: rewarding every web search can produce an agent that searches unnecessarily just to satisfy the test. A golden set should capture expert judgment about the requirement, not reward a superficial proxy.

As production failures become new examples, protect against overfitting the judge prompt to the cases you repeatedly inspect. Voss suggests a 75/25 development–held-out split. Use development examples to refine the rubric, and reserve held-out cases to check whether the rule generalizes beyond those examples.

The notebook retrieves annotated examples with a span query and renames attributes.input.value and attributes.output.value to the input and output fields expected by the evaluator. Its comparison shows two disagreements among six examples, but Voss explicitly says the demonstration’s human labels were assigned roughly at random. The table demonstrates comparison mechanics; it does not validate the actionability judge.

To interpret disagreements, define the positive class first. For an evaluator whose job is to detect failures, positive means failure detected.

Precision asks how often a flag is justified. Recall asks how many real failures are caught. The spam example emphasizes avoiding false positives that hide legitimate messages; the cancer-screening example emphasizes avoiding missed cases. The appropriate trade-off follows the error costs, not a universal preference for one metric.

The toy comparison reports 100% precision and poor recall against those roughly random labels. That result is too small and too weakly labeled to establish judge quality. Voss suggests a larger, carefully annotated golden set and generally favors recall for failure detection: a false alarm creates review work, while a missed failure can reach a user. The metric must still match the application’s consequences.

LLM judges can favor an answer because of its position, length, confident wording or similarity to the judge’s own output. Using a different model or provider is one mitigation to test, not a substitute for calibration. Inspect performance by response length and query category to find systematic preferences. Compare model agreement with human agreement on the same task; the workshop’s companion slide names Cohen’s kappa, which is not a raw agreement percentage. Its suggested typical values should not be treated as universal acceptance thresholds.

A failure should make sense when you inspect the answer and the requirement. Voss illustrates this with Claude Opus 4.5 on CORE-Bench. Anthropic reports a rise from 42% to 95% after grading fixes and a less constrained scaffold, not grading fixes alone. Its published rounding example penalized 96.12 when the grader expected 96.124991…; the spoken example reverses those values. The engineering lesson is the same: numerical tolerance and task constraints can make a capable answer appear wrong. Read the failing examples and check the evaluator before attributing every low score to the model.

Once a failure is understood, a prompt edit needs a repeatable comparison. In Phoenix, select relevant traces and choose Add to Dataset to create or extend a dataset. The workshop collects six actionability failures into a focused set. That makes iteration faster than rerunning the entire corpus after every edit.

The changes follow the explanations. The research prompt now asks for financial ratios, current price data and recent news, including news from the last six months. The writing prompt explicitly demands a buy/sell/hold recommendation. Each instruction addresses a missing ingredient identified by the evaluator, rather than an intuition about what might make the prompt better.

Wrap the improved financial agent in an experiment task, retrieve the failure dataset and reuse the actionability template. The notebook runs the experiment through AsyncClient. All six selected, previously failing examples pass after the demonstrated prompt changes. This is a result on the selected failure subset, not a verified full-suite improvement rate. Voss contrasts the dramatic demonstration with production work, where gains are often incremental and some prompt edits require rollback.

An experiment task can also isolate one component, such as tool selection, instead of rerunning the entire agent. Keep inputs and evaluators constant to make comparisons more informative, and repeat trials to account for nondeterminism. A live web search can still return different evidence between runs unless retrieval is frozen or replayed; fixed queries alone do not remove that variation.

The same loop suggests an extension: send evaluation feedback to Claude Code and ask it to propose application changes, then evaluate those changes. Voss describes this as closed-loop evaluation. The workshop demonstrates human-directed iteration, not an autonomous system that reliably improves its own software.

Suppose the release target is a defect rate below 5%, and the observed rate is 3%. The sample size determines how much uncertainty remains around that estimate. Voss’s example uses the rounded, two-sided 95% normal-approximation intervals below.

The larger sample appears to put the whole interval below the release threshold. But that conclusion depends on the interval method: NIST’s guidance warns about the symmetric approximation when failures are few, and a Wilson interval for the second case still extends above 5%. A release gate therefore needs an agreed statistical method and representative sampling, not merely a convenient rounded bound.

More examples also mean more execution, grading and annotation effort. Voss suggests 12–20 examples for directional workshop feedback and 200–400 as a starting target for shipping decisions. Those are planning heuristics; the required assurance depends on the acceptable defect rate, the consequence of a miss and how well the sample represents actual use.

Voss’s impact hierarchy orders the next round of work:

1. Data quality: fix stale knowledge, wrong sources and missing evidence.
2. Prompting: add useful examples, explicit requirements and constraints.
3. Model selection: test whether additional capability justifies additional cost.
4. Hyperparameters: investigate settings such as temperature and top_p after the larger causes.

Prompt engineering cannot recover facts the system never receives. The hierarchy keeps attention on the source of the observed failure before reaching for a more expensive model or a parameter adjustment.

You can also write a capability eval before implementing a feature. A refund agent, for example, must verify customer identity before processing the refund; that ordering is a genuine requirement, unlike an incidental research-tool sequence. Product managers, customer-success teams and other nontechnical stakeholders can specify such requirements. They can contribute to code evals too by identifying content that must appear in every answer.

The data flywheel is the accumulated result: log, sample, review, improve and repeat. Expert judgments enlarge the golden dataset, production failures become test cases and the evaluation suite captures domain knowledge specific to the product. That history also makes model adoption easier: a regression suite can reveal whether a new model improves the application’s actual tasks instead of forcing another round of informal testing.

After release, evaluate a sample of live traffic to detect quality drops, adversarial use and changes in the product’s workload. Those changes can break an agent even without a model upgrade. Voss presents continuous monitoring as an enterprise emphasis, extending the offline iteration workflow into ongoing observation.

Model routing introduces another measurable trade-off. Answering business hours does not require the same capability as comparing financial ratios across several semiconductor companies. Cost-normalized accuracy—accuracy divided by cost—makes the economics visible. Voss’s illustrative comparison is 92% accuracy at $0.02 per query versus 95% at $0.15 per query. The cheaper option may provide better value, but the application still needs to decide whether its remaining errors are acceptable.

For prompt or model comparisons, pairwise evaluation asks which of two concrete outputs better satisfies the requirement. That can be easier to calibrate than an undefined one-to-ten score. Repeated-trial evaluation asks a different question:

Under independent attempts with the same success probability p, the distinction is:

For 0 < p < 1, increasing K drives these quantities in opposite directions. The assumptions matter: correlated attempts need not behave like independent trials. Multi-judge systems add another layer by collecting different opinions or checking claims with external evidence.

The starting investment is modest: read real outputs, write one format or required-string check, and build the suite incrementally. As a capability becomes reliable, retain its test as a regression check. Evals become infrastructure when informal checking slows improvement or a change fixes one issue while silently breaking others. Catching a regression before it reaches users supplies a concrete return on that investment.

Voss closes the presentation by pointing to Phoenix documentation and open-source contribution, then outlines AX’s enterprise positioning: compliance controls, SAML/SSO, multi-team operation, Arize DB for large trace volumes, session-aware tracing, the Alyx assistant, graphical views, dashboards and monitoring. These are the deployment distinctions he presents, rather than prerequisites for the notebook workflow.

The first audience question clarifies execution location. In the demonstrated Phoenix workflow, code and LLM evaluations run on the client and send annotations to the server; Voss contrasts that with AX’s platform-side execution. Current Phoenix documentation also lists server-side evaluations, so the workshop distinction should be read historically, not as a permanent product boundary. A follow-up question about cheaper Anthropic batch APIs is deferred and receives no on-record implementation answer.

An attendee working on architectural compliance describes using Claude Code to scope checks when labeled data is scarce. Repeating the scoping process and comparing consistency offers a signal about the process, although consistency alone does not establish correctness. Voss suggests evaluating alternative agent configurations and testing prompt variants against the same evaluator. This extends evaluation to how a solution is proposed, not only to its final output.

Another attendee asks when evaluation spending becomes excessive—even more expensive than the feature itself. Voss distinguishes maintaining established behavior from developing a new capability. Redundant regression cases can be reduced to a representative set or graded more cheaply, while the capability under active development deserves stronger evaluation. His suggestion to discard a large fraction of a hypothetical regression suite is about redundancy, not a universal pruning rule. For unchanged live behavior, he emphasizes regression monitoring rather than repeatedly paying for a capability-development exercise.

The actionability rubric raises a related granularity question: should its positive and negative criteria become separate checks? The answer depends on what the stakeholder actually requires. If every report must contain a P/E ratio, that deserves its own eval. If the requirement is an investment recommendation, P/E may be supporting evidence rather than a mandatory field. Grouped criteria can represent alternative acceptable ways to reach the required outcome; independent obligations should remain independently measurable.

When a score changes, the defect could be in the agent prompt, the evaluator rubric or the human annotations. Build confidence in those layers incrementally: validate a code eval, then one LLM judge, and work on one capability while trusted regression checks remain stable. Introducing several uncertain judges at once makes it harder to identify which measurement changed and why.

A rubric or prompt edit does not require rerunning the whole corpus during every exploratory step. Use a small failure subset for rapid iteration. When improvement slows, when the work reaches a stopping point, and especially before shipping, rerun the full dataset to look for overfitting and regressions outside the subset. A focused experiment helps you find a fix; broader validation tells you what else the fix changed.

The final question returns to autonomous optimization. An attendee describes using DSPy and mentions Karpathy’s autoresearch, while finding that automated improvement reaches a plateau that still needs manual refinement. Voss agrees that reliable, autonomous, self-improving software remains difficult frontier work. He hopes to show further progress at AI Engineer World’s Fair, but future model improvements are a possibility, not a capability established by this workshop. The working system demonstrated here is a disciplined feedback loop whose measurements—and proposed fixes—still need to earn trust.