From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI

An AI feature answers three queries convincingly, so the team ships it. Then an unfamiliar input breaks it. Laurie Voss, Head of Developer Relations at Arize AI, opens this workshop with the gap between that familiar development habit and a repeatable test suite. “Three times is not a test suite.” The problem becomes especially visible when a prompt change improves tone while also making the application invent product features: a local improvement can change behavior across many other inputs.

The workshop uses a Colab notebook, the Claude Agent SDK, and Arize AX. AX collects execution traces, stores evaluation results, and supports production monitoring. The setup needs an Anthropic API key for generation and judging, an Arize API key to authenticate telemetry, and an Arize space ID to select the workspace. AX and the open-source Phoenix project share core observability ideas, but this exercise uses AX.

The first useful distinction is between recording behavior and judging it. A trace records an execution as a tree of spans. An LLM invocation, a tool call, and an enclosing agent turn can each be spans; a parent span contains the steps it coordinates. Each span records inputs, outputs, timing, token counts, and other metadata. “Traces tell you what happened. Evals tell you whether it was any good.”

An evaluation needs a definition of acceptable behavior that survives changes in wording. Two reports can answer the same question correctly without sharing an expected output string. That makes whole-response string assertions a poor general test, while leaving plenty of room for deterministic checks on specific properties. A useful suite combines three approaches:

• Code evaluations: Check properties such as valid JSON, required fields, length limits, or forbidden phrases. They are reproducible and inexpensive, but overly specific checks can reject valid variations.
• LLM judges: Apply a rubric to meaning, including factual accuracy, faithfulness to sources, or appropriate tone. They cost money and can make inconsistent or incorrect judgments.
• Human review: Use domain expertise to identify unfamiliar failures and calibrate automated judges. It takes time, and fatigue makes human labels fallible too.

An agent adds decisions between model calls: which tool to use, what arguments to supply, whether the returned information is relevant, and when to stop. Multi-agent systems add routing and handoffs. A plausible final answer can conceal a mistake much earlier in the chain. Voss illustrates this with a Tesla query that retrieves information about the inventor rather than the car company. Search succeeds, writing succeeds, and formatting succeeds; the report still answers the wrong question because the retrieved entity never matched the user's intent.

The reverse problem is a grader rejecting a valid solution. In Voss's account of τ²-bench, an agent was expected to be unable to reschedule an economy flight. The supplied policies allowed it to upgrade the ticket to first class, where rescheduling was permitted. The agent found that route and failed the expected-result test. This is why an evaluator should usually check the requested outcome and applicable constraints rather than insist on the designer's imagined sequence of tool calls.

Evaluation suites also change purpose as an agent improves. A capability evaluation asks whether the system can perform a task it has not yet mastered; a low score supplies a hill to climb. Once that behavior works reliably, the same task becomes a regression evaluation, protecting an ability the product already depends on. A growing suite records both the next improvement target and the behaviors that must survive it.

Results need to support debugging as well as comparison. The workshop uses a score and a readable label, often a binary zero or one. LLM judges also return explanations. A travel response that fails because it omits cost estimates gives a much more useful repair target than a bare failure score. If the same omission appears across fifty traces, the repeated explanation points toward a systematic problem. Those explanations can later guide a coding agent's changes.

Instrumentation separates collection from analysis. The OpenInference instrumentation package captures Claude Agent SDK activity. The registration call configures OpenTelemetry export to AX using the space ID, API key, and project name. OpenInference adds AI-specific attributes such as prompt and completion text, model identity, token counts, and invoked tools. The SDK instrumenter then captures model and tool calls as spans. A separate Arize client reads the collected data back into the notebook.

The notebook sets batch=False so spans arrive promptly while participants inspect a run. Batching is useful for efficient production export, but waiting for a batch would make this interactive exercise harder to follow. Voss also introduces Arize Skills, which can help coding agents instrument applications, run evaluations, and create datasets; the manual setup teaches what those automated operations actually do.

The financial analyst accepts a stock ticker and a focus area. Its first turn researches current information through web tools; its second compiles that research into a report. Claude Haiku supplies a fast, inexpensive worker, and the deliberately basic research and writing prompts leave room for mistakes. An enclosing OpenTelemetry span groups both turns into one execution, so the UI can show their relationship rather than two disconnected top-level calls.

The initial request asks for Tesla's financial performance and growth outlook. The agent searches, reads the results, decides whether it has enough information, and searches again if it does not. The recorded example makes four web searches before writing. That stopping decision is part of the agent's behavior: the application does not prescribe a fixed number of searches. Repeating the request can change the searches, sources, and report.

The trace immediately reveals a surprise. Asked to “write a report,” the agent attempts to create a Markdown file. In this configured notebook execution, the write fails, yet the agent still returns output. The Tesla report looks promising—with financial tables, growth drivers, risks, price targets, and a conclusion—but the trace exposes unnecessary failed work beneath it. Observability makes the distinction visible: receiving a report does not establish that every tool operation succeeded.

These same spans become evaluation inputs. The request describes what the agent was supposed to do, the returned report is the object being graded, and intermediate outputs can supply context. Capturing those pieces now makes it possible to ask more precise questions later, including whether the writer stayed within the research it actually received.

The notebook adds twelve test queries to the initial Tesla run, producing thirteen complete executions. Diversity comes from changing the task as well as the ticker: Apple versus Microsoft introduces comparison; Amazon profitability versus AWS profitability changes the analytical focus; a Coca-Cola dividend-yield question asks for different reasoning from a growth-stock report. The point is to exercise the kinds of distinctions users will ask the agent to handle.

Before writing an evaluator, read a dozen or more executions end to end. Ask what was requested, what was returned, and what specifically failed. Otherwise, automation tends to measure whatever is easy to count. For this analyst, the success criteria are concrete: reference the correct ticker, include real recent financial data, offer actionable recommendations, and distinguish forward-looking analysis from historical summary. Product managers, QA, support staff, and other domain experts help define that bar.

Synthetic queries provide a starting point before real traffic exists. Vary wording, specificity, and complexity: a formal Tesla research request and “Yo, is Tesla a buy right now” may express similar intent through very different inputs. Domain experts should review the set because generated queries tend to favor obvious phrasings. Include nonexistent tickers, multipart questions, and attempts to push the agent outside its intended behavior. Replace imagined cases with actual production traffic as it becomes available.

Reading the thirteen outputs exposes a failure that a polished sample concealed. Most reports contain 3,000–7,000 characters inline, including summaries, tables, and recommendations. Three instead return a short summary or a claim that the report was saved to disk. In the Microsoft example, repeated searches precede an attempted file write, and the final response points to a report that does not exist. The user receives neither the full analysis nor a usable artifact. This is a delivery failure, even if substantial research and writing happened internally.

Error analysis has two useful passes:

• Open coding: Write down the observed problem in ordinary language—vague recommendation, missing report, wrong ticker, unsupported number—without forcing it into a category immediately.
• Axial coding: Group similar observations afterward and investigate causes. Bad search results suggest a retrieval problem; appropriate data followed by an unsupported conclusion suggests a reasoning problem; answering outside the product's remit suggests a scope problem.

The cause matters because it determines the repair. A retrieval change and a writing-prompt change address different defects.

Voss's displayed category-frequency table uses random demo codes, so its rankings do not establish which failures dominate these reports. The practical prioritization rule still applies: weigh severity alongside frequency. A rare dangerous response can deserve attention before a frequent minor omission. Evaluation coverage should also come in layers. Code checks, semantic judges, and human review leave different gaps—the Swiss cheese analogy ends with the appropriately informal instruction, “stack the cheese.”

The first automated check asks whether the requested stock ticker appears in the report. Its Python logic identifies uppercase words, removes known non-ticker acronyms, and checks the remaining candidates against the requested ticker. The evaluator accepts query and report and returns a score and label. This is a narrow requirement that does not need another model.

AX can run the function through code or its UI. In the UI, a representative trace supplies the mapping from recorded fields to evaluator parameters: select the request for query and the returned text for report. Scope matters because a full-report check needs the relevant execution output, while a check on one tool invocation needs that span's output. Offline results can be logged back to AX as annotations, making failing runs filterable alongside their traces.

Twelve reports pass and one fails. The failure is the Amazon query focused on AWS performance and profitability. Research produces material about AWS, and the writing turn stays entirely within that subsidiary-level discussion. The final report never mentions Amazon's stock ticker. A tiny deterministic check exposes the observable consequence of that narrowing: the requested company identity disappears from the answer. It does not establish that the AWS facts are wrong or that the whole analysis is poor; it identifies one unmet requirement.

Deterministic grading can do more than inspect strings. A code evaluator can check structure or compare a value with a database or API result. The important property is reproducible grading for the same evaluation inputs. For most outcome checks, avoid requiring a particular tool order: the ticker test cares that the answer retains the requested ticker, regardless of how the agent found its information. Intermediate checks still have a role when tool choice, arguments, or stopping behavior are themselves requirements.

Semantic grading separates three ingredients: the judge model, the rubric, and the examples. Keeping them separate allows the same rubric to be tested with another model, or a revised rubric to be tested against the same examples. The workshop chooses Claude Sonnet as a more capable judge for Haiku's reports and first runs a built-in correctness evaluator.

Every report receives zero: correctness rejects all thirteen. The explanations reveal an information mismatch. The financial analyst researched live financial information, while this judge configuration grades the request and answer using the model's own knowledge without the collected research. Voss attributes the rejections to the judge treating recent information as beyond what it knows. The zero score therefore fails to distinguish useful reports from bad ones.

Faithfulness changes both the information supplied and the question asked. Extract the first turn's research output from the trace, attach it as a context column to the corresponding report, and evaluate the request, report, and context together. Now the judge can ask whether the writer's claims are supported by the material it received. What changes in this data flow when the judge gets context? The diagram shows the same research output feeding both writing and evaluation.

The resulting split is six faithful reports and seven unfaithful ones. Those are the judge's assessments of support in the supplied research, not independent verification of every financial fact or source. Unlike the blanket rejection, the split supplies a usable capability target: improve how the writer stays grounded. Correctness and faithfulness answer different questions, and this live-research application needs the judge to see the research context.

A grounded summary can still leave the reader without a decision. This analyst is supposed to offer recommendations—buy, sell, or hold—and forward-looking analysis, so it needs a domain-specific actionability evaluator. “Helpful and accurate” does not tell the judge what to look for. The rubric must describe evidence a reader can actually identify in the report.

The custom rubric separates several jobs:

• Task context: Tell the judge that it is evaluating a financial report for useful financial analysis. The relevant context matters more than an elaborate expert persona.
• Pass and fail criteria: Require specific recommendations and distinguish forward-looking analysis from a historical recap. Derive criteria from observed failures rather than an imagined ideal response.
• Tagged inputs: Delimit the user query, financial report, and examples with XML tags so their roles are clear.
• Configured output choices: Set actionable and not actionable, with scores one and zero, in evaluator configuration. AX uses a tool to collect the classification, so the rubric need not also demand a particular free-text output string.

Examples make the distinction concrete. The actionable example includes percentage revenue growth, a PE ratio, a quantified risk, and a specific recommendation. The non-actionable example can be factually acceptable yet offer only the vague suggestion that investors consider various factors. Voss favors examples because they clarify borderline judgments, while acknowledging the tradeoff: rigid examples can encourage overfitting and reduce variation in what the evaluator accepts.

Binary labels keep the decision easier to specify. A one-to-five scale requires separate rules explaining the difference between adjacent scores; a pass/fail criterion already describes a meaningful distinction. Voss also recommends asking for an explanation before the score, both to help the judge apply its criteria and to make disagreements inspectable.

The evaluator can run in the notebook or continuously against incoming traces. Its configuration combines a name, model, template, input mappings, and label choices. The resulting annotations sit beside ticker and faithfulness scores. Filtering for eval.actionability.label equal to not actionable isolates the cases to investigate and later rerun with changed prompts.

An evaluator prompt is another application component. Small wording changes can alter its judgments, so version the rubric and test it on examples with known labels. The “God evaluator”—one prompt grading accuracy, tone, completeness, policy compliance, and formatting—makes calibration difficult. A failure leaves the team unsure what to repair. Separate dimensions preserve a useful diagnosis: ticker identity, faithfulness, and actionability each name a different concern.

Scores also need different operational consequences. A fabricated stock price is a guardrail failure that can block shipping. A preference for complementary investment recommendations is an aspirational metric. Treating them identically either weakens an essential requirement or makes an optional improvement unnecessarily disruptive. In production, that distinction determines whether a regression should trigger an urgent alert or appear in a periodic quality report.

An actionability judge is a classifier: it reads a report and predicts one of two labels. Human annotations provide a comparison set. In AX, a domain expert can read a span, select a Human Actionable label, and move to the next report without writing code. Exporting those spans returns the annotations as columns, allowing the notebook to compare human and model judgments on the same rows.

A trustworthy comparison set needs specific labeling rules, solvable tasks, and reference outputs that pass the graders. Test behavior in both directions: an agent should search when outside information is needed and avoid searching when it is unnecessary. Otherwise, a suite that rewards only tool use can teach the wrong habit. Voss suggests using 75% of labeled examples to develop the judge and holding out 25% to test whether revised criteria generalize.

The demo comparison agrees on six of thirteen reports, or 46%, because Voss deliberately assigned random human labels to create disagreements. That number is not a measurement of judge accuracy. With genuine annotations, disagreement prompts a close reading of the report, rubric, and judge's explanation. Either label can be wrong, or the rule can be ambiguous. Tightening “includes forward-looking analysis” to “includes forward-looking analysis with specific recommendations or guidance” resolves one such ambiguity: discussing the future alone does not make a report actionable.

Treat not actionable as the positive class when interpreting error-detection metrics:

• Precision: Of the reports the judge flags as not actionable, how many really fail that criterion?
• Recall: Of all reports that really are not actionable, how many does the judge catch?

Voss generally favors recall when missing a defect is more costly than investigating a false alarm, while recognizing domains where precision matters more. Small samples make both estimates unstable; they can guide early iteration without establishing shipping confidence.

Several biases can make the judge reward presentation rather than quality:

• Position bias: In comparisons, the order of candidate answers can influence preference.
• Length bias: Extra text can receive a better score even when it adds filler.
• Confidence bias: Assertive wording can make unsupported claims look credible.
• Self-preference bias: A model may favor outputs resembling its own generation. Voss uses different worker and judge models and recommends considering different providers as well.

Human judgment is a fallible reference too. Some disagreement is expected, especially on subjective criteria, and should lead to investigation rather than automatic distrust. The useful test is whether failures make sense: a failing trace should reveal what the agent got wrong and why. If the response satisfies the intended requirement but the grader rejects it, revisit the evaluator—the same lesson as the unexpectedly valid flight-rescheduling route.

A prompt edit is only a proposed fix until the agent runs again. AX turns filtered failing traces into a dataset—in the workshop, AIEWF Financial Demo Fails—so iteration can concentrate on known weaknesses. Save passing examples too. The failure set makes focused experiments cheaper; the passing set checks that the repair preserves working behavior. As real traffic arrives, both sets should grow from observed cases.

The repair input combines requirements with failing traces and their explanations. A missing-recommendation explanation suggests making recommendations explicit in the writing prompt. A risk presented without evidence suggests requiring supporting evidence. The workshop uses Claude to produce revised research and writing prompts from this feedback, then substitutes those prompts into the otherwise unchanged agent. The changes have concrete reasons, but the prompt-improvement step remains a fallible LLM operation.

An experiment combines a dataset, a task, and evaluators. The task is a Python function that accepts an example and returns an output. It can run the whole agent, one component, or an API call. Here it reruns the complete financial analyst, then applies the same evaluator to the new reports. Each result retains the output and score, making the before-and-after difference inspectable at the example level.

Previously non-actionable cases become actionable, and Voss reports a displayed result of 100% after the prompt change. This is a rehearsed workshop result on a small selected dataset, not a demonstrated production success rate. Holding inputs and evaluators fixed removes variation from case selection, but live search and agent decisions still vary between runs. The comparison supports investigating the prompt improvement; it does not isolate the prompt from every source of randomness.

The useful observable change is that the same selected requests now produce reports accepted by the actionability rubric. The causal development is straightforward: identify failing outputs, read explanations, translate recurring omissions into stronger instructions, rerun the agent, and apply the same criteria. Real projects repeat this process rather than expect one perfect edit. The score tells the team whether to keep investigating; the explanations tell it where to look next.

Voss offers 12–20 examples as a directional workshop-scale signal and 200–400 as a practical target for shipping decisions. These are starting ranges rather than guarantees: the task and consequences of missed failures still determine what confidence is needed. More examples also have diminishing returns; the workshop notes the familiar sampling relationship that halving a margin of error requires roughly four times the sample size.

The proposed order of investment follows the source of the failure. Fix data quality first: wrong sources or stale knowledge cannot be reliably repaired by better wording. Next improve prompts with examples, explicit instructions, and constraints. Then consider a more capable model, accepting additional latency and cost. Temperature and top_p are easy to adjust, but Voss places them last because they usually address less than a clear data or task-specification fix.

Eval-driven development moves the definition of done ahead of implementation. If a refund agent must verify identity before processing a refund, write the evaluation for that requirement first and build until it passes. Here, sequence is consequential because verification must precede the action; it is different from prescribing an arbitrary research-tool order. People closest to product requirements can describe the intended behavior in natural language, which the team then turns into a testable rubric.

Passing a development experiment does not finish the work. Online evaluations apply the same ticker, faithfulness, and actionability checks to new production traces. Choose scope to match the question: a span-level evaluator can inspect a particular model or tool call; a trace-level evaluator can inspect the full execution tree. Sample expensive LLM evaluations—for example, 1% or 10% of traffic—to obtain a directional signal without paying to judge every run.

AX also includes Alyx, an agent that can turn a plain-language evaluation request into rubric and UI configuration. That helps domain experts contribute without implementing the plumbing themselves. Automation does not replace defining success: it moves the description into an evaluator that still needs the calibration and testing developed earlier.

How does a newly discovered production failure become protection for the next release? Online scores feed monitors, monitors prompt investigation, and the team saves relevant failing traces as regression examples. After a repair, an experiment checks the candidate before shipping it. The cycle below makes that return path visible: production creates new cases, while accumulated cases constrain subsequent changes.

The dataset becomes increasingly specific to the application: its users, edge cases, failure modes, and quality bar. Every caught failure can add a test that did not exist before. That compounding value requires continued work on evaluators and monitoring as new traffic reveals behaviors the original synthetic queries never anticipated.

The workshop edits two prompts, but real agent behavior can depend on prompts across many files, retrieval settings, and tool definitions. A repository-aware coding agent can work across those components. Export a batch of failing traces with their explanations and use them as repair context; Arize Skills can handle the AX operations needed to retrieve that data.

Two instructions keep the proposed repairs aimed at the product:

• Supply requirements as well as failures: Asking only for passing scores invites shortcuts, including fitting directly to test cases. Specify the behavior users need, with evaluations measuring whether the implementation provides it.
• Find themes across traces: Ten failures with one cause deserve a shared repair. Chasing each case separately can produce unnecessary code that solves examples while leaving the recurring problem intact.

Experiments remain the check on those changes. The coding agent proposes fixes from explanations; it does not get to make its own success claim sufficient. New production traffic returns to the same evaluations, revealing both remaining defects and unfamiliar ones. This closes the improvement loop across execution, diagnosis, implementation, and measurement.

The ending recommends starting smaller than the full workflow: instrument one agent, read real traces, and add one inexpensive deterministic check for an important requirement. Then add the semantic evaluator the application actually needs, inspect its results, and expand from recurring failures. Voss's closing judgment is that fifteen minutes reading real outputs can teach more than an hour building dashboards. The useful habit is routine attention to what the system did—and a growing set of tests that catches the next regression before users do.