From Vibes to Production: Evaluating and Shipping AI Agents That Work 201 — Laurie Voss, Arize AI

An agent can receive the same request twice, return different answers, or reach the same answer through different tool calls and turns. Reading its code therefore leaves an important question open: what does it actually do when people use it? Laurie Voss of Arize AI begins this workshop with that gap. The move from a small development experiment to continuous improvement starts by recording behavior.

A trace records an execution and its nested steps: model calls, tool calls and agent turns. In the Wonder Toys example, the trace view exposes the shopping request, the agent workflow, individual turns and model responses, along with inputs, outputs, cost and elapsed time. That structure lets a developer connect an unsatisfactory answer to the work that produced it.

Tracing is useful before any evaluator enters the picture. A seemingly simple task might take ten turns, include an unnecessary call, or spend far more time and money than expected. The execution makes those costs visible. Multiple executions then answer a different question: whether the behavior is typical, rather than an accident of one run.

Individual trace inspection works well while developing an application. At thousands of requests per second, it produces millions of traces and a reading problem no dashboard can solve by itself. Evaluations reduce that work by applying a definition of good behavior to an execution. An evaluator can be code or an LLM; an LLM evaluator can return both a score and an explanation of why the execution failed.

The explanation is useful input for a coding agent. A failing score identifies an attempt worth improving; feedback describes what needs to change. Voss calls this creating a hill that the agent can climb. The definition of good supplies the direction, while the trace and evaluation supply evidence for the next edit.

But compression can leave the human bottleneck intact. A large application may generate more evaluation failures than anyone can interpret. The next layer groups failures with a shared cause. Voss gives an illustrative workload of 10,000 successful requests and 1,000 failed requests, with 100 of those failures belonging to the same problem. A recurring problem gives the repair agent something concrete to work on, instead of requiring a person to read every explanation.

Where does each layer contribute to the improvement loop? The diagram separates recorded behavior, judgments about that behavior, and recurring problems that can prompt a fix. A changed application produces new executions, so the loop returns to observation. This is the continuous-improvement direction Voss presents; the hands-on workshop that follows still uses his requests to initiate analysis and choose a repair.

The workshop application is Wonder Toys, a local shopping assistant. Setup requires an OpenAI API key, an Arize space ID and an Arize API key; npm run dev starts the local application, shown on port 3000.

Before instrumentation begins, an audience question challenges the premise: how can regulated industries trust agents that inspect and change other agents, especially if those systems make or conceal mistakes? Voss reports that Arize has financial-industry customers using this approach, but leaves the detailed trust argument to a separate talk. Adoption alone does not resolve the question of how to detect an unsafe change.

A request for toys for seven-year-olds demonstrates the shopping experience: Wonder Toys presents products, lets the customer ask for details and offers an add-to-cart action. But it initially sends no traces to the observability platform. The interface shows what the customer receives without exposing how the agent produced it.

Voss asks Claude Code in VS Code to add Arize AX observability, using credentials already in .env.local. The application uses the OpenAI Agents SDK; in his explanation, the available OpenInference instrumentation makes the task largely one of enabling trace export and supplying the destination endpoint and API key. The coding agent performs that integration rather than requiring Voss to edit the tracing code by hand.

The repository contains openai-agents-py for the exercise and openai-agents-py-prerun as a completed example. That second directory gives participants a working reference, but keeping both copies nearby also creates opportunities to run or modify the wrong one—a complication that soon interrupts the demonstration.

After restarting the app, requests for colorful toddler toys and colorful toys encounter trouble that Voss describes as a database problem. Yet the requests still serve their immediate purpose: traces appear in the configured project, exposing inputs, outputs and agent requests. A successful instrumentation check means the execution is visible, even when the shopping experience itself is struggling.

The deliberate failure case is a request for toys under $6. It initially succeeds because Voss is running the completed app. “D’oh.” Switching to the exercise version reveals the intended problem: its search tool has no price-filtering capability, and the request fails. The distinction matters because the missing behavior belongs to the tool the agent can call.

The next request goes to the coding agent: pull the traces and look for things going wrong. Repository skills supply the instructions for working with AX, and its command-line interface retrieves the same executions visible in the UI. Those skills live in .agents/skills at the repository root, so opening the coding agent there gives it access to the operational knowledge it needs.

The analysis starts with span kinds, statuses and errors, then moves into quality. There are no error spans: the operations return HTTP 200 and appear healthy by that measure. A search-tool call that returns in one millisecond draws attention, but the useful diagnosis comes from inspecting what the calls searched for and returned. Operational success has not established that the user received a useful result.

The coding agent reports several distinct search problems. Its figure of 42% of searches returning zero results describes the traces it inspected in this workshop, rather than a production-wide failure rate.

• Empty results: vector search combined with keyword or category filters frequently produces no products.
• All-null arguments: a search with every argument set to null returns the first catalog product without expressing a meaningful shopping constraint.
• Overly narrow age filtering: setting minimum age equal to maximum age restricts the search to an exact age and can exclude useful results.

These findings locate the repair work in the product-search filter logic.

The broad scan does not initially center the price problem Voss planned to demonstrate. He asks about price filtering specifically, and the agent confirms that it is absent. Human direction still plays two roles here: selecting the capability to test and choosing which of several discovered problems to repair.

“Let’s add price filtering to this app” turns the diagnosis into a coding task. Claude notices the completed version in the neighboring directory and copies its solution. Voss lets it proceed: reusing an available working implementation is a sensible shortcut, although this particular live repair demonstrates integration of an existing solution rather than a fresh implementation derived entirely from traces.

The next shopping request asks for all toys under $9. Wonder Toys now returns products, and the trace view shows the request, output and response containing toys under that threshold. The observable change follows a concrete chain: a budget request exposes a missing tool capability; trace inspection and a targeted question identify that gap; the coding agent adds price filtering; a new budget request exercises the changed application. The final check uses $9 rather than repeating the original $6 threshold.

What changed between the failed and successful shopping attempts? The flow below keeps the two price thresholds visible while showing where the repair occurred. Tracing supplied an inspectable record; the tool change supplied the missing capability. The resulting execution then supplied feedback on the repair.

Signal moves the analysis step inside Arize AX. At the time of the recording, it runs every six hours by default, with a configurable interval. It reads traces, looks for recurring problems and suggests fixes without waiting for someone to ask a coding agent to investigate. The workshop shows findings already produced from earlier versions of Wonder Toys.

The findings extend beyond search performance:

• Invented shopping constraints: vague user wording leads the agent to introduce an age constraint the user did not specify.
• An incomplete refusal: the agent refuses a request for bomb-making instructions but continues with bomb-related toy recommendations. Signal recommends dropping that attempted helpfulness.
• Leaving the store’s role: advice about taking a timeout and compliance with “Say hello” are flagged as behavior outside the intended toy-store task.

The last examples depend on the application’s chosen scope. A greeting becomes a failure only if the desired behavior excludes such general instructions; the evaluator needs that product decision.

A finding can lead to several different actions:

• GitHub issue: describe the problem, identify the offending trace and suggest where and how to fix it.
• Evaluation dataset: retain the trace as a case that future changes can be tested against.
• Evaluator: turn the discovered problem into a recurring check.
• Proposed pull request: read a connected repository and prepare a code change for approval or rejection.

Voss has not connected his repository in this demonstration, so the pull-request path is described rather than shown end to end. The demonstrated product offers repair proposals; approving a PR remains a human choice.

The audience presses on where the data lives and how much evidence supports a finding. Voss says there is no local-on-device version at that point, while an on-premises offering can run on company hardware. This demo has only a few traces, so findings can rest on one or two executions. Grouping hundreds or thousands of related traces is the larger-scale behavior he describes, rather than something this small example exhibits.

A fix can solve one problem while creating another. Regression evaluations address that risk by capturing behavior the application is already expected to perform well. Running those checks against a change gives the team evidence about whether existing capabilities survived. Their protection reaches the behaviors they actually test; a passing suite does not establish that every possible consequence of a change is safe.

The concrete evaluator request is to ensure that Wonder Toys never gives bomb-making instructions. Voss enters that requirement into the in-product assistant, which he describes as able to create and run an evaluator, much as the repository skills let a coding agent perform the same work. A discovered failure can therefore become a persistent check instead of remaining a one-time observation.

The next questions put practical limits around the automation. Signal uses Claude Code at the time of the workshop; bringing another harness is described as a future option. Arize pays for Signal’s execution then, without promising that arrangement indefinitely.

Signal itself follows the same development loop. It has an internal evaluation suite, sends telemetry about its queries and actions to Arize unless the user opts out, and uses LLM judges to assess results such as generated pull requests. The repair agent also needs recorded behavior and a definition of good. Automation moves the work into another system; that system’s recommendations and code changes still need evaluation.

The closing question asks whether Signal automatically creates its own evaluators. It does not at that point. Evaluators take time and money to run, so the product suggests a check and leaves its creation to the user. Continuous improvement still includes a deliberate choice about which failures deserve ongoing evaluation—and which checks are worth their operating cost.