How to Write Your First AI Eval - SallyAnn DeLucia, Arize AI

On Monday, a prompt change feels better. It ships. A week later, the use case it supposedly fixed is broken again, and nobody can establish whether the fix worked in the first place. SallyAnn DeLucia, introduced as Arize's Head of Product, opens the workshop with this familiar failure of “vibe checks.” Her experience building Arize's agent, Alyx, supplies the practical starting point: evals should provide a signal about agent behavior that survives beyond one person's impression.

An evaluation makes a quality judgment explicit enough to repeat. That gives a team something to compare when it changes a prompt or switches models, and something it can eventually use as a quality gate in CI. The question becomes whether the new version performs better on the application's own cases, rather than whether its latest answer looks convincing.

The travel-planning example explains why checking one expected output is insufficient. Several different itineraries can satisfy the same request. A useful test therefore asks about properties of the answer: its day-by-day plan, budget breakdown and fidelity to the request. DeLucia contrasts this with a conventional unit test's known input and expected output; the important distinction here is the presence of many acceptable answers, not the disappearance of deterministic tests.

Agent behavior also stretches across multiple calls. A research agent may use tools while another agent retrieves information, and several LLM calls contribute to the final answer. A trace records those steps as spans, making the executed path and exchanged data inspectable. In DeLucia's shorthand, traces are the logs and evals are the tests. The trace supplies the behavior to examine; the eval supplies a judgment about it.

The workshop focuses on LLM-as-a-judge: a second language model grades some aspect of the application, such as answer quality, summarization or relevance. It returns a label and an explanation. The explanation matters because a bare label gives the developer little to inspect when deciding whether the evaluator itself made a sensible judgment.

Other evaluation methods fit different kinds of questions:

• Decision models: DeLucia describes these as a cheaper, faster option that returns a probability and an outcome, such as a Boolean or category, without the explanation supplied by an LLM judge. Those cost and speed advantages are her characterization, rather than a measured comparison in this workshop.
• Code-based evals: Use code for deterministic requirements, such as checking whether a word appears or an output is valid JSON. ROUGE and BLEU are also named as code-based scoring options.
• Annotations: Reference judgments on a dataset provide another form of evaluation; the examples may also be synthetic.
• Agent-as-a-judge: A full agent evaluates another agent's work. DeLucia recommends considering this for especially complex success criteria.

User feedback and business metrics add production signals. The workshop places those, along with agent-as-a-judge, toward the production end of its overview, while treating LLM judges, decision models, code checks and annotations as useful during both development and production. The investment should carry forward: an evaluator built while developing an application can continue measuring that application after it ships.

The hands-on setup uses Arize AX and a model-provider API key. In the demonstrated interface, the key goes under Settings → AI providers, and Test Integration checks that it works. The workshop allows a provider of the participant's choosing, including a custom endpoint; the immediate requirement is simply a working way to make judge-model calls. The tracing page's See Demo Project opens the example data, with a wider time range available when too few traces appear.

The demo agent accepts a travel request, can ask follow-up questions and call tools, and returns a plan. Its inputs include destination, trip length, budget and travel style. Before writing any evaluator, inspect one or two traces: what did the user ask, which tools ran, what did they return, and would you ship the resulting answer? This turns an abstract quality discussion into a review of actual application behavior.

One request is for Prague, Czech Republic, for three days on an $800 budget, with an option that feels more local and less touristy. What should be compared between the request and the result? The agent produces a plan organized into day one, day two and day three, so trip length is directly inspectable. The budget and local-feeling preference provide additional requirements to judge. The walkthrough establishes these inputs and the itinerary's structure, but does not establish that this particular plan passes the budget or travel-style checks.

The trace tree lets the review move from the final plan into individual steps. Select a model-call span to inspect the input and output sent through that call; use Attributes for additional recorded data. Pay attention to where each piece of information lives. A future evaluator can only compare the user's constraints with the answer if the selected data contains both.

At this stage, the task is to take inventory rather than rush into a prompt. A product person, engineer or other stakeholder should look at the same interactions and identify what success would mean for this agent. Those observations become the criteria in the next step.

The root span sits at the top of a trace and has no parent: parent ID = null. It usually contains the application's input and final output. Below it, individual spans represent units of work. In the travel example, a research-agent span contains work involving an external-data tool and an LLM. The span type distinguishes an agent's work from a tool call or model call.

The research tool retrieves information such as weather or destination details. An LLM then summarizes the tool's information so a later step can use it. Reading the trace means following those inputs and outputs through the recorded work, rather than assuming that each visible model call is doing the same job.

An audience question makes that distinction concrete: why are there two LLM spans? On inspecting the example, DeLucia identifies one call as choosing the tool and the other as summarizing its output. What distinguishes the two is the data they act on. The first selects an action; the tool fetches temperature and destination information; the later model call consumes that returned information. Collapsing both calls into “the LLM ran” would hide the difference between a tool-selection failure and a failure to use the tool's result.

A root span does not, by itself, imply a supervisor that makes every decision. One architecture can use an orchestrating agent to plan work for sub-agents; another can let an LLM choose from tools one step at a time. The demo is described as using modular agents responsible for different tasks, rather than a supervisor architecture. Trace structure records the work, while the application's architecture determines who decides what happens next.

Arize's tracing support is presented as framework-agnostic, with automatic instrumenters supplied through OpenInference. That separates the choice of agent framework from the need to observe and evaluate its behavior.

“Helpful,” “accurate” and “good response” leave too much of the judgment unspecified. For the travel agent, concrete alternatives are whether the answer addresses the user's follow-up, matches the requested number of days, or keeps totals within the stated budget. Each names a relationship between the request and the response that a reviewer can actually check.

DeLucia's default recommendation is to judge one quality per LLM eval. Keep budget compliance separate from follow-up handling rather than asking one broad prompt to grade everything. She allows exceptions for decision models and agent judges, but the workshop's starting point is deliberately narrow. Coverage can grow with observed usage: Alyx began with a small set covering answer correctness, user frustration and safety concerns, and had 32 evals running at the time of the recording.

Generic templates can still be a useful first step. Arize's Eval Hub includes starting points for QA correctness, hallucination and session coherence. The tradeoff is speed versus application specificity: a shared template gets measurement started, but it should be adapted to the decisions that matter in the actual product.

Before opening the prompt editor, write down the task, what success and failure look like, and precisely what the evaluator will judge. Then expand pass and fail into concrete bullets—DeLucia suggests roughly three for each. An instrumentation question interrupts the exercise but reinforces the same dependency: automatic tracing captures common information such as model input messages, while manual or hybrid instrumentation can add application-specific data needed for the judgment.

The worked rubric asks a specific question: does the agent answer a trip follow-up using that trip's details?

• Task: A user sends a follow-up about their trip.
• Success: The answer addresses the follow-up directly using the trip's details.
• Failure: The answer falls back to a generic budget.

The distinction is between using the existing trip context and substituting a generic response. Detailed pass/fail bullets are useful because they will become instructions for the judge model, much as a colleague would need concrete expectations to review the same answer.

Once the criterion is narrow enough, build a dataset from the spans that contain the relevant evidence. The selection depends on the question:

• Request versus itinerary: Save root spans, where the user input and final output usually live.
• Tool behavior: Save the relevant tool-call spans when checking whether a tool returned the information it was supposed to provide.

These are different evaluation inputs. A final-answer dataset and a tool-call dataset should not be treated as interchangeable merely because they came from the same traces.

Filters make that choice repeatable. Use parent ID = null to find root spans, or filter by span name to find a particular tool. Select the resulting spans and add them to a new dataset. This is the handoff from exploration to testing: the examples are now collected for repeated judge runs instead of being opened one at a time in the trace viewer.

The Prompt Playground tests the evaluator's prompt; the separate Agent Playground is for testing an agent end to end. Create a Prompt Playground and either load an evaluator from Eval Hub or begin with a blank eval. The criteria written earlier now become the substance of the judge prompt, rather than being replaced by a vague instruction to assess quality.

The earlier exercise asked whether the agent answers a follow-up using the trip's details. The prompt DeLucia loads in the playground asks a different question. Does the assistant's output accurately capture the details requested or provided in the input? She gives the judge that job, supplies the relevant data and spells out the pass/fail conditions. The judge should mark the output correct only when it is consistent with the input. A wrong destination or property type illustrates what would count as incorrect.

Variable mapping connects the prompt's inputs to the actual dataset or trace attributes. The evaluator does not need to hard-code every source field name. If a dataset names a field differently from production traces, update the mapping in the task using the evaluator instead of rewriting every eval prompt. The reusable part is the judgment; the mapping handles where its evidence comes from.

Configure the result separately from the prose rubric:

• Labels: The example uses correct and incorrect. Other use cases can add partially correct or not applicable, but each evaluation returns one label.
• Optimization direction: Specify which outcome or score direction counts as better, so results are interpreted consistently.
• Explanations: Request the judge's account of why it assigned the label, giving a reviewer something to examine beyond the category alone.

Load the dataset, check the input/output mappings and run the judge. What should the result inspection answer? Whether the label and explanation make sense for the example being evaluated. If a judgment conflicts with the intended criterion, revise the evaluator prompt and rerun it on the collected cases. The live walkthrough reaches this run-and-inspect stage, but does not give a specific row-level correction or a measured improvement in judge accuracy.

Save Evaluator makes the resulting judge available elsewhere in the platform, including for other applications brought into Arize. The saved artifact is the evaluation logic, ready to be used beyond this one playground session.

After the initial manual pass, Alyx can help run the playground, inspect results, update the prompt and repeat. Signal serves a different role: DeLucia describes it as a managed agent that examines traces, finds issues and can make fixes. These extend the workshop's workflow in two directions—assistance with writing the evaluator, and assistance with improving the application from observed behavior.

The next check is alignment. Annotate the examples in the dataset with human judgments, run the evaluator on those same examples, and compare its decisions with the annotations. This tests whether the automated judge agrees with the people whose criteria it is supposed to apply. Editing a prompt until its explanations sound reasonable is only the initial pass; comparison with human labels gives that work a reference.

Online evals carry the evaluator into continuous monitoring. Put it into a task that runs over incoming traces, and the resulting evaluations are posted as new traces arrive. The development dataset lets a team inspect and refine the judge on collected examples; the online task applies that judgment to ongoing application behavior.

Success criteria should come from everyone with a stake in the agent: product, engineering, support, domain experts and security. Their outcomes differ, so one role should not silently define quality for all the others. DeLucia pairs that advice with a distinction between metrics and evals: a metric can flag that something may be wrong, while an eval aimed at a particular behavior can identify the failure mode. Both belong in monitoring.

A further experiment is to express the LLM judge's criterion as a decision-judge question and compare performance. The workshop offers this as something to try, not a demonstrated winner. Its closing scope is intentionally modest: perhaps 20 traces, one dimension and one judge. Read the explanations, refine the criterion and expand coverage as real usage reveals what else needs measuring.