# How to Write Your First AI Eval - SallyAnn DeLucia, Arize AI

Sally-Ann DeLucia, Arize

[Talk page](https://ai.engineer/talks/gv4w23xYj-0-write-your-first-ai-eval) · [Video](https://www.youtube.com/watch?v=gv4w23xYj-0)

## How to Write Your First AI Eval

SallyAnn DeLucia builds an LLM judge for a travel-planning agent, moving from trace inspection to concrete pass/fail criteria and a reusable evaluation prompt. She then explains how to check the judge against human annotations and run it on incoming traces.

### A prompt fix needs a way to stay fixed

[0:12](https://www.youtube.com/watch?v=gv4w23xYj-0&t=12s)

On Monday, a prompt change feels better. It ships. A week later, the use case it supposedly fixed is broken again, and nobody can establish whether the fix worked in the first place. SallyAnn DeLucia, introduced as Arize's Head of Product, opens the workshop with this familiar failure of “vibe checks.” Her experience building Arize's agent, Alyx, supplies the practical starting point: evals should provide a signal about agent behavior that survives beyond one person's impression.

An evaluation makes a quality judgment explicit enough to repeat. That gives a team something to compare when it changes a prompt or switches models, and something it can eventually use as a quality gate in CI. The question becomes whether the new version performs better on the application's own cases, rather than whether its latest answer looks convincing.

The travel-planning example explains why checking one expected output is insufficient. Several different itineraries can satisfy the same request. A useful test therefore asks about properties of the answer: its day-by-day plan, budget breakdown and fidelity to the request. DeLucia contrasts this with a conventional unit test's known input and expected output; the important distinction here is the presence of many acceptable answers, not the disappearance of deterministic tests.

Agent behavior also stretches across multiple calls. A research agent may use tools while another agent retrieves information, and several LLM calls contribute to the final answer. A trace records those steps as spans, making the executed path and exchanged data inspectable. In DeLucia's shorthand, traces are the logs and evals are the tests. The trace supplies the behavior to examine; the eval supplies a judgment about it.

### Choose a judge that fits the question

[5:15](https://www.youtube.com/watch?v=gv4w23xYj-0&t=315s)

The workshop focuses on **LLM-as-a-judge**: a second language model grades some aspect of the application, such as answer quality, summarization or relevance. It returns a label and an explanation. The explanation matters because a bare label gives the developer little to inspect when deciding whether the evaluator itself made a sensible judgment.

Other evaluation methods fit different kinds of questions:

- **Decision models:** DeLucia describes these as a cheaper, faster option that returns a probability and an outcome, such as a Boolean or category, without the explanation supplied by an LLM judge. Those cost and speed advantages are her characterization, rather than a measured comparison in this workshop.
- **Code-based evals:** Use code for deterministic requirements, such as checking whether a word appears or an output is valid JSON. ROUGE and BLEU are also named as code-based scoring options.
- **Annotations:** Reference judgments on a dataset provide another form of evaluation; the examples may also be synthetic.
- **Agent-as-a-judge:** A full agent evaluates another agent's work. DeLucia recommends considering this for especially complex success criteria.

User feedback and business metrics add production signals. The workshop places those, along with agent-as-a-judge, toward the production end of its overview, while treating LLM judges, decision models, code checks and annotations as useful during both development and production. The investment should carry forward: an evaluator built while developing an application can continue measuring that application after it ships.

### Start with an actual travel request

[7:29](https://www.youtube.com/watch?v=gv4w23xYj-0&t=449s)

The hands-on setup uses Arize AX and a model-provider API key. In the demonstrated interface, the key goes under Settings → AI providers, and Test Integration checks that it works. The workshop allows a provider of the participant's choosing, including a custom endpoint; the immediate requirement is simply a working way to make judge-model calls. The tracing page's See Demo Project opens the example data, with a wider time range available when too few traces appear.

The demo agent accepts a travel request, can ask follow-up questions and call tools, and returns a plan. Its inputs include destination, trip length, budget and travel style. Before writing any evaluator, inspect one or two traces: what did the user ask, which tools ran, what did they return, and would you ship the resulting answer? This turns an abstract quality discussion into a review of actual application behavior.

One request is for **Prague, Czech Republic, for three days on an $800 budget**, with an option that feels more local and less touristy. What should be compared between the request and the result? The agent produces a plan organized into day one, day two and day three, so trip length is directly inspectable. The budget and local-feeling preference provide additional requirements to judge. The walkthrough establishes these inputs and the itinerary's structure, but does not establish that this particular plan passes the budget or travel-style checks.

The trace tree lets the review move from the final plan into individual steps. Select a model-call span to inspect the input and output sent through that call; use Attributes for additional recorded data. Pay attention to where each piece of information lives. A future evaluator can only compare the user's constraints with the answer if the selected data contains both.

At this stage, the task is to take inventory rather than rush into a prompt. A product person, engineer or other stakeholder should look at the same interactions and identify what success would mean for this agent. Those observations become the criteria in the next step.

### Read spans by their work, not just their position

[14:06](https://www.youtube.com/watch?v=gv4w23xYj-0&t=846s)

The root span sits at the top of a trace and has no parent: `parent ID = null`. It usually contains the application's input and final output. Below it, individual spans represent units of work. In the travel example, a research-agent span contains work involving an external-data tool and an LLM. The span type distinguishes an agent's work from a tool call or model call.

The research tool retrieves information such as weather or destination details. An LLM then summarizes the tool's information so a later step can use it. Reading the trace means following those inputs and outputs through the recorded work, rather than assuming that each visible model call is doing the same job.

An audience question makes that distinction concrete: why are there two LLM spans? On inspecting the example, DeLucia identifies one call as choosing the tool and the other as summarizing its output. What distinguishes the two is the data they act on. The first selects an action; the tool fetches temperature and destination information; the later model call consumes that returned information. Collapsing both calls into “the LLM ran” would hide the difference between a tool-selection failure and a failure to use the tool's result.

A root span does not, by itself, imply a supervisor that makes every decision. One architecture can use an orchestrating agent to plan work for sub-agents; another can let an LLM choose from tools one step at a time. The demo is described as using modular agents responsible for different tasks, rather than a supervisor architecture. Trace structure records the work, while the application's architecture determines who decides what happens next.

Arize's tracing support is presented as framework-agnostic, with automatic instrumenters supplied through OpenInference. That separates the choice of agent framework from the need to observe and evaluate its behavior.

### Turn “helpful” into a pass/fail decision

[18:49](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1129s)

“Helpful,” “accurate” and “good response” leave too much of the judgment unspecified. For the travel agent, concrete alternatives are whether the answer addresses the user's follow-up, matches the requested number of days, or keeps totals within the stated budget. Each names a relationship between the request and the response that a reviewer can actually check.

DeLucia's default recommendation is to judge **one quality per LLM eval**. Keep budget compliance separate from follow-up handling rather than asking one broad prompt to grade everything. She allows exceptions for decision models and agent judges, but the workshop's starting point is deliberately narrow. Coverage can grow with observed usage: Alyx began with a small set covering answer correctness, user frustration and safety concerns, and had 32 evals running at the time of the recording.

Generic templates can still be a useful first step. Arize's Eval Hub includes starting points for QA correctness, hallucination and session coherence. The tradeoff is speed versus application specificity: a shared template gets measurement started, but it should be adapted to the decisions that matter in the actual product.

Before opening the prompt editor, write down the task, what success and failure look like, and precisely what the evaluator will judge. Then expand pass and fail into concrete bullets—DeLucia suggests roughly three for each. An instrumentation question interrupts the exercise but reinforces the same dependency: automatic tracing captures common information such as model input messages, while manual or hybrid instrumentation can add application-specific data needed for the judgment.

The worked rubric asks a specific question: does the agent answer a trip follow-up using that trip's details?

- **Task:** A user sends a follow-up about their trip.
- **Success:** The answer addresses the follow-up directly using the trip's details.
- **Failure:** The answer falls back to a generic budget.

The distinction is between using the existing trip context and substituting a generic response. Detailed pass/fail bullets are useful because they will become instructions for the judge model, much as a colleague would need concrete expectations to review the same answer.

### Select the spans that contain the evidence

[25:02](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1502s)

Once the criterion is narrow enough, build a dataset from the spans that contain the relevant evidence. The selection depends on the question:

- **Request versus itinerary:** Save root spans, where the user input and final output usually live.
- **Tool behavior:** Save the relevant tool-call spans when checking whether a tool returned the information it was supposed to provide.

These are different evaluation inputs. A final-answer dataset and a tool-call dataset should not be treated as interchangeable merely because they came from the same traces.

Filters make that choice repeatable. Use `parent ID = null` to find root spans, or filter by span name to find a particular tool. Select the resulting spans and add them to a new dataset. This is the handoff from exploration to testing: the examples are now collected for repeated judge runs instead of being opened one at a time in the trace viewer.

### Build a reusable judge in the Prompt Playground

[27:23](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1643s)

The Prompt Playground tests the evaluator's prompt; the separate Agent Playground is for testing an agent end to end. Create a Prompt Playground and either load an evaluator from Eval Hub or begin with a blank eval. The criteria written earlier now become the substance of the judge prompt, rather than being replaced by a vague instruction to assess quality.

The earlier exercise asked whether the agent answers a follow-up using the trip's details. The prompt DeLucia loads in the playground asks a different question. Does the assistant's output accurately capture the details requested or provided in the input? She gives the judge that job, supplies the relevant data and spells out the pass/fail conditions. The judge should mark the output correct only when it is consistent with the input. A wrong destination or property type illustrates what would count as incorrect.

**Variable mapping** connects the prompt's inputs to the actual dataset or trace attributes. The evaluator does not need to hard-code every source field name. If a dataset names a field differently from production traces, update the mapping in the task using the evaluator instead of rewriting every eval prompt. The reusable part is the judgment; the mapping handles where its evidence comes from.

Configure the result separately from the prose rubric:

- **Labels:** The example uses `correct` and `incorrect`. Other use cases can add `partially correct` or `not applicable`, but each evaluation returns one label.
- **Optimization direction:** Specify which outcome or score direction counts as better, so results are interpreted consistently.
- **Explanations:** Request the judge's account of why it assigned the label, giving a reviewer something to examine beyond the category alone.

Load the dataset, check the input/output mappings and run the judge. What should the result inspection answer? Whether the label and explanation make sense for the example being evaluated. If a judgment conflicts with the intended criterion, revise the evaluator prompt and rerun it on the collected cases. The live walkthrough reaches this run-and-inspect stage, but does not give a specific row-level correction or a measured improvement in judge accuracy.

Save Evaluator makes the resulting judge available elsewhere in the platform, including for other applications brought into Arize. The saved artifact is the evaluation logic, ready to be used beyond this one playground session.

### Check the judge against people, then monitor real traces

[32:21](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1941s)

After the initial manual pass, Alyx can help run the playground, inspect results, update the prompt and repeat. Signal serves a different role: DeLucia describes it as a managed agent that examines traces, finds issues and can make fixes. These extend the workshop's workflow in two directions—assistance with writing the evaluator, and assistance with improving the application from observed behavior.

The next check is **alignment**. Annotate the examples in the dataset with human judgments, run the evaluator on those same examples, and compare its decisions with the annotations. This tests whether the automated judge agrees with the people whose criteria it is supposed to apply. Editing a prompt until its explanations sound reasonable is only the initial pass; comparison with human labels gives that work a reference.

**Online evals** carry the evaluator into continuous monitoring. Put it into a task that runs over incoming traces, and the resulting evaluations are posted as new traces arrive. The development dataset lets a team inspect and refine the judge on collected examples; the online task applies that judgment to ongoing application behavior.

Success criteria should come from everyone with a stake in the agent: product, engineering, support, domain experts and security. Their outcomes differ, so one role should not silently define quality for all the others. DeLucia pairs that advice with a distinction between metrics and evals: a metric can flag that something may be wrong, while an eval aimed at a particular behavior can identify the failure mode. Both belong in monitoring.

A further experiment is to express the LLM judge's criterion as a decision-judge question and compare performance. The workshop offers this as something to try, not a demonstrated winner. Its closing scope is intentionally modest: perhaps 20 traces, one dimension and one judge. Read the explanations, refine the criterion and expand coverage as real usage reveals what else needs measuring.

## Key takeaways

- Inspect traces before writing the eval. Root spans usually contain the request and final answer; tool and model spans expose the intermediate work needed to diagnose narrower failures.
- Replace broad qualities such as “helpful” with concrete checks, and start with one quality per LLM judge. For the travel agent, trip length, budget compliance and handling follow-ups are distinct criteria.
- Match the dataset to the judgment: use root spans for request-versus-itinerary checks and relevant tool-call spans for tool behavior. Variable mapping lets the same evaluator work with differently named source fields.
- Evaluate the evaluator. Inspect labels and explanations, compare its decisions with human annotations, and then use online tasks to apply the judge to incoming traces.
- Begin with a small, inspectable set—perhaps 20 traces, one dimension and one judge—and expand with production evidence and criteria from all relevant stakeholders.

## Related talks

- [Building Self-Learning Loops for Your Agent — Fuad Ali, Arize AI](https://ai.engineer/talks/fnuNVFgLjHA-building-self-learning-loops-your-agent-fuad) — Extends the first-eval workflow into investigating production failures, proposing repairs and replaying failure cases to compare quality, latency and cost before release.
- [How to Know When an AI Agent Can Work Unsupervised — Evan Kormos, Coinbase](https://ai.engineer/talks/_4Ph2dhmUFI-know-when-ai-agent-work-unsupervised-evan) — Connects explicit quality criteria, representative datasets and traces to decisions about bounded autonomy, human verification and exception handling.
- [The Benchmarks Game: Why It's Rigged and How You Can (Really) Win](https://ai.engineer/talks/EnT4Wej5M5k-benchmarks-game-why-its-rigged-you-really) — Explains why broad benchmark rankings can mislead model selection and why evaluation should reflect the task and deployment domain.

## Slides

### Workshop goals

[0:40](https://www.youtube.com/watch?v=gv4w23xYj-0&t=40s) [Three columns introduce the definition, criteria and working judge participants will build.](https://assets.ai.engineer/sites/aiecode2025/sha256/c8eed427c290c0fac0905ce034d26ba6fe73e82248216d8bbbdc861a5fe618e9.webp)

What you'll leave with
01
02
A clear definition
Your own criteria
What an eval is - and what it isn't.
Written for a real agent scenario.
03
A working judge
LLM-as-a-judge on workshop traces
in Arize AX.
Open
03

### The vibes problem

[2:10](https://www.youtube.com/watch?v=gv4w23xYj-0&t=130s) [A timeline follows a prompt change from a Monday edit to a broken use case the following week.](https://assets.ai.engineer/sites/aiecode2025/sha256/c49bbb4012d34dff27c41cbe2857ec2314efd053869a308078fcbfe2227d07a4.webp)

The vibes problem
Monday
You change a
prompt
It "feels" better.
Tuesday
You ship it
A week later, a use case is broken.
Next week
Nobody can prove
why
What got worse - or when.
Why evals 07

### What evals make possible

[2:41](https://www.youtube.com/watch?v=gv4w23xYj-0&t=161s) [Four boxes list what teams can check after changing a prompt or model.](https://assets.ai.engineer/sites/aiecode2025/sha256/10da3fcbbfc452705ac8f26123f23b5e6d1e54ea41b318b2d3a73b0cda299e9e.webp)

What you can't do without evals
01
Detect regressions when you change
a prompt
02
Compare prompt and model
versions objectively
03
Know if a new model is actually
better
04
Put a quality gate in CI
Why evals
08

### Software tests and AI evals

[3:12](https://www.youtube.com/watch?v=gv4w23xYj-0&t=192s) [A deterministic assertion sits alongside an open-ended request for a trip to Lisbon.](https://assets.ai.engineer/sites/aiecode2025/sha256/f7b2aa0da15e96d7ff9c40943e1d89ce3fcbc8f466701f3ca55e94111d6014a6.webp)

Software tests vs. AI evals
SOFTWARE TEST
assert total (cart) == 42.00
pass
One input, one correct output
AI EVAL
"Plan 3 days in Lisbon under $1,200"
Day-by-day itinerary, $1,050 total
Budget breakdown, no itinerary
Great 5-day plan, $1,400 total
pass
fail • no plan
fail • over budget
Many valid outputs. You have to define what "good" means.
Why evals 10

### Why agents are harder to evaluate

[4:11](https://www.youtube.com/watch?v=gv4w23xYj-0&t=251s) [A trace tree for a travel agent shows nested agents, model calls and tools.](https://assets.ai.engineer/sites/aiecode2025/sha256/d428325216c5b3500d49aa422433b637689b3edb8c974fa817b12181ecce44e9.webp)

Why agents make this harder
Single LLM call: One span: input → output.
Agent: A tree of agents, tools and model calls.
Errors cascade — and bad answers often look polished.

### Types of evals

[5:14](https://www.youtube.com/watch?v=gv4w23xYj-0&t=314s) [A table compares evaluation methods, including model judges, code and human review.](https://assets.ai.engineer/sites/aiecode2025/sha256/0c2fb3b8387f4ad5203761115bfac53bdfe75e32395014a634c0cb6230367de3.webp)

Types of evals
Today
LLM-as-a-
judge
A model grades
with your rubric
QA,
summarization,
relevance
Label +
explanation
Decision
models
Code / logic
Yes/no, choices,
scores
Fast and cheap
Usually no
explanation
Structured output
checks
Code runs, regex,
validation
ROUGE / BLEU
DEVELOPMENT
Annotations
Golden datasets
Labels, scores,
notes
Can be synthetic
Agent-as-a-j
udge
An agent
evaluates an
agent
Handles complex
success
Can use skills
User
feedback
Thumbs up /
down
Accept / reject
Business
metrics
Handoffs to a
human
Conversations
ended early
Adoption, CTR
PRODUCTION
Why evals 13

### A travel planning agent

[10:15](https://www.youtube.com/watch?v=gv4w23xYj-0&t=615s) [The workshop scenario shows the agent's task and an example of its response.](https://assets.ai.engineer/sites/aiecode2025/sha256/3e1743606474e3ec6ddec0970ad7fe2ac602106b3d09a4b6065a27baadbf58fc.webp)

The scenario: a travel
planning agent
It takes a trip request plus a follow-up from the user,
calls tools, and returns a plan.
"GOOD" IN PRODUCT TERMS
Answers the follow-up the user actually asked
Respects the trip's days and budget
WHAT A REQUEST LOOKS LIKE
destinationMarrakech, Morocco
duration1 week
budget$1500
travel styleauthentic
user_input
"Explain what etiquette and local
customs I need to know."
Read the data
16

### Reading a trace

[10:51](https://www.youtube.com/watch?v=gv4w23xYj-0&t=651s) [Three questions guide inspection of a travel-planning request, the agent's actions and the resulting answer.](https://assets.ai.engineer/sites/aiecode2025/sha256/293f57694ea1cddf7863edff77370775e21a080cb9616af69a05f8676e357cba.webp)

What you’re looking for
01 What is the user asking for?
02 What did the agent do — which tools, what answer?
03 Would you ship this answer?

### Measuring success

[13:41](https://www.youtube.com/watch?v=gv4w23xYj-0&t=821s) [The audience is asked to define success for the travel agent.](https://assets.ai.engineer/sites/aiecode2025/sha256/a42a11d1010db33070d88238cd07f1ef2ad5d0aec1f6fb216fd575d0ac17ee71.webp)

THINK ABOUT IT
How would you measure
success for this agent?
What would a user call a good answer?
What would stop you shipping it?
Where in the trace would you look?
Read the data
19

### Vague and concrete criteria

[19:16](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1156s) [Crossed-out adjectives are contrasted with observable behaviors in a follow-up reply.](https://assets.ai.engineer/sites/aiecode2025/sha256/52e199a210a1e3949866d35d024aa82daaff42c804cec65af9753a2ac41c1da7.webp)

Vague vs. concrete criteria
VAGUE
"Helpful"
"Accurate"
"Good response"
CONCRETE
Answers the user's follow-up first
Matches the requested number of days
Keeps totals within the stated budget
Define good 21

### One quality per eval

[19:51](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1191s) [Four candidate qualities are shown, with instruction-following highlighted.](https://assets.ai.engineer/sites/aiecode2025/sha256/25df98608fbc8cc3576ae40fbd77ff5892b69c48201d4df4a801cc1005a49bc4.webp)

Judge one quality per eval
Good practice for any eval - and essential for LLM-as-a-judge. A judge asked to
check everything gets vague. Pick one:
Groundedness
Actionability
Tone
Instruction-foll
owing
Start with one. Add more evals later.
Define good 22

### Writing evaluation criteria

[22:06](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1326s) [Six cells prompt participants to narrow their criterion and describe passing and failing examples.](https://assets.ai.engineer/sites/aiecode2025/sha256/344dfc4429761cab3d6d05d729c503afa6a225472e78b00b5dad9cd2babbd283.webp)

Write your criteria
TASK
One line
SUCCESS
One sentence
JUDGING
One quality, e.g. relevance
PASS WHEN
Up to 3 bullets
IN PAIRS
FAILURE
One sentence
FAIL WHEN
Up to 3 bullets
Define good
23

### Travel agent follow-ups

[24:10](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1450s) [A worked example contrasts passing and failing follow-up answers.](https://assets.ai.engineer/sites/aiecode2025/sha256/d6fec1c37c654de45c7b31d5422a65fdee1cbb1ecb670578307a17076540ab9d.webp)

Example: travel agent follow-ups
Judging: instruction-following
TASK
trip
User sends a follow-up about their
SUCCESS
Answers the follow-up directly,
using the trip's details
pass
WHEN
Follow-up is answered in the
opening lines
Stays within the given days and
budget
Extra detail is brief, if any
fail
WHEN
Leads with a budget breakdown
nobody asked for
Contradicts the requested days or
budget
Never answers the follow-up
FAILURE
Falls back to a generic budget
breakdown
Define good 24

### What the evaluator will test

[25:21](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1521s) [Three boxes ask whether the agent answered the follow-up, used context and respected guardrails.](https://assets.ai.engineer/sites/aiecode2025/sha256/fb48b69ba7f834f641ac7d06a1cd5dd698fb706c03a29ce5320287484190e177.webp)

What you're building
Every eval today is span-level. Pick any span - a few examples:
LLM-as-a-Judge
agent span
Did the final response
answer the follow-up?
tool span
Did the tool return what
the agent needed?
e.g. TripAgentGraph
e.g. weather_brief, local_flavor
span-level
label + explanation
retriever span
Did retrieval find relevant
results?
e.g. VectorStoreRetriever
Build the eval 27

### A first evaluator pass

[27:07](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1627s) [Six lab steps trace the path from choosing a span to running a first evaluation in the playground.](https://assets.ai.engineer/sites/aiecode2025/sha256/2517fde7b5c7e325a562f9c20fe464254db5ba35d24b645581b4eae1ddaa7f99.webp)

Lab: take a first pass
1
In Traces, filter to the span you're judging
2 select the spans relevant to your criteria
3 Add them to a new dataset
Name it
workshop-[namel-[quality]
ON YOUR OWN
4 Open a new Playground
5
Add evaluator → New; write your criteria
6 Load your dataset, map {input /{output), run
Build the eval
28

## Full transcript

[0:12](https://www.youtube.com/watch?v=gv4w23xYj-0&t=12s) Um, before we get started, I just wanted to introduce myself. I'm Sally Ann. I'm Head of Product at Arize. Uh, been at Arize for, like, four years now. Um, one of the first projects that I owned actually was building our AI agent, Alex, into our platform. So I've been building evals for a little while now and, uh, even building a lot of our evals product. And so I think the way that I think about evals is that they're our primary signal for our agents, um, and that includes all of our evals at Arize. So my goal

[0:42](https://www.youtube.com/watch?v=gv4w23xYj-0&t=42s) today is you'll leave with three things: um, a clear definition of what an eval is and what it isn't, uh, being able to write your own criteria. So I'm hopefully gonna be able to provide you with, like, a little bit of a recipe, uh, for how you can think about evals and how you can apply that for any of your real agent scenarios. And then we're actually gonna build a working judge in Arize. Um, and I'll show you a little bit about the life cycle there and how you can think about improving your evals over time. So this is a bit of our agenda. So again, I'm gonna give you a little bit of

[1:12](https://www.youtube.com/watch?v=gv4w23xYj-0&t=72s) context. I don't wanna spend too much time in slides, but I wanna make sure everyone has a really solid foundation for what an eval is and how you should think about it. Um, we're gonna explore some data in Arize together, um, write our criteria, and then we'll actually go build our eval. That's the main part of our workshop today. Um, and then I'll give you a little bit of, like, where you could take this for what's next. Um, I'd love it to kind of have some, like, working sessions so people can collaborate together. I'll try to, like, walk around the room, help folks out. Um, and then I will show kind of like a one-pager that's in

[1:42](https://www.youtube.com/watch?v=gv4w23xYj-0&t=102s) Notion that you can kind of open and follow along as well. So just check the room a little bit. Who here has either, like, shipped a chatbot or an agent, ever written an eval or even just opened the Arize product today? Anyone here? All right. We got a f-a few folks here. So I think this is gonna be great for folks who maybe haven't done any of this. And even if you have, I think today should still be pretty helpful. So gonna start off the context with, like, why I don't think, uh, vibe checks actually scale. So,

[2:12](https://www.youtube.com/watch?v=gv4w23xYj-0&t=132s) uh, this is the vibes problem as I see it. Um, you know, you have Monday, you maybe change something to your prompt, you ship it. Um, it overall feels better. Um, you don't really have a way to know, but overall things are looking good. Um, you know, Tuesday, ship it. A week later, now that use case that you'd shipped a fix for is broken. Um, and then next week, as the problem continues to get worse and worse, there's no way for anybody to prove whether something was fixed or that it's actually failing. You just have vibes to go off of.

[2:41](https://www.youtube.com/watch?v=gv4w23xYj-0&t=161s) What you can't do without evals is you can't detect regressions when you change a prompt. Um, you can't objectively compare a prompt, um, or even different model versions. Um, you don't know if a new model is any better for your use case, and you can't actually put any official quality gates into CI as you're shipping your agent. Um, so without evals, it-- you can imagine it's, it's pretty hard, uh, to actually know how your eva-- uh, agent is performing. Uh, a question I commonly get is, ev-- is an eval

[3:11](https://www.youtube.com/watch?v=gv4w23xYj-0&t=191s) just a unit test? Um, I think that there's a lot of comparison of AI evals to, like, kind of traditional software tests. And I do think that they're a little bit different. Uh, 'cause for a software test, you kind of just have, like, one input, and there's one correct output. You can also read all of the logic in the code. With an AI agent, you actually cannot read any of that logic. That actually comes out after in the form of traces. Um, and so you kind of need to have this AI eval where you can look at, like, the day by day

[3:41](https://www.youtube.com/watch?v=gv4w23xYj-0&t=221s) itinerary in this travel planner example I have here, um, the budget breakdown or, or multiple criteria basically. So there's these many valid outputs that you just can't have, you know, a more traditional unit test on like you can with software.

[3:57](https://www.youtube.com/watch?v=gv4w23xYj-0&t=237s) So basically, how I like to think about this is traces are the logs for AI, um, and evals are our actual test. And I always say that, uh, I used to have a meme. I didn't include it today, but the hard pill is you kinda need traces, you need observability to actually do your evals. So this is an example here for why agents make this harder. So this is just a sample, example of what, like, a trace could look like. And for anybody in the room who's maybe not familiar with what a trace is, um, essentially it is all

[4:27](https://www.youtube.com/watch?v=gv4w23xYj-0&t=267s) of the steps that an AI agent took to be able to get an answer back to a response. So a user says something, and then what I'm showing here in all those spans is all the individual steps the agent took, uh, to get that answer. So in this example, there's like a research agent that's going to go perform some research, uh, maybe even do like some tool calls there. You have, you know, your local agent that's maybe retrieving something from a knowledge base. So lots of back and forths are happening, and this is where, I mean, you can see the logic of your agent. Um, and we're gonna explore some traces in the Arize

[4:56](https://www.youtube.com/watch?v=gv4w23xYj-0&t=296s) UI just a little bit. Um, so you can see there's lots of LLM calls, lots of things composed together. And so you can imagine just having one single unit test like in software not really going to accurately capture all the complexity that's in an agent and be able to tell us how it's performing.

[5:15](https://www.youtube.com/watch?v=gv4w23xYj-0&t=315s) There's actually lots of different types of evals. Today we're gonna focus on LLM-as-a-judge. Um, LLM-as-a-judge is where we take a secondary LLM, and we ask it to grade some aspect of our agent. Um, and so there can be things like QA, summarization, relevance, and then you get a label and an explanation out of that. Explanations are really helpful. Explanations are kind of like the, the reasoning or the thinking behind providing a label. It gives you a little bit of understanding and not just have to take a label at face value. But there's lots of different types of

[5:45](https://www.youtube.com/watch?v=gv4w23xYj-0&t=345s) evals. I want to call out decision models. Uh, this is something that's pretty new. Uh, if anybody kind of took or was a part of the, the Jev-as-a-judge storm that happened about two weeks ago and OpenAI just last week released their decision model API. So I think decision models are gonna be a huge part of evals in the future. Um, those are pretty cheap and fast and it's- Uh, more like a question-answer, and you just get like a probability and an output, whether it's like a true or false or a label. So you don't get that explanation that you get with LLM-as-a-judge. The

[6:15](https://www.youtube.com/watch?v=gv4w23xYj-0&t=375s) other type of eval is code. Uh, these are really great for any deterministic tests. So maybe you have some eval criteria that's like checking if a word is in an output or making sure that something's in JSON. Uh, that's going to be really great for code-based evals. There's also things like ROUGE and BLEU scores that are popular code evals. Um, annotations I do see as a form of evaluation. This is where we're having, uh, you know, golden truths on golden datasets. Um, sometimes they're synthetic as well.

[6:45](https://www.youtube.com/watch?v=gv4w23xYj-0&t=405s) Um, agent-as-a-judge is a newer type of eval that we've seen maybe over the last six months. Um, that's where we're gonna take a full agent and evaluate something in a different agent. I think this is really good for super complex, um, like success criteria. Um, then we have user feedback and business metrics, and those three at the end are gonna be more for your production use cases. But you'll notice the first four are something that you want to use across your life cycle in development and production. Um, and that's pretty important because you want this to be able to evolve with your use

[7:15](https://www.youtube.com/watch?v=gv4w23xYj-0&t=435s) case, and you don't wanna like put a lot of investment in something you're using in development and just throw it away. Those evals can grow with you over time, and it gives you that consistent, um, performance.

[7:29](https://www.youtube.com/watch?v=gv4w23xYj-0&t=449s) All right. Let's go ahead and look a little bit at the data. Um, so I'm gonna pause here on this slide, give everyone a few minutes to go get started. That QR code is going to be for the one-pager, which has all of the instructions for our workshop today. Um, but I want everyone here to go ahead to go to, you know, app.arize.com. Uh, we want to log in to AX. So if you don't have an account, um, encourage you to make a free, um, account today. Um, and then I'll go over the steps from there in just a second. Um,

[7:59](https://www.youtube.com/watch?v=gv4w23xYj-0&t=479s) and for anybody who already, I guess, might have Arize account, the next step is going to be getting an API key from, um, a model provider. So I'll pause there, uh, walk around, make sure nobody has any issues logging into the platform. The question was, can you-- do you have to use like, um, can you use a custom endpoint? Do you have to use a, a common provider for step two? Um, no. Feel free to set up whatever LLM provider you want. We'll just want something that we can make LLM calls to so we can test our eval out. But yeah, step two, basically, uh, go to your Anthropic console, your

[8:29](https://www.youtube.com/watch?v=gv4w23xYj-0&t=509s) OpenAI console, whatever provider of your choosing. Um, you can find your AI providers under Settings in the product, and we're just gonna wanna go ahead, um, and add that API key in. So let me go to my laptop here and show y'all a little bit of how to do this.

[8:46](https://www.youtube.com/watch?v=gv4w23xYj-0&t=526s) So settings here in the nav. Um, you're gonna go click into that, and then under your account settings... Let me try... Cool. You're gonna come to AI providers. And you can see here we have a whole bunch of different providers. So just choose one, um, that is your favorite to use, Anthropic, OpenAI, Gemini. Doesn't matter. Just something that we can use to make API calls with.

[9:20](https://www.youtube.com/watch?v=gv4w23xYj-0&t=560s) And I encourage you to use the Test Integration button when you're doing that, just so we can make sure we have a working API key. Yeah. For, for folks, uh, you'll find the demo project... Sorry. I'll go back. Just trying to make sure everyone has time to, to catch up. But, um, you can access the demo project that we're going to use, uh, in one of two ways. Uh, the first is on from this-- when you go to this tracing page here, you'll see See Demo Project. Um, I have the LLM agent. I know some folks might have our

[9:50](https://www.youtube.com/watch?v=gv4w23xYj-0&t=590s) older demo model there. Does not matter which one you have. You should be able to do our exercise today. So, uh, we just wanna be able to look at some traces. Um, and, you know, you can expand the time if you wanna see a little bit more data there. But, um, give everyone a few more minutes 'cause I only saw a handful of hands that got the AI provider. And I'm gonna see what I can do to see if I can get, um, one that we can share really quickly too. All right. So if you have that, uh, travel agent, uh, demo project opened,

[10:20](https://www.youtube.com/watch?v=gv4w23xYj-0&t=620s) um, this is a travel planning agent. Uh, it takes a request from a user and then can make follow-up questions, call tools, um, and basically returns a plan. Um, what I think good looks like in product terms is it answers the follow-up questions the user asked for and respects the trip stays in budget. So this is generally what a user request looks like. You can see the destination, how long they wanna go, what their budget is, and what their kind of general travel style looks like. Um, and that's an example of what we might see from a user

[10:50](https://www.youtube.com/watch?v=gv4w23xYj-0&t=650s) who goes to this travel agent. So I'd love everyone take a, a few minutes. We want to look at the traces. We kind of wanna just take inventory. What are users asking for? What did the agent tool, uh, do? Which tools? What are the answers? Um, and then just kind of make note, is this an answer that you would ship? So just wanna kind of explore the data and get familiar with it.

[11:15](https://www.youtube.com/watch?v=gv4w23xYj-0&t=675s) So yeah, open one or two traces, um, and maybe start thinking a little bit about what you'd want to measure as you're, you're looking through the data here. So as an example, let me hop on over to the product. I expanded mine to last day. Um, let's go ahead and take a look at one of these traces. So you can see here, um, this destination is for Prague, uh, Czech Republic for three days, eight hundred dollar budget. Um, the user asked, um, for an option that feels more local, less touristy. Um, we

[11:45](https://www.youtube.com/watch?v=gv4w23xYj-0&t=705s) have some kind of our ideas here, and then we can start to view the plan that the, the trip, uh, planning agent created. So you have a detailed day one, day two, um, day three. Um, and if you wanna see any additional, you know, attributes, any additional data that we have, you can see that in Attributes. And so a little bit of, um, how to navigate our product here. So again, this is the trace tree. This is every individual step. If you want to see the data that was passed in for a step, like for example, we

[12:15](https://www.youtube.com/watch?v=gv4w23xYj-0&t=735s) wanna see the data that went into OpenAI to process our request, we can switch to that span, um, and then look at the input and output. So, uh, we're gonna get to this in a little while, but we'll want to pay attention to where the data lives, um, and what data we think might be needed to do any of our evals. So we'll spend a few minutes here just exploring and seeing what kind of insights we find. Getting some questions about where to find, uh, actual data here, so let me just show

[12:45](https://www.youtube.com/watch?v=gv4w23xYj-0&t=765s) that one more time. Um, a lot of times in the new spaces it can be a little hidden, but didn't want folks to spend too much time tracing today when we're here to talk about eval. So you'll just want to click the traces, um, option on the nav here. This will kinda prompt you to get started with Arize, but we're just gonna go to, uh, See Demo Project, and then, um, you can see it in Recently Viewed from there. Um, and you can always expand the time to see a little bit more data there. Has everyone generally had time to, like,

[13:15](https://www.youtube.com/watch?v=gv4w23xYj-0&t=795s) find the project, took a, a quick look?

[13:20](https://www.youtube.com/watch?v=gv4w23xYj-0&t=800s) Seeing a few head nods, we're good. Oh, no.

[13:25](https://www.youtube.com/watch?v=gv4w23xYj-0&t=805s) Let me fix that.

[13:29](https://www.youtube.com/watch?v=gv4w23xYj-0&t=809s) All right. So basically what I want folks to do now before we get into the, the next part of, of this, um, is just, like, look at the agent and think about how you would measure this agent, uh, or success for this agent. Um, think of yourself as the product person, engineer, any real stakeholder of this project, and look at the traces, look at how users are interacting with it, how it's responding, um, and start making kind of like mental note of what the things are that you would measure. Um, in the next part of the workshop, we're actually gonna start defining

[13:59](https://www.youtube.com/watch?v=gv4w23xYj-0&t=839s) what that criteria would be. But high level, as you're going through, think about some of those things that you would measure.

[14:06](https://www.youtube.com/watch?v=gv4w23xYj-0&t=846s) Yeah. So the question was, like, how we should think about reading a, a trace here. So, um, again, for, for anyone who may be just not familiar, a trace is essentially a collection of all the steps an agent took, uh, to answer a user's question. So let's go ahead and open this one here. So at the top, we call that our root span. Um, like, the technical definition of that is parent ID equals null. This means that this span here doesn't have anything ahead of it. It's just sits at the top. And this is usually where the input and final output live.

[14:37](https://www.youtube.com/watch?v=gv4w23xYj-0&t=877s) Um, and then from there we have our individual spans. So each of these is what we call a span. I like to think of it as a unit of work. So let's take a look at this first collection of spans here. So from the root span, the first thing that this agent did, um, along with calling some other agents it looks like, uh, called the research agent. So this is a, what we call a multi-agent, uh, system, where there's multiple agents doing different tasks on behalf of the bigger agent. So in this first one we have the research agent, and that is

[15:07](https://www.youtube.com/watch?v=gv4w23xYj-0&t=907s) going to go ahead and call a tool. So each of these has, like, a type. You can see it, um, different icons. This is an agent span. This is a tool span. Tool span, that's going to be for getting external data. So you can imagine, like in this one, research, um, there's some essential info that we have to grab, like things like weather or sites or adequate, things like that. Um, and then from there we're gonna use an LLM. Um, you can see here all of the inputs and outputs that actually went into

[15:37](https://www.youtube.com/watch?v=gv4w23xYj-0&t=937s) the LLM. So this says, "You're a concise travel agent." Um, and this is using that information from that tool, uh, to actually summarize so that we can use it in a later step here. And so you can just go down the trace tree just like that, like one step at a time. Um, you can see how they all connect, but each of these is going to be doing a step or unit of work so that it can go ahead and answer. So again, the goal of this application is to look at a user's request for a trip

[16:07](https://www.youtube.com/watch?v=gv4w23xYj-0&t=967s) and plan it for them, give them an itinerary of things that they can do, um, places they can stay, all in accordance with their kind of preferences. Um, and so each of these spans is the units that were done to ultimately get this final, um, kind of itinerary. Any questions on that? Happy to answer some questions about tracing while we're, we're going through this. Yeah. So the question is, is, like, there's two LLM spans? Yeah. So sometimes it's, um... I'll go back and look at the exact trace of what happened there. Um, sometimes it's

[16:37](https://www.youtube.com/watch?v=gv4w23xYj-0&t=997s) instrumentation of showing the, the different, like, the,  um, LLM, like choosing the tool to use and then actually, like, passing that information to the tool. So let's go ahead and take a look at what happened here.

[16:54](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1014s) Um, and some of this is just, uh, kind of part of instrumentation here. Um, so this looks like here this went and I think the first LLM call chose the tool, and then I think in this second LLM call, uh, that's where the actual data of the tool was passed to the LLM to su-summarize. So here, um, it's kinda hard sometimes. You have to go down the trace to come back up. So this LLM span chose this tool, and then this LLM span summarized the output of the tool. So that tool

[17:24](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1044s) went and grabbed that, like, temperature data, destination data, and then the second LLM basically summarized it, um, so that we can do kind of further work with it. Yeah. Great question. So the question for, for everyone in case they couldn't hear it is, is that kind of root node making all of the decisions, um, or is there some flexibility with how that works? And there, there's a lot of flexibility. It comes down to what your agent kind of infrastructure is. So you can have what's known as kind of like a supervisor or, um, an orchestrating agent, and a lot of times that will make a, a plan for the

[17:54](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1074s) sub-agents to follow. Um, and then sometimes it's just passed kind of to an LLM with a set of tools, and each step is, goes, like, one by one. So depends a little bit about, like, what framework you're using, what your infra looks like, and the general architecture, um, of your agent. In this case here, this is a LangGraph agent, I believe. Um, so we're not doing any, like, supervisor architecture really, just we have a, a few, like, modular agents that are responsible for different tasks. Great question. The question was, like, which agentic frameworks do we support with, um, Arize? Uh, we're

[18:24](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1104s) agnostic to frameworks. Uh, we have a bunch of what we call auto instrumenters available through our tracing package, um, OpenInference. Uh, we don't have a preference of which framework we wanna use. Our goal is to be really flexible, um, and provide you with really good observability and evals on top of it. So, um, I can link our docs, um, where you can see all the different frameworks that are supported.

[18:49](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1129s) Okay. Let's, uh, keep going a little bit here. Sounds like everyone had some time to take a look at the traces, get theirselves familiar with the data here, um, and hopefully was able to spot some things that they would measure, uh, for the success. So, um, the first step of defining what your eval is, is defining what good means. And this is sometimes tricky for folks to do, so I kind of have this, uh, little trick here for vague versus concrete criteria. Um,

[19:19](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1159s) I generally like to just write down what good means, but with really specific, um, criteria. So instead of just being like helpful, accurate, good response, um, you wanna get a little bit more specific. So answer that follows the user's follow-ups, uh, matches the requested number of days, keeps totals within the stated budget. For this application, those are all really concrete criteria, uh, for what success looks like. And I encourage you, you know, if you're ever building this for your application, you know, sit

[19:49](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1189s) down and, and kinda think about what that criteria is. Another helpful hint that I have is judge one quality, uh, per eval. There are some exceptions to this. I like to point out that, uh, for especially the decision models, um, and agent-as-a-judge, you can really get away with, you know, doing multiple criteria in one. But for LLM-as-a-judge, there's lots of research that shows that it really only does well with handling one criteria at a time. So I think it's good practice, um, to just think of one of them and then

[20:19](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1219s) add more evals over time. Building evals is an iterative thing. You don't just build your evals and then, like, set it and forget, and you're set for the life of the application. You kind of have to iterate. And so, um, you know, when we're defining, like, what success looks like, I usually write down a handful, um, and then kinda start to build those, and then over time, we can add more and more. Um, I mentioned I worked on Alex, our AI agent, um, that lives in Arize, and when I first built the evals, I had a very simple set. It was like QA correctness. I think

[20:49](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1249s) I had some, like, user frustration. Of course, like, jailbreak toxicity, those types of safety evals. Um, and now we have thirty-two evals that run, um, on Alex. So you can see how you wanna start with something small, um, and then as your application gets usage, you can ground that in real production data and then evolve your evals to see what else you might need to define success. I see some questions over there. Yeah. So the question for the, the group is: Is a generic eval better than a more specific one? I

[21:19](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1279s) think it really depends. I think generally speaking, I say building a specific eval for your use case is better, but what I would say is an eval is better than no eval at all. So a lot of folks like at Arize, for example, when they're getting started, we have an Eval Hub with a bunch of out-of-the-box templates in it, and they're really great for starting. We have things like QA correctness, hallucination, uh, session coherence, all sorts of different evals, and I think they're great for getting started. But the, the key is, and we'll go through this a little bit today, is to kind of align that to your use case, um, and then create

[21:49](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1309s) more specific ones over time. So they're-- I think basic ones and reusing them, especially across an org, are great because it's better to have some evals, but then just make them more specific.

[22:02](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1322s) All right. So start with one. Add more evals later. Um, I think in pairs or whoever the group around you, um, I would like to kind of have y'all jot down some ideas. Um, basically, there is six things we want to write down. This is gonna be really helpful for us actually building our eval. Um, so first it's going to be the task. Like what do you want this eval, uh, to be going off of? Uh, what success looks like, what failure looks like, um, what they're judging, like what the eval is actually supposed to

[22:32](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1352s) judge here in this evaluation. And then, you know, basically three bullets on what pass would look like, um, and what fail looks like. And remember from this slide over here, the more specific and concrete you can be, the better your eval will be. So we're gonna use all of this, uh, all these notes that you're writing to actually build our eval in the next step. So the more specific we can be, um, the better. And so yeah, I think it's a great opportunity to kind of chat with folks around you, see what you're thinking, what they noticed in the traces, and I'll walk around and let me know if

[23:02](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1382s) you have any questions. I got a question over here that was like, um, the, the way that the data looks on the trace for this application is pretty specific. It has all, like, the request info. Um, and the question was like, does that come out of the box or do I have to like build that specific to my use case, right? That was the... Just making sure. Uh, and the question is, is with our out-of-the-box instrumentation, we kind of abstract kind of all the tracing best practices for you. So there's a lot of common attributes that we send in. So things like input div value or the input messages

[23:32](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1412s) that we send to the LLM. That's pretty standard. But what you can do is called, um, you know, manual instrumentation or hybrid, which allows you to customize. So, uh, tracing is really flexible. We wanna make sure it's tailored and you get the information you need for your use case, but we try to make it easy at the same time. Um, and if you have, um, you know, your own applications that you want to trace on that kind of starter page was our like MPX evals. That call brings in all of our skills, uh, which is the best way to do tracing these days. Let your agent figure out how to instrument it, and you can kind of nudge it along. So

[24:02](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1442s) highly recommend that if you're gonna send in your own traces, uh, to evaluate later. All right. I'm just gonna click to the next slide to show you an example of what this could look like. And then I know there were some questions over here that I'm gonna come to. So just gonna show an example, um, here of like what your criteria you generally want it to look like. Um, so this is like the tasks that we want to judge. User sends a follow-up about their trip. Uh, the success is answers the follow-up directly using the trip's details. Failures is

[24:32](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1472s) falls back to a generic budget. And you can see here this is the pass and fail. Some folks ask me like why we want to have the bullets and why the one-liner is not enough. And, uh, the reason for that is basically we're gonna use this to write, um, our LLM prompt to do our LLM as a judge. And so we want to be really specific so the LLM, um, can understand. The way that I like to think about this is describing to like a coworker or, you know, a stakeholder, like what success looks like. You almost wanna treat this criteria, uh, creation the same. So

[25:02](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1502s) I'm gonna answer a few more questions and then we're gonna move on to the next step. Uh, for folks who want to get a little bit ahead of that, um, you can... This is just describing make sure that, um, if you have too much in your criteria, you can cut it back. But I think for the sake of time, I'm gonna move here. Um, we want to go ahead and start building our eval. The first step is you're gonna wanna go to those traces, and let me go there now. And you're going to want to select a few, um, traces to

[25:32](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1532s) actually add to your dataset. Um, so for example, say you wanted to just go ahead and evaluate the input versus the itinerary. Like, does the itinerary have the general input? You would want to save all of these root spans to a dataset. Um, if you've decided you wanted to maybe test the tool call and see if the tool call returns, um, the right information that it was supposed to, you would want to add in all of these tools-- uh, calls to your dataset. Um, we have filters here. It's the best way to

[26:02](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1562s) identify data. Um, so one really great example, if it's like root spans, you could do just parent ID equals null or go off of the default traces. Um, and if you wanted to do like a tool call, for example, that, um, I think was the essential tool, you could also filter by name that way. Um, and then you would just wanna grab these or go from the span view to, you know, select those and add to a new dataset. So, um, that's kinda the, the next step, and then we're gonna head on over, um,

[26:31](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1591s) to our playground to actually build and test our eval. So, um, folks can head over there, um, while I answer some questions. And let me know if there are any that come up for adding to the dataset, um, and how you should think about filtering. I wanna show some folks kind of the next steps here for time 'cause I know we don't have that much left here.

[26:53](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1613s) Um, so everybody hopefully has been able to go through, write their criteria, um, identify some spans, uh, for what they're building. Um, and then this is essentially kind of the first pass. This is what you probably saw, um, which I'm gonna do now live for everyone to kind of follow along with. Um, but we wanted to filter the spans you're judging. That's what we just walked through. So look through the traces, identify what span you need to evaluate, build your filter, add those to a dataset. Um, and now what we're

[27:23](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1643s) gonna do is go into our Prompt Playground. That's my favorite place to go to actually test out our prompts here. So you can go to the playground. I already have one set up, but I'm gonna go ahead and create a new one. Uh, we have two types of playgrounds, um, in our product. Uh, we're gonna use the Prompt Playground. That's great for testing eval prompts or, you know, any LLM prompts for your agent. We do also have an Agent Playground. That's for testing kind of your agent end-to-end here. So not so relevant for today. So

[27:52](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1672s) just give it a name. I'm gonna do AI Eng Evals, and then we can go ahead and create. So, um, if you were in our Eval Hub, um, you were over here, and you created, um, an eval, you kind of have a slide over that looks like this. Um, you can go ahead in the playground, just add that, um, from the dropdown here. So you'll see your eval, um, pop up here. For anybody who's just starting the eval process, we're just gonna do blank eval. Um, and you can

[28:22](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1702s) actually just start writing your template here. Um, so what you're gonna basically do is take all that criteria and turn that into a prompt. Um, a lot of it should be like paste. You can paste in those like bullets that you had as criteria, but, um, a lot of times you want to first give it kind of an intro. Like you are an eval agent, uh, or an eval LLM, uh, looking at the output accuracy for your application. Let me actually just

[28:51](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1731s) load this one in. Okay. Perfect. Um, so I like to start my evals with an example of what the LLM is supposed to do. I, I always kind of center it in its job. You're evaluating whether an assistant's output actu-- accurately captures the details requested, um, or input in the-- or provided in the input. Um, you then wanna provide the data. So this is really important. When we were looking at those traces, there was probably, uh, attributes that you want to identify. You don't really

[29:21](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1761s) need to worry about writing the specific names in your eval prompt. We do something called variable mapping, um, that allows you to connect it to your exact data. Uh, the benefit of that is that your evals can be flexible. Like, if you have it running on a dataset that calls it one thing and then your traces call it something different, rather than needing to update every eval, you can just update the task where it's being used. So don't worry too much about the name of the variable. Um, and then this is an example of what I have written for mine. Um, these are the check-in particulars. Those are things like

[29:51](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1791s) those three bullet points for pass and fail. Um, and then here I have mark only correct if, you know, the output is consistent with the input, incorrect if there's a wrong destination, property type, whatever it is that's going to be incorrect. Um, and under params here, you'll see if you set it up in the task, that's where we have the labels. This one here is just correct or incorrect, but you can have, you know, a multi-label problem. Maybe you do correct, incorrect, partially correct. I have use cases in Alex where it's

[30:21](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1821s) actually relevant to me to know when something is somewhat correct but not fully. Um, you can also do things like not applicable. Um, so lots of flexibility. The eval will return only one label, though. Um, and then you have some things here around the optimization direction, which says like, is good or is, uh, correct, um, the good thing, or, um, is a lower score a good thing? And then explanations. Again, those are kind of the reasoning for why the template, um, or why the eval

[30:51](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1851s) gave a certain response. And then what you're gonna wanna do is load in your dataset. So those spans that you created and saved to your dataset, you can load them in. And now what we're in the spot to do is actually start testing our eval. Um, we have all those inputs and outputs that are here. Um, if you need to map them to different variables, mine happen to be the same, but you can do that here. Um, and so now what you can do is actually start to run. I'm gonna run with four one mini. That seems good to me.

[31:21](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1881s) Um, and we'll actually start to see the output of our eval right here in the playground. And this is a great spot to just start iterating on your eval template. Um, look at the response, you know, kind of see if there's something that you agree with or disagree, and then you can adjust the prompt and then continue to run it over time. So, um, that's gonna be what we're-- we do for the next little bit here, is try to get to a template in a good spot. Um, and then what you can do is save that off, um, here in this little button here, Save Evaluator, and then that will be

[31:51](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1911s) available to use, um, across the platform for either a demo project, um, or for any application that you build and bring into Arize after. So, I'm gonna walk around, help some folks write their templates, and then I have just, you know, one or two, uh, slides here to, to close this out. I did wanna kind of point out that we do have some tools to help you. I had everybody write out their criteria, handwrite their eval, and test it, but we do have Alex in the platform, uh, that can help you with that. So I definitely recommend helping or

[32:21](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1941s) having Alex help you, especially after you've done kind of your initial eval pass. It can actually run the playground for you, look at the results, update the prompt, and repeat. Um, so that's really helpful for when you're writing evals. Um, pulling out a little bit of the, the remainder of the loop here. So we spent a lot of time on the second loop or second step today, um, where you instrument, you write your eval, um, but then what the rest of the loop looks like is we have things like Signal. That's our managed agent in the platform that is able

[32:51](https://www.youtube.com/watch?v=gv4w23xYj-0&t=1971s) to look at your traces, find issues for you, even make fixes. Um, so this becomes Signal that our managed agents can then consume and ultimately help improve your application. Um, these are some things that I think good teams do next. Alignment. Um, alignment is going to be the process of, like, annotating all of those examples that we added to our dataset and then going to run that eval and seeing if, um, it matches the human feedback. Um, online evals will be essentially

[33:21](https://www.youtube.com/watch?v=gv4w23xYj-0&t=2001s) us putting that into a task and have it running continuously over our traces. So what that means is in Arize, every time you send in a trace, we'll evaluate it according to all your evals, and then that automatically gets posted. And then you can use that to actually monitor your application. Um, a key thing that I want everybody to walk away here with is that every stakeholder evaluates the agent. This is not just one kind of role's job. You really wanna get everybody in the room who has some kind of stake in the agent and have them basically define what success

[33:51](https://www.youtube.com/watch?v=gv4w23xYj-0&t=2031s) looks like according to their outcome. So, uh, product, eng, support, um, experts, um, security, those are all folks. And then, um, the key difference between, like, a metric and an eval is, like, a metric tells you that something might be broken, but evals tell you what the actual failure mode is. So you really want to use both there. Um, and then I think what might be fun to try next is we do have all these new decision judges. Um, and so I think it's always interesting to take your LLM-as-a-judge, you know, convert it to a decision judge question,

[34:21](https://www.youtube.com/watch?v=gv4w23xYj-0&t=2061s) um, and then maybe compare the performance there, uh, see which one might be better. Um, and yeah. And then again, start small, twenty traces maybe, one dimension, one judge. You know, read the explanations, iterate over it, and then you can expand to more and more evals.

[34:39](https://www.youtube.com/watch?v=gv4w23xYj-0&t=2079s) Um, and just some resources. We have some docs, um, evaluate your LLM-as-a-Jev to your Jev docs. Uh, we do have some deep dives on, like, the decision mo- model benchmarking. So I think somebody had a question of, like, when should I use LLM as a judge? When should I use Jev? I think that's a really great use, uh, case. And, uh, yeah. I'll be hanging out for the rest of the day. Arize has a booth in the expo starting at, I believe, four today. So if you wanna learn more about evals, see a little bit more of our product, definitely come find us. Um, and if there are any questions, I'm happy to answer them. And

[35:09](https://www.youtube.com/watch?v=gv4w23xYj-0&t=2109s) thanks so much for joining me today.