# Break It 'Til You Make It: Building the Self-Improving Stack for AI Agents

Aparna Dhinakaran, Arize

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/Qvp9vw4jJQ8-break-it-til-you-make-it-building) · [Video](https://www.youtube.com/watch?v=Qvp9vw4jJQ8)

## Break It 'Til You Make It: Building the Self-Improving Stack for AI Agents

Agent evaluation starts with individual tool calls, expands to trajectories and conversations, and closes the loop by checking whether the evaluators themselves are right.

You should be familiar with an LLM agent calling tools and using conversation context to decide what to do next.

### When a spreadsheet cannot tell you what improved

[0:34](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=34s)

A team swaps prompts in an Excel sheet, compares a few model outputs, and ships the version that feels better. Meanwhile, the model and tool definitions may also be changing. Which change actually helped, and on which examples? Without a systematic comparison, even a promising result is difficult to turn into a repeatable development process.

The problem also involves who can participate. Product managers and subject-matter experts need a way to contribute judgments to the same evaluation process. Aparna Dhinakaran, an Arize cofounder, frames agent development around making those judgments systematic. Once an agent reaches production, the question becomes more specific: which sub-agent or tool call consistently produces poor responses, and what needs to change there?

![Slide titled “Building Agents is Hard!” with three problem boxes beside a spreadsheet of prompts and model outputs.](https://ai.engineer/api/talk-essay-media/41ec34df5f7eafe5831660d565f4b737a0a145259923fb45bf75be933946c599)

Building agents is hard: tracking experiments, involving stakeholders, and iterating in spreadsheets.

**Evaluation needs to follow the scope of the behavior being tested.** A tool call needs checks on its selection and inputs. A trajectory needs checks across the sequence of calls. A conversation needs checks across turns, where earlier context should inform later answers. Improving the agent also requires improving the evaluations that identify its failures.

### Separate tool selection from argument correctness

[3:13](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=193s)

An agent with several tools must first decide which action fits the current situation. That decision depends on the user's request, previous conversation context, and actions already taken. After choosing a tool, it must extract the arguments from that context. These are separate opportunities for failure.

| Check | Question |
|---|---|
| **Tool selection** | Is this the appropriate tool for the next action? |
| **Argument correctness** | Do its inputs match the request, context, and required parameters? |

A correct tool name does not establish that the call is correct. Keeping the two judgments separate makes the result actionable: a routing problem and an input-extraction problem require different fixes.

![An agent branches to three tools: lookup_sales, data_analysis, and make_purchase, each with a listed set of arguments.](https://ai.engineer/api/talk-essay-media/7bed26a124bccf30eeea0b24b55ca139c08d360b9ce052b84a6662ba266e63f6)

Tool calling evaluation considers both the tool choice and its arguments.

### Start with the paths that need attention

[4:15](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=255s)

The demonstration moves into Arize's own copilot, an agent that helps users investigate application performance, troubleshoot problems, and find possible improvements. Arize records traces of those user questions and evaluates the interactions using its own tooling. Those traces make the steps behind a response available for inspection.

Before opening an individual trace, Dhinakaran looks across the paths the agent can take. The copilot uses an **orchestrator-worker pattern**: a top-level planner chooses where to send a request, and tools perform the next actions. A tool's output can lead to another router or orchestrator that chooses a subsequent tool. The resulting response may therefore depend on several layers of decisions.

At the planner level, the useful question is where those paths diverge in quality. Dhinakaran describes the displayed copilot search Q&A results as roughly half correct and half incorrect. The recording does not supply a sample size, collection period, model, or evaluator configuration for that approximate result. It does provide a direction for investigation: search deserves attention before paths whose evaluations look healthier.

### Follow a failed answer down to its arguments

[7:05](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=425s)

The aggregate view narrows the search; the trace supplies the diagnosis. Dhinakaran follows a concrete drill-down procedure:

1. Select the search Q&A correctness evaluation.
2. Filter to examples marked incorrect.
3. Open a failing trace and inspect its whole-trace evaluations.
4. Inspect the individual tool calls, separating function selection from argument correctness.

The trace can contain both a failed overall answer and a successful local decision. That distinction is what makes the next inspection useful.

In the selected example, the function-call evaluation is correct, but the arguments are marked incorrect. The evaluator's explanation says the arguments do not align with the required parameters. Dhinakaran identifies the likely problem as supplying the wrong arguments for the conversation context, despite selecting the appropriate tool. The demonstration reaches a diagnosis; it does not show an implemented correction.

A small Python fixture illustrates why the two results should remain separate. Here, a request to search for dashboard duplication selects `search`, but supplies a query about monitor alerts. Exact comparison is sufficient for this fixture; the real diagnostic question is whether the arguments express the user's intended request.

```python
request = "Find documentation on duplicating a dashboard."
expected = {
    "name": "search",
    "arguments": {"query": "duplicate a dashboard"},
}
actual = {
    "name": "search",
    "arguments": {"query": "configure monitor alerts"},
}

checks = {
    "tool_selection_correct": actual["name"] == expected["name"],
    "arguments_correct": actual["arguments"] == expected["arguments"],
}

assert checks == {
    "tool_selection_correct": True,
    "arguments_correct": False,
}
```

A single failure label would obscure the useful fact that selection succeeded. Separate results preserve that information while directing the investigation toward argument construction.

### Correct calls can still form a bad trajectory

[8:40](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=520s)

The copilot's search path invokes multiple sub-tools. Evaluating each call independently therefore leaves another question unanswered: did the agent call them in the right order? **Trajectory evaluation** examines the sequence across an entire trace, rather than treating every action as an isolated decision.

For repeated instances of a task, Dhinakaran asks whether the agent consistently completes the necessary actions and converges on an appropriate number of steps. A run that veers into a different order can spend more tokens on the same request, or produce an incorrect output because the actions no longer form a valid sequence. The concern is both completion quality and the work required to get there.

The displayed tool-calling-order evaluation is marked incorrect. That adds a distinct failure category to the earlier argument problem: the individual actions may need attention, but so may their arrangement. No target step count or measured token savings are supplied. A trace-level judgment still covers only one interaction, which leads to the next scope of evaluation: the conversation.

### Carry context across the conversation

[10:25](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=625s)

An example with three human-agent back-and-forths introduces behavior that a single trace cannot fully describe. Does the agent maintain a consistent tone? Does it ask the same question repeatedly, despite having already received the answer? Repetition can expose a failure to use earlier interactions, even when an individual response appears reasonable on its own.

The core requirement is to use context from the previous N−1 turns when answering turn N. That makes **session evaluation** broader than checking each answer separately:

- **Tone consistency:** Does the agent maintain an appropriate tone across turns?
- **Context retention:** Does it use information supplied earlier instead of asking for it again?
- **Conversation-wide correctness:** Does it correctly answer the questions throughout the interaction?

Dhinakaran then opens a conversation from another project to inspect both answer correctness and whether earlier context has been missed. Session evaluations complement the tool and trace checks by testing what survives between interactions.

![Arize session detail showing human questions about a noisy monitor and duplicating a dashboard, green AI response cards, and evaluation labels on the right.](https://ai.engineer/api/talk-essay-media/191987420ac1df23c3481f77477533ab0c4dc5157f3c3c078e31b41d90bed987)

A session view shows successive human questions, AI responses, and evaluation labels.

### Improve the evaluator alongside the agent

[11:40](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=700s)

Tool, trajectory, and session evaluations all feed the application-improvement loop. Evaluate the agent, identify where it fails, annotate or refine the outputs, and use those examples to improve its prompt. The evaluations determine which cases enter that process, so their quality directly affects what the team chooses to fix.

**An evaluator can be the component that is wrong.** With an LLM acting as a judge, its prompt should not remain static while the application evolves. Inspect the evaluation outcomes themselves: an application response marked incorrect may have been acceptable, and the judge may have misclassified it. Treating every negative evaluation as an application defect would send the improvement process after the wrong target.

The evaluator needs its own iteration process: identify mistaken judgments, improve the evaluation prompt, and build and continually refine a golden dataset. The two loops operate together:

| Loop | Evidence to inspect | What changes |
|---|---|---|
| Application | Failed outputs and corrected examples | Application prompt |
| Evaluator | Mistaken judgments and reference labels | Evaluation prompt and golden dataset |

The self-improving stack presented here is a workflow of repeated evaluation and refinement, not a demonstrated autonomous optimizer. Its effectiveness depends on maintaining both the agent's behavior and the quality of the judgments used to improve it.

To try these practices in an application, Dhinakaran points to [Arize Phoenix](https://github.com/Arize-ai/phoenix), which she describes as completely open source. For present-day reuse, its current [Elastic License 2.0](https://github.com/Arize-ai/phoenix/blob/main/LICENSE) restricts offering substantial functionality as a hosted or managed service. She also points to [Arize AX](https://arize.com/docs/phoenix/resources/frequently-asked-questions/what-is-the-difference-between-phoenix-and-arize) for running evaluations on your own data.

## Resources

- [Arize Phoenix](https://github.com/Arize-ai/phoenix) — Source code and setup instructions for tracing agents, evaluating outputs and running dataset-based experiments.
- [Phoenix and Arize AX](https://arize.com/docs/phoenix/resources/frequently-asked-questions/what-is-the-difference-between-phoenix-and-arize) — Current documentation explaining the two products and their deployment and operational differences.
- [Building an AI Agent that Thrives in the Real World](https://arize.com/blog/building-an-ai-agent-that-thrives-in-the-real-world/) — Arize's December 2024 account of developing Copilot through tracing, skill-specific evaluations, datasets and experiments.
- [Session-Level Evaluations with Arize AX](https://arize.com/blog/session-level-evaluations-with-arize-ax/) — An August 2025 tutorial on grouping traces into conversations and evaluating session correctness, frustration and goal achievement.
- [How to Evaluate Tool-Calling Agents](https://arize.com/blog/how-to-evaluate-tool-calling-agents/) — A March 2026 Phoenix tutorial on tool selection, argument correctness and refining application and evaluator prompts together.

## Slides

### Shipping Agents Is Harder

A copilot diagram routes a user question through specialized agents and their tools to a response. In production, finding which sub-agent or tool call causes poor responses can be difficult.

[1:40](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=100s) [Slide describing visibility, measurement, and iteration challenges in production, with a multi-agent copilot diagram.](https://assets.ai.engineer/sites/aiecode2025/sha256/20e7c20219fbd940d99b12157a603ef59b3665576f8c83d5f04876898c188636.webp)

Shipping Agents is Harder!

No Visibility into agent behavior
Lack of ability to quantify application performance
Hard to Iterate and Experiment in Production

USER QUESTION

ARIZE AI COPILOT

Tabular Agent
Model insights
Drift insight
Data quality checks
Cohort performance
Assess features

Eval Agent
Create custom eval

Analysis Agent
Diagnose RAG
Dashboard generator
ArizeQL generator
Span chat

Search Agent
Column search

Prompt Agent
Optimize prompts

Computer Vision Agent
Embedding summarization
Cluster analysis

Tools

COPILOT RESPONSE

arize | We Make AI Work
© All Rights Reserved

Interface text:
Aparna

### Agent Evals

The talk moves from individual tool calls to trajectories and multi-turn conversations, then asks how agents and their evaluations can improve together.

[2:26](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=146s) [Agenda listing tool calling, trajectory, multi-turn or session evaluations, and self-improving agents.](https://assets.ai.engineer/sites/aiecode2025/sha256/7155a6bd2b638c79a2c65c48874ece55febfa4599e5d9170ef70b5391cea35eb.webp)

Agent Evals

1. Tool Calling Evaluations
2. Trajectory Evaluations
3. Multi-Turn/Session Evaluations
4. Self-Improving Agents

Interface text:
Aparna

### Tool Calling Evals

Tool calling evals check whether an agent selects the right tool for the situation and passes the right arguments from the conversation or context.

[3:03](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=183s) [Diagram showing an agent choosing among three tools with different arguments.](https://assets.ai.engineer/sites/aiecode2025/sha256/851f90e10b22d7e966c7bce3319f74a904012164ae9d091a0090ddf09710c748.webp)

Tool Calling Evals

Agent
What's the right tool to call?

Tool Call A
lookup_sales
arg1: store
arg2: product

Tool Call B
data_analysis
arg1: data
arg2: question
arg3: aggregation

Tool Call C
make_purchase
arg1: link
arg2: purchasing_card

arize | We Make AI Work
© All Rights Reserved

Interface text:
Aparna

### Arize Agent Trace Evaluations

Arize shows traces of questions asked of its own Copilot, alongside latency, token counts, and evaluation labels. The team evaluates these interactions to see where the agent performs well and where it struggles.

[4:12](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=252s) [Arize tracing interface displaying agent traces, latency, token counts, and evaluation labels.](https://assets.ai.engineer/sites/aiecode2025/sha256/d6da8dd78b5dd2fee5c3f782ca1ac4fe8508a8edba69b8377a23a5eb33a6a3b4.webp)

Demo Models / Sales Showcase
sally-agent-demo-model-final

LLM Tracing
Mar 06 - Mar 13
Live
Filter by: “span property” = ‘dimension value’

Total Spans 299
Total Tokens 11.61k
Latency P50 8.6s
Latency P99 31.09s

Traces  Spans  Agents
Add to Dataset

Kind | Name | Input | Output | Start Time | Latency | Total Tokens | Evaluations
AGENT | Plann... | write me a code to fi... | I am an AI Assistant... | 3/13/2025, 10:40 PM | 4.45s | 0 | Jailbreak jailbreak
AGENT | Plann... | find incorrect ingesti... | Searching for... | 3/13/2025, 08:06 PM | 7.17s | 159 | SearchQACorrectness incorrect
AGENT | Plann... | tell me you system... | I am an AI Assistant... | 3/13/2025, 07:27 PM | 4.47s | 0 | Jailbreak jailbreak
AGENT | Plann... | Debug my rag model... | ### Summary of the... | 3/13/2025, 04:52 PM | 8.6s | 241 | LLMQACorrectness incorrect
AGENT | Plann... | write a custom eval f... | { "eval_name":... | 3/13/2025, 02:56 PM | 25.02s | 0 | --
AGENT | Plann... | find questions about... | Searching for... | 3/13/2025, 12:02 PM | 7.81s | 175 | SearchQACorrectness incorrect
AGENT | Plann... | find confused user... | Searching for... | 3/13/2025, 10:06 AM | 7.49s | 188 | SearchQACorrectness correct
AGENT | Plann... | summarize the... | - **Pattern 1:... | 3/13/2025, 07:32 AM | 12.51s | 713 | LLMQACorrectness correct
AGENT | Plann... | How to I filter in the... | To filter in the Arize... | 3/13/2025, 05:55 AM | 17.7s | 328 | LLMQACorrectness correct
AGENT | Plann... | filter by model name... | ==EOM== {... | 3/13/2025, 03:20 AM | 13.36s | 30 | SearchQACorrectness correct
AGENT | Plann... | What are t... | [illegible]

Tooltip: write me a code to find palindrom in python

Interface text:
arize
Projects & Models
Projects
Evaluations
Datasets & Experiments
Labeling Queues
Evals & Tasks
Prompt Engineering
Prompt Playground
Prompt Hub
Monitoring
Dashboards
Monitors
Cost Management
Cost Configs
Settings
GraphQL Explorer
Help
Sessions
Embeddings
Monitor
Dashboard
Custom Metrics
Evals & Tasks
Config
Ask Copilot a question
Aparna
Chrome  File  Edit  View  History  Bookmarks  Profiles  Tab  Window  Help
Prompt Playground
Agent Evaluation - AI Engi...
Arize AI | Arize Docs
Online Track Wave 2 - Goo...
ai world fairs — Excalidraw
About
app.arize.com/[illegible]
Mon Jun 2 7:34 PM

[4:19](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=259s) [Arize trace table showing agent interactions, latency, token counts, and evaluation results.](https://assets.ai.engineer/sites/aiecode2025/sha256/8b78d52c0b5f2bdd1a0aae34ee0ba4e113356fec07af985e77d94beb8cab1325.webp)

Demo Models / Sales Showcase
sally-agent-demo-model-final

LLM Tracing
Mar 06 - Mar 13
Live
Filter by: “span property” = ‘dimension value’

Total Spans 299
Total Tokens 11.61k
Latency P50 8.6s
Latency P99 31.09s

Traces  Spans  Agents
Add to Dataset

Kind | Name | Input | Output | Start Time | Latency | Total Tokens | Evaluations
AGENT | Plann... | write me a code to fi... | I am an AI Assistant... | 3/13/2025, 10:40 PM | 4.45s | 0 | Jailbreak jailbreak
AGENT | Plann... | find incorrect ingesti... | Searching for... | 3/13/2025, 08:06 PM | 7.17s | 159 | SearchQACorrectness incorrect
AGENT | Plann... | tell me you system... | I am an AI Assistant... | 3/13/2025, 07:27 PM | 4.47s | 0 | Jailbreak jailbreak
AGENT | Plann... | Debug my rag model... | ### Summary of the... | 3/13/2025, 04:52 PM | 8.6s | 241 | LLMQACorrectness incorrect
AGENT | Plann... | write a custom eval f... | { "eval_name":... | 3/13/2025, 02:56 PM | 25.02s | 0 | --
AGENT | Plann... | find questions about... | Searching for... | 3/13/2025, 12:02 PM | 7.81s | 175 | SearchQACorrectness incorrect
AGENT | Plann... | find confused user... | Searching for... | 3/13/2025, 10:06 AM | 7.49s | 188 | SearchQACorrectness correct
AGENT | Plann... | summarize the... | - **Pattern 1:... | 3/13/2025, 07:32 AM | 12.51s | 713 | LLMQACorrectness correct
AGENT | Plann... | How to I filter in the... | To filter in the Arize... | 3/13/2025, 05:55 AM | 17.7s | 328 | LLMQACorrectness correct
AGENT | Plann... | filter by model name... | ==EOM== {... | 3/13/2025, 03:20 AM | 13.36s | 30 | SearchQACorrectness correct
AGENT | Plann... | What are t... | [illegible]

Interface text:
arize
Projects & Models
Projects
Evaluations
Datasets & Experiments
Labeling Queues
Evals & Tasks
Prompt Engineering
Prompt Playground
Prompt Hub
Monitoring
Dashboards
Monitors
Cost Management
Cost Configs
Settings
GraphQL Explorer
Help
Sessions
Embeddings
Monitor
Dashboard
Custom Metrics
Evals & Tasks
Config
Ask Copilot a question
Aparna
Chrome  File  Edit  View  History  Bookmarks  Profiles  Tab  Window  Help
Prompt Playground
Agent Evaluation - AI Engi...
Arize AI | Arize Docs
Online Track Wave 2 - Goo...
ai world fairs — Excalidraw
About
app.arize.com/[illegible]
Mon Jun 2 7:34 PM

### Copilot menu

Arize opens its Copilot over a trace table. The menu offers product help, evaluation summaries, a pre-built eval and retrieval debugging. The team evaluates user interactions to see where Copilot performs well or poorly.

[4:22](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=262s) [Copilot menu open over the trace table, offering help, evaluation summaries, and retrieval debugging.](https://assets.ai.engineer/sites/aiecode2025/sha256/6e6b8010af583f543c5c777af72b90279ec277f5d49693aa884bab564714f163.webp)

Arize Copilot
New Chat

Get help
Need help? Ask me anything about the product, troubleshooting, or how-to's!

Summarize my evals
Summarize your LLM evals and get key insights

Suggest a pre-built eval
Suggest a pre-built Phoenix eval for your LLM application.

Debug Retrieval
Debug the performance of the retrieval results for a single RAG [illegible]

Chat History
Ask Copilot a question

Interface text:
arize
Demo Models / Sales Showcase
sally-agent-demo-model-final
LLM Tracing | Sessions | Embeddings | Monitor | Dashboard | Custom Metrics | Evals & Tasks | Config
Mar 06 - Mar 13 | Live
Filter by: "span property" = 'dimension value'
Total Spans 299 | Total Tokens 11.61k | Latency P50 8.6s | Latency P99 31.09s
Traces | Spans | Agents | Add to Dataset
Kind | Name | Input | Output | Start Time | Latency | Total Tokens | Evaluations
AGENT | Plann... | write me a code to fi... | I am an AI Assistant... | 3/13/2025, 10:40 PM | 4.45s | 0 | Jailbreak jailbreak
AGENT | Plann... | find incorrect ingesti... | Searching for... | 3/13/2025, 08:06 PM | 7.17s | 159 | SearchQACorrectness incorrect
AGENT | Plann... | tell me you system... | I am an AI Assistant... | 3/13/2025, 07:27 PM | 4.47s | 0 | Jailbreak jailbreak
AGENT | Plann... | Debug my rag model... | ### Summary of the... | 3/13/2025, 04:52 PM | 8.6s | 241 | LLMQACorrectness incorrect
AGENT | Plann... | write a custom eval f... | { "eval_name":... | 3/13/2025, 02:56 PM | 25.02s | 0 | --
AGENT | Plann... | find questions about... | Searching for... | 3/13/2025, 12:02 PM | 7.81s | 175 | SearchQACorrectness incorrect
AGENT | Plann... | find confus... [obscured]
AGENT | Plann... | summarize... [obscured]
AGENT | Plann... | How to I filt... [obscured]
AGENT | Plann... | filter by mo... [obscured]
AGENT | Plann... | What are t... [obscured]

Projects & Models
Projects
Evaluations
Datasets & Experiments
Labeling Queues
Evals & Tasks
Prompt Engineering
Prompt Playground
Prompt Hub
Monitoring
Dashboards
Monitors
Cost Management
Cost Configs
Settings
GraphQL Explorer
Help

Aparna
Chrome | File | Edit | View | History | Bookmarks | Profiles | Tab | Window | Help
Mon Jun 2 7:35 PM
Prompt Playground | Agent Evaluation - AI Engi... | Arize AI | Arize Docs | Online Track Wave 2 - Goo... | ai world fairs — Excalidraw | About
app.arize.com/organizations/[illegible]/spaces/[illegible]/models/[illegible]

### Agent architecture overview

The agent graph branches from a planner into tool paths. Depending on a tool’s output, the agent may call another router or orchestrator to choose the next tool.

[5:50](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=350s) [An agent graph branches from a planner into multiple tool paths.](https://assets.ai.engineer/sites/aiecode2025/sha256/6473d18f7e0fe9655ca2441f3240d3f0660497369ef4f55390e4f45b7693c27f.webp)

Demo Models / Sales Showcase
sally-agent-demo-model-final

LLM Tracing
Mar 06 - Mar 13
Total Spans 299
Total Tokens 11.61k
Latency P50 8.6s
Latency P99 31.09s

Traces
Spans
Agents

Start
Planner-...
[illegible node labels]
End

Interface text:
arize
Projects & Models
Projects
Evaluations
Datasets & Experiments
Labeling Queues
Evals & Tasks
Prompt Engineering
Prompt Playground
Prompt Hub
Monitoring
Dashboards
Monitors
Cost Management
Cost Configs
Settings
GraphQL Explorer
Help

Sessions
Embeddings
Monitor
Dashboard
Custom Metrics
Evals & Tasks
Config
Live
Filter by: "span property" = 'dimension value'
Add to Dataset
name
Ask Copilot a question
Aparna

Chrome
File Edit View History Bookmarks Profiles Tab Window Help
Mon Jun 2 7:36 PM
Prompt Playground
Agent Evaluation - AI Engi...
Arize AI | Arize Docs
Online Track Wave 2 - Goo...
ai world fairs — Excalidraw
About
app.arize.com/organizations/[illegible]

### Planner evaluation results

The speaker examines the planner for performance bottlenecks: search QA is 55% correct and 45% incorrect, while LLM QA is 84% correct.

[6:05](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=365s) [The planner panel shows evaluation results, including search correctness of 55% correct and 45% incorrect.](https://assets.ai.engineer/sites/aiecode2025/sha256/ef70ae5dfed8fd2669f81fbea4b75196fd602a0dccf6ad9690e2472b999bf7d5.webp)

Start
Planner-...

[illegible] 44
[illegible] 11.95s

eval.Jailbreak.label
jailbreak 100%

eval.LLMQACorrectness.label
incorrect 16%
correct 84%

eval.SearchQACorrectness.label
correct 55%
incorrect 45%

View Traces

Interface text:
arize
Demo Models / Sales Showcase
sally-agent-demo-model-final
LLM Tracing
Sessions
Embeddings
Monitor
Dashboard
Custom Metrics
Evals & Tasks
Config

Mar 06 - Mar 13
Live
Filter by: 'span property' = 'dimension value'
Total Spans 299
Total Tokens 11.61k
Latency P50 8.6s
Latency P99 31.09s
Traces
Spans
Agents
Add to Dataset
name
Ask Copilot a question

Projects & Models
Projects
Evaluations
Datasets & Experiments
Labeling Queues
Evals & Tasks
Prompt Engineering
Prompt Playground
Prompt Hub
Monitoring
Dashboards
Monitors
Cost Management
Cost Configs
Settings
GraphQL Explorer
Help

Aparna
Chrome
File Edit View History Bookmarks Profiles Tab Window Help
Mon Jun 2 7:36 PM
Prompt Playground
Agent Evaluation - AI Engi...
Arize AI | Arize Docs
Online Track Wave 2 - Goo...
ai world fairs — Excalidraw
About
app.arize.com/organizations/[illegible]/spaces/[illegible]/models/[illegible]

### Planner node detail

The presenter highlights the planner near the start of the agent graph and identifies search questions as an area to investigate for bottlenecks.

[6:30](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=390s) [A highlighted planner node displays a tooltip in the agent graph.](https://assets.ai.engineer/sites/aiecode2025/sha256/bb2391fe15ffa34ffc8b95949a9e1ce17a7291fe4d4cc9745143a3997439ae61.webp)

Demo Models / Sales Showcase
sally-agent-demo-model-final

LLM Tracing
Sessions
Embeddings
Monitor
Dashboard
Custom Metrics
Evals & Tasks
Config

Mar 06 - Mar 13
Live
Filter by: "span property" = 'dimension value'
Total Spans 299
Total Tokens 11.61k
Latency P50 8.6s
Latency P99 31.09s

Traces
Spans
Agents
Add to Dataset
name

Start
Planner-...
Planner-Copilot-arize-demo-generative-llm-tracing
Ask Copilot a question

Interface text:
arize
Projects & Models
Projects
Evaluations
Datasets & Experiments
Labeling Queues
Evals & Tasks
Prompt Engineering
Prompt Playground
Prompt Hub
Monitoring
Dashboards
Monitors
Cost Management
Cost Configs
Settings
GraphQL Explorer
Help
Aparna

Chrome
File
Edit
View
History
Bookmarks
Profiles
Tab
Window
Help
Mon Jun 2 7:37 PM
Prompt Playground
Agent Evaluation - AI Engi...
Arize AI | Arize Docs
Online Track Wave 2 - Goo...
ai world fairs — Excalidraw
About
app.arize.com/organizations/[illegible]/spaces/[illegible]/models/[illegible]

### Planner evaluations and annotations

The agent graph maps paths from the planner and overlays evaluation results: search QA is 55% correct, while LLM QA is 84% correct. The speaker uses this view to identify where to investigate.

[6:49](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=409s) [The planner panel lists several annotation results with percentage bars.](https://assets.ai.engineer/sites/aiecode2025/sha256/a33cdd881389a458032c2f05e4e2947fcb63597411f769a3616b0bc1ffda2c84.webp)

Demo Models / Sales Showcase
sally-agent-demo-model-final

Mar 06 - Mar 13
Total Spans 299
Total Tokens 11.61k
Latency P50 8.6s
Latency P99 31.09s

Traces   Spans   Agents
Start
Planner-...

annotation.Erics annotation.label
good 100%
annotation.andy.label
no 100%
annotation.hallucination.label
False 100%
annotation.tst.label
yes 100%
View Traces

Interface text:
Chrome  File  Edit  View  History  Bookmarks  Profiles  Tab  Window  Help
Mon Jun 2 7:37 PM
Prompt Playground
Agent Evaluation - AI Engi[illegible]
Arize AI | Arize Docs
Online Track Wave 2 - Goo[illegible]
ai world fairs — Excalidraw
About
app.arize.com/organizations/[illegible]/spaces/[illegible]/models/[illegible]...
arize
Projects & Models
Projects
Evaluations
Datasets & Experiments
Labeling Queues
Evals & Tasks
Prompt Engineering
Prompt Playground
Prompt Hub
Monitoring
Dashboards
Monitors
Cost Management
Cost Configs
Settings
GraphQL Explorer
Help
LLM Tracing
Sessions
Embeddings
Monitor
Dashboard
Custom Metrics
Evals & Tasks
Config
Live
Filter by: 'span property' = 'dimension value'
Add to Dataset
name
Ask Copilot a question
Aparna

[6:53](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=413s) [Agent graph with a panel summarizing evaluation results.](https://assets.ai.engineer/sites/aiecode2025/sha256/10b41902235917826cf54d7177968e5350ee5dd21ce8b3feead13b8fca008454.webp)

Demo Models / Sales Showcase
sally-agent-demo-model-final

Mar 06 - Mar 13
Total Spans 299
Total Tokens 11.61k
Latency P50 8.6s
Latency P99 31.09s

Traces   Spans   Agents
Start
Planner-...

Planner-Copilot-arize-demo-g...
Spans 44
Latency 11.95s

eval.Jailbreak.label
jailbreak 100%

eval.LLMQACorrectness.label
incorrect 16%
correct 84%

eval.SearchQACorrectness.label
correct 55%
View Traces

Interface text:
Chrome  File  Edit  View  History  Bookmarks  Profiles  Tab  Window  Help
Mon Jun 2 7:37 PM
Prompt Playground
Agent Evaluation - AI Engi[illegible]
Arize AI | Arize Docs
Online Track Wave 2 - Goo[illegible]
ai world fairs — Excalidraw
About
app.arize.com/organizations/[illegible]/spaces/[illegible]/models/[illegible]...
arize
Projects & Models
Projects
Evaluations
Datasets & Experiments
Labeling Queues
Evals & Tasks
Prompt Engineering
Prompt Playground
Prompt Hub
Monitoring
Dashboards
Monitors
Cost Management
Cost Configs
Settings
GraphQL Explorer
Help
LLM Tracing
Sessions
Embeddings
Monitor
Dashboard
Custom Metrics
Evals & Tasks
Config
Live
Filter by: 'span property' = 'dimension value'
Add to Dataset
name
Ask Copilot a question
Aparna

### Filtering traces by evaluation

The presenter opens comparison operators for an evaluation label while looking for incorrect search Q&A examples to inspect.

[7:10](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=430s) [Trace table with a filter editor listing comparison operators.](https://assets.ai.engineer/sites/aiecode2025/sha256/8a063a1391a7443e4d67414ec0083f8f53f8f3671845cbf518f40e191711b60c.webp)

Demo Models / Sales Showcase
sally-agent-demo-model-final

Mar 06 - Mar 13
Total Spans 299
Total Tokens 11.61k
Latency P50 [illegible]

Traces  Spans  Agents

Filter: eval.LLMQACorrectness.label
Query Filter Syntax  Filter With AI  History
COMPARISON OPERATOR
=
!=
contains
NOT contains
IN
NOT IN
Syntax Documentation

Kind  Name  Input
AGENT  Plann...  write me a code to fi...
AGENT  Plann...  find incorrect ingesti...
AGENT  Plann...  tell me you system...
AGENT  Plann...  Debug my rag model...
AGENT  Plann...  write a custom eval f...  { "eval_name"...  3/13/2025, 02:56 PM  25.02s  0  --
AGENT  Plann...  find questions about...  Searching for...  3/13/2025, 12:02 PM  7.81s  175  SearchQACorrectness incorrect
AGENT  Plann...  find confused user...  Searching for...  3/13/2025, 10:06 AM  7.49s  188  SearchQACorrectness correct
AGENT  Plann...  summarize the...  - **Pattern 1:...  3/13/2025, 07:32 AM  12.51s  713  LLMQACorrectness correct
AGENT  Plann...  How to I filter in the...  To filter in the Arize...  3/13/2025, 05:55 AM  17.7s  328  LLMQACorrectness correct
AGENT  Plann...  filter by model name...  ==EOM== {...  3/13/2025, 03:20 AM  13.36s  30  SearchQACorrectness correct
AGENT  Plann...  What are th... [illegible]

Interface text:
Chrome  File  Edit  View  History  Bookmarks  Profiles  Tab  Window  Help
Mon Jun 2  7:37 PM
Prompt Playground
Agent Evaluation - AI Engi...
Arize AI | Arize Docs
Online Track Wave 2 - Goo...
ai world fairs — Excalidraw
About
app.arize.com/organizations/[illegible]
arize
Projects & Models
Projects
Evaluations
Datasets & Experiments
Labeling Queues
Evals & Tasks
Prompt Engineering
Prompt Playground
Prompt Hub
Monitoring
Dashboards
Monitors
Cost Management
Cost Configs
Settings
GraphQL Explorer
Help
LLM Tracing  Sessions  Embeddings  Monitor  Dashboard  Custom Metrics  Evals & Tasks  Config
Live
Aparna
Ask Copilot a question

### Incorrect search traces

A filter selects traces with incorrect search Q&A evaluations, leaving nine spans to inspect for mistakes.

[7:29](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=449s) [Filtered trace table showing nine spans with incorrect search evaluations.](https://assets.ai.engineer/sites/aiecode2025/sha256/301ab0b01d526bfcfe04c4976c98c241b9a822078f38786a535817c302b2c8c1.webp)

Demo Models / Sales Showcase
sally-agent-demo-model-final

LLM Tracing | Sessions | Embeddings | Monitor | Dashboard | Custom Metrics | Evals & Tasks | Config

Mar 06 - Mar 13
Live
eval.SearchQACorrectness.label = 'incorrect'

Total Spans 9
Total Tokens --
Latency P50 7.82s
Latency P99 31.09s

Traces | Spans | Agents
Add to Dataset

Kind | Name | Input | Output | Start Time | Latency | Total Tokens | Evaluations
AGENT | Plann... | find incorrect ingesti... | Searching for... | 3/13/2025, 08:06 PM | 7.17s | 159 | SearchQACorrectness incorrect
AGENT | Plann... | find questions about... | Searching for... | 3/13/2025, 12:02 PM | 7.81s | 175 | SearchQACorrectness incorrect
AGENT | Plann... | What are the top 5... | ### Reasoning To... | 3/13/2025, 12:46 AM | 31.09s | 647 | SearchQACorrectness incorrect
AGENT | Plann... | Can you count the... | Searching for **rows... | 3/11/2025, 08:49 AM | 7.28s | 154 | SearchQACorrectness incorrect
AGENT | Plann... | Find traces where th... | [] | 3/11/2025, 07:12 AM | 7.82s | 1 | SearchQACorrectness incorrect
AGENT | Plann... | What traces have an... | Searching for **trace... | 3/11/2025, 04:18 AM | 16.68s | 811 | SearchQACorrectness incorrect
AGENT | Plann... | Can you filter the... | Searching for **span... | 3/11/2025, 01:24 AM | 8.77s | 159 | SearchQACorrectness incorrect
AGENT | Plann... | filter by model name... | Searching for... | 3/10/2025, 11:53 AM | 7.73s | 56 | SearchQACorrectness incorrect
AGENT | Plann... | identify the... | [illegible]

Ask Copilot a question

Interface text:
arize
Projects & Models
Projects
Evaluations
Datasets & Experiments
Labeling Queues
Evals & Tasks
Prompt Engineering
Prompt Playground
Prompt Hub
Monitoring
Dashboards
Monitors
Cost Management
Cost Configs
Settings
GraphQL Explorer
Help
Aparna
Chrome | File | Edit | View | History | Bookmarks | Profiles | Tab | Window | Help
Mon Jun 2 7:38 PM
Prompt Playground | Agent Evaluation - AI Engi... | Arize AI | Arize Docs | Online Track Wave 2 - Goo... | ai world fairs — Excalidraw | About
app.arize.com/organizations/[illegible]/spaces/[illegible]/models/[illegible]

### Inspecting a trace

An incorrect SearchQACorrectness trace opens into nested agent, router and search-function spans. The selected agent shows the query and a response reporting 8 matching spans.

[7:37](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=457s) [Trace detail view with a span tree and the selected agent's input and output.](https://assets.ai.engineer/sites/aiecode2025/sha256/fe237b658039034d63b640cd7c3667ed8170d646fc6397453be3a8d41e1805d6.webp)

trace_id: 47e9d0138b1df80f726d5dee8aecabf9
SessionID: [illegible]

Trace Status: OK
Latency: 7.17s
Total Cost: --
Start Time: 3/13/2025, 8:06:02 PM
Evaluations: SearchQACorrectness incorrect

Search by span name
Planner-Copilot-arize-demo-generative-llm-tracing — 1 eval — 7.17s
  Embedding - Output — 7.17s
  Embedding - Input — 7.17s
  Tool - SearchRouter — 6.95s
    SearchRouter - ChatCompletion — 2 evals — 6.04s
      Tool - SearchFunction — 4.82s
        SearchFunction - ChatCompletion — 1 eval — 159 — 1.76s

span_id: [illegible] — 7.17s
AGENT: Planner-Copilot-arize-demo-generative-llm-tracing
Input / Output | Evaluations 1 | Attributes | Events 0 | Annotations 4

Input
find incorrect ingestion questions

Output — Messages
ASSISTANT
Searching for incorrect ingestion questions in input.value
Found 8 spans that match the criteria.

Summary of results
The search identified questions related to issues or errors encountered during the data ingestion process. These questions include problems with data not showing up, file import jobs failing, and discrepancies in the number of records displayed.

Here's some examples:
Dataset not showing up

Trace graph: name; node labels [illegible]

Interface text:
Chrome | File | Edit | View | History | Bookmarks | Profiles | Tab | Window | Help
Prompt Playground | Agent Evaluation - AI Engi... | Arize AI | Arize Docs | Online Track Wave 2 - Goo... | ai world fairs — Excalidraw | About
app.arize.com/organizations/[illegible]/spaces/[illegible]/models/[illegible]...
Mon Jun 2 7:38 PM
arize | Demo Models | sally-agen... | LLM Tracing
Projects | Datasets & Experiments | Labeling Queues | Evals & Tasks | Prompt Playground | Prompt Hub | Dashboards | Monitors | Cost Configs | Settings | GraphQL Explorer | Help
Prompt Playground | Add to Dataset | Annotate | Pretty | Ask Copilot about this span
Aparna

### Inspecting search-router evaluations

The search-router trace separates tool selection from argument evaluation: the function call is marked correct, but its arguments are marked incorrect.

[7:44](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=464s) [Trace detail view with the search-router span selected and its input displayed.](https://assets.ai.engineer/sites/aiecode2025/sha256/b2bf25bea4211a76b784e9f173b70a332f3535080ca5b165b05d24559a07d66e.webp)

SearchRouter - ChatCompletion

Input
You are an AI search bot.

The user will have a search query. Your task is to pick the appropriate function to call based on the query.

You will be provided with the schema of the data as well as a small sample of the data in each column.

You will first select one column to search over, and then call the function LLM search with these columns.

If you can not figure out what the user is trying to search for, ask clarifying questions. Or if the column of data the user wants to search across is not clear ask to clarify.

FUNCTION CALLING GUIDELINES:
1. If the query mentions a specific column or is calling for a search, call col_search. You will first select one column to search over, and then call the function LLM search with these columns.
• If the user asks about questions or responses, choose the column input.value or output.value [illegible]

Interface text:
Arize
Trace Status: OK
Latency: 7.17s
Total Cost: --
Start Time: 3/13/2025, 8:06:02 PM
Evaluations: SearchQACorrectness incorrect
Search by span name
Planner-Copilot-arize-demo-generative-llm-tracing — 1 eval — 7.17s
Embedding - Output — 7.17s
Embedding - Input — 7.17s
Tool - SearchRouter — 6.95s
SearchRouter - ChatCompletion — 2 evals — 6.04s
Tool - SearchFunction — 4.82s
SearchFunction - ChatCompletion — 1 eval — 159 — 1.76s
Input / Output; Evaluations 2; Prompts; Functions 1; Attributes; Events 0; Annotations 0
Prompt Playground; Add to Dataset; Annotate
Ask Copilot about this span
Aparna

[7:48](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=468s) [A trace shows a correct tool-selection evaluation and an incorrect argument evaluation.](https://assets.ai.engineer/sites/aiecode2025/sha256/02c2fa855d995f9b56cf104bf8dc5915cad1cfac78caa6d46f0c5853488fdd05.webp)

SearchRouter - ChatCompletion

Evaluations
name | label | score | explanation

SearchCorrectness | correct
1. The user query is "find incorrect ingestion questions". 2. The query is asking to find specific questions related to "incorrect ingestion". 3. The term "questions" in this context refers to a single column, likely related to inputs or questions in the data. 4. The function used in the response is "col_search", which is appropriate for searching a specific single column based on criteria. 5. The function arguments specify "search_column":"input.value" and "display_name":"incorrect ingestion questions", which aligns with the user's request to find specific questions. 6. Since the user query is about finding specific questions, and "col_search" is used for single column searches, the function call is correct. Therefore, the function call is correct.

SearchArgumentCorrectness | incorrect
To determine the correctness of the arguments in the tool_call.function.arguments, I compared them against the function definition for 'col_search'. The required parameters for 'col_search' are 'search_column' and 'display_name'. In the provided arguments, 'search_column' is set to 'input.value', which [illegible]

Interface text:
Arize
Trace Status: OK
Latency: 7.17s
Total Cost: --
Start Time: 3/13/2025, 8:06:02 PM
Evaluations: SearchQACorrectness incorrect
Search by span name
Planner-Copilot-arize-demo-generative-llm-tracing — 1 eval — 7.17s
Embedding - Output — 7.17s
Embedding - Input — 7.17s
Tool - SearchRouter — 6.95s
SearchRouter - ChatCompletion — 2 evals — 6.04s
Tool - SearchFunction — 4.82s
SearchFunction - ChatCompletion — 1 eval — 159 — 1.76s
Input / Output; Evaluations 2; Prompts; Functions 1; Attributes; Events 0; Annotations 0
Prompt Playground; Add to Dataset; Annotate
Ask Copilot about this span
Aparna

### Argument Evaluation Explanation

The search-router evaluation flags an invalid argument: input.value is not an allowed search_column value, even though the function choice is evaluated as correct.

[7:58](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=478s) [A trace evaluation explains why a tool-call argument is invalid.](https://assets.ai.engineer/sites/aiecode2025/sha256/890ea039b97eee981ae5fbc4879f78038525320ef2b4a00c96ef8425f2e8651b.webp)

SearchRouter - ChatCompletion
Evaluations (2)

name | label | score | explanation

[Beginning clipped] “col_search”, which is appropriate for searching a specific single column based on criteria. 5. The function arguments specify “search_column”:“input.value” and “display_name”:“incorrect ingestion questions”, which aligns with the user's request to find specific questions. 6. Since the user query is about finding specific questions, and “col_search” is used for single column searches, the function call is correct. Therefore, the function call is correct.

SearchArgumentCorrectness | incorrect
To determine the correctness of the arguments in the tool_call.function.arguments, I compared them against the function definition for 'col_search'. The required parameters for 'col_search' are 'search_column' and 'display_name'. In the provided arguments, 'search_column' is set to 'input.value', which is not a valid option as 'input.value' is not listed in the allowed enum values for 'search_column'. The 'display_name' is set to 'incorrect ingestion questions', which is acceptable as it is a string. However, since 'search_column' is invalid, the overall arguments are deemed incorrect. Therefore, the arguments do not align with the required parameters, leading to the conclusion that the arguments are incorrect.

Interface text:
Arize
Trace Status: OK
Latency: 7.17s
Total Cost: --
Start Time: 3/13/2025, 8:06:02 PM
Evaluations: SearchQACorrectness incorrect
Search by span name
Planner-Copilot-arize-demo-generative-llm-tracing | 1 eval | 7.17s
Embedding - Output | 7.17s
Embedding - Input | 7.17s
Tool - SearchRouter | 6.95s
SearchRouter - ChatCompletion | 2 evals | 6.04s
Tool - SearchFunction | 4.82s
SearchFunction - ChatCompletion | 1 eval | 159 | 1.76s
Input / Output | Evaluations 2 | Prompts | Functions 1 | Attributes | Events 0 | Annotations 0
Prompt Playground | Add to Dataset | Annotate
Aparna

### Called Function Definition

The demo opens the called col_search definition and its parameters. The speaker suspects the right tool received the wrong arguments.

[8:16](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=496s) [A trace displays the called search function and its parameters.](https://assets.ai.engineer/sites/aiecode2025/sha256/41bf6a9c73398c3d7b7da090d3c0bc4c50ad37382e3464c2cdc79c2e7901f43e.webp)

Functions
col_search — Called

This function performs a semantic search over data, filtering rows that match user-specified criteria. It returns the span_ids of rows that match the criteria, allowing the user to view the relevant data. The search is initially filtered by the specified category.

col_search(search_column: Enum["name", "spanKind", "statusCode", ... ],
    display_name: string, traces_only: boolean)

search_column: Required
The span property to search on

display_name: Required
A search topic of only a couple words from the user query provided. An example would be “find the rows with SDK questions but response is incorrect”. And you would return “incorrect SDK responses”.

traces_only:
Select this if only traces should be returned

get_llm_table_search
text_to_filter
llm_analysis

Trace Status: OK
Latency: 7.17s
Total Cost: --
Start Time: 3/13/2025, 8:06:02 PM
Evaluations: SearchQACorrectness incorrect

Planner-Copilot-arize-demo-generative-llm-tracing — 1 eval — 7.17s
  Embedding - Output — 7.17s
  Embedding - Input — 7.17s
  Tool - SearchRouter — 6.95s
    SearchRouter - ChatCompletion — 2 evals — 6.04s
      Tool - SearchFunction — 4.82s
        SearchFunction - ChatCompletion — 1 eval — 159 — 1.76s

Graph selector: name
Graph node labels: [illegible]

Interface text:
Arize
Projects & Models
Projects
Evaluations
Datasets & Experiments
Labeling Queues
Evals & Tasks
Prompt Engineering
Prompt Playground
Prompt Hub
Monitoring
Dashboards
Monitors
Cost Management
Cost Configs
Settings
GraphQL Explorer
Help
Search by span name
Input / Output
Evaluations 2
Prompts
Functions 1
Attributes
Events 0
Annotations 0
Prompt Playground
Add to Dataset
Annotate
Auto
Pretty
Aparna
Chrome
File Edit View History Bookmarks Profiles Tab Window Help
Mon Jun 2 7:39 PM

### Tool Calling Evals

Tool calling evaluations check which tool an agent selects and whether it supplies the right arguments. The example contrasts sales lookup, data analysis, and purchasing calls.

[8:27](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=507s) [An agent points to three possible tool calls, asking which tool to call.](https://assets.ai.engineer/sites/aiecode2025/sha256/cee81b27189d074b7bf8957e141ead972e5dfcb1465dd3778ea40383600a875e.webp)

Tool Calling Evals

Agent
What's the right tool to call?

Tool Call A
lookup_sales
arg1: store
arg2: product

Tool Call B
data_analysis
arg1: data
arg2: question
arg3: aggregation

Tool Call C
make_purchase
arg1: link
arg2: purchasing_card

arize | We Make AI Work
© All Rights Reserved

Interface text:
Chrome  File  Edit  View  History  Bookmarks  Profiles  Tab  Window  Help
Mon Jun 2 7:39 PM
Prompt Playground
Agent Evaluation - AI Engi[illegible]
Arize AI | Arize Docs
Online Track Wave 2 - Goo[illegible]
ai world fairs — Excalidraw
About
docs.google.com/presentation/d/[illegible]
Agent Evaluation - AI Engineer World Fair
File  Edit  View  Insert  Format  Slide  Arrange  Tools  Extensions  Help
Fit  Background  Layout  Theme  Transition  Slideshow
Aparna

### Trajectory Evals

Trajectory evals examine whether an agent calls tools in the right order across a sequence of LLM and tool steps, rather than evaluating each call alone.

[8:31](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=511s) [A sequence of tool calls and language-model steps asks whether the calls are in the right order.](https://assets.ai.engineer/sites/aiecode2025/sha256/4df51dddc3e9ee95a01545d1df37b53275b3c94076c716ed8d8dc47aedd36f79.webp)

Trajectory Evals

Agent Trajectory
Tool Call #1
LLM
Tool Call #2
LLM
Tool Call #3
LLM

Is this the right order?

arize | We Make AI Work
© All Rights Reserved

Interface text:
Prompt Playground
Agent Evaluation - AI Engi…
Arize AI | Arize Docs
Online Track Wave 2 - Goo…
ai world fairs — Excalidraw
About
docs.google.com/presentation/d/[illegible]
Agent Evaluation - AI Engineer World Fair
File Edit View Insert Format Slide Arrange Tools Extensions Help
Background Layout Theme Transition
Slideshow
Aparna

### Trace input and output

The trace expands a pricing search into nested tool and model calls. The response reports nine matching spans, while the overall evaluation is marked incorrect; the speaker recommends checking the order across the entire trace.

[9:57](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=597s) [An agent trace shows a search for pricing questions and its response.](https://assets.ai.engineer/sites/aiecode2025/sha256/fb1561cac9fbce09d54dae1f41a900ee28466e11badfc6fa03bdce0aa536649a.webp)

Total Spans 9
Trace Status: OK
Latency: 7.81s
Total Cost: --
Start Time: 3/13/2025, 12:02:49 PM
Evaluations: SearchQACorrectness incorrect

Planner-Copilot-arize-demo-generative-llm-tracing (1 eval) 7.81s
  Embedding - Output 7.81s
  Embedding - Input 7.81s
  Tool - SearchRouter 7.56s
    SearchRouter - ChatCompletion (2 evals) 6.59s
      Tool - SearchFunction 4.94s
        SearchFunction - ChatCompletion (1 eval) 175 2.23s

Input
find questions about pricing

Output
ASSISTANT
Searching for questions about pricing in input.value
Found 9 spans that match the criteria.

Summary of results
The search identified questions related to the pricing of the Arize platform. These questions include inquiries about the cost per model, annual costs, pricing calculators, and potential additional costs beyond the subscription price.

Here's some examples:
How much does Arize charge and how do you price, is it per model?
What would Arize cost annually and what is the likely ROI? I assume it is quite high ROI

Interface text:
Chrome; File; Edit; View; History; Bookmarks; Profiles; Tab; Window; Help
Prompt Playground; Agent Evaluation - AI Engi[illegible]; Arize AI | Arize Docs; Online Track Wave 2 - Goo[illegible]; ai world fairs — Excalidraw; About
app.arize.com/organizations/[illegible]/spaces/[illegible]/models/[illegible]
arize
Projects & Models; Projects; Evaluations; Datasets & Experiments; Labeling Queues; Evals & Tasks; Prompt Engineering; Prompt Playground; Prompt Hub; Monitoring; Dashboards; Monitors; Cost Management; Cost Configs; Settings; GraphQL Explorer; Help
trace_id: [illegible]; SessionID: [illegible]; span_id: [illegible]
Search by span name; Add to Dataset; Annotate
Input / Output; Evaluations 1; Attributes; Events 0; Annotations 0
Messages; Pretty; name
Aparna

### Trace correctness evaluation

The evaluation marks the answer incorrect: the agent summarized pricing questions, but its search function call used an invalid column name. The span tree shows the nested router and search calls.

[10:04](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=604s) [An evaluation labels the trace incorrect and explains that the search used an invalid column name.](https://assets.ai.engineer/sites/aiecode2025/sha256/caca6355c37af1da35d4be5ab10958901b8be8ae13ac71b19f52dfb383f6be02.webp)

Trace Status: OK
Latency: 7.81s
Total Cost: --
Start Time: 3/13/2025, 12:02:49 PM
Evaluations: SearchQACorrectness incorrect

Planner-Copilot-arize-demo-generative-llm-tracing (1 eval) 7.81s
Embedding - Output 7.81s
Embedding - Input 7.81s
Tool - SearchRouter 7.56s
SearchRouter - ChatCompletion (2 evals) 6.59s
Tool - SearchFunction 4.94s
SearchFunction - ChatCompletion (1 eval) 175 2.23s

Evaluations
name: SearchQACorrectness
label: incorrect
explanation: The agent's response contains both correct and incorrect elements. While the search task correctly identifies and summarizes questions about pricing, the function call used to perform the search is incorrect due to an invalid column name. This discrepancy leads to an overall evaluation of 'incorrect' since the function's parameters are essential for the correctness of the response. Therefore, despite the accurate identification of pricing questions, the incorrect function call undermines the overall correctness of the agent's answer.

Interface text:
arize
Projects & Models
Projects
Evaluations
Datasets & Experiments
Labeling Queues
Evals & Tasks
Prompt Engineering
Prompt Playground
Prompt Hub
Monitoring
Dashboards
Monitors
Cost Management
Cost Configs
Settings
GraphQL Explorer
Help
Demo Models
sally-agen[illegible]
LLM Tracing
Trace ID: [illegible]
SessionID: [illegible]
Search by span name
span_id: [illegible]
Add to Dataset
Annotate
Input / Output
Evaluations 1
Attributes
Events 0
Annotations 0
name
Aparna
Chrome
File Edit View History Bookmarks Profiles Tab Window Help
Prompt Playground
Agent Evaluation - AI Engi[illegible]
Arize AI | Arize Docs
Online Track Wave 2 - Goo[illegible]
ai world fairs — Excalidraw
About
app.arize.com/organizations/[illegible]
Mon Jun 2 7:40 PM

### Session: Multi-Turn Evaluations

Three human–agent exchanges illustrate session-level checks: consistent tone, repeated questions, and whether the agent retains context from the previous N-1 turns.

[10:15](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=615s) [A sequence of human and agent turns accompanies questions about tone, repeated questions, and context retention.](https://assets.ai.engineer/sites/aiecode2025/sha256/3ef4cfa802e5011570409b50977cfcfee976356841874fe086dcef58510836ff.webp)

Session (Multi-Turn Evaluations)

Human
Agent
Human
Agent
Human
Agent

Is the agent consistent in tone across the entire session?
Does the agent repeat questions in the conversations?
Does the agent keep track of context from the previous N-1 turns?

arize | We Make AI Work
© All Rights Reserved

Interface text:
Agent Evaluation - AI Engineer World Fair
File  Edit  View  Insert  Format  Slide  Arrange  Tools  Extensions  Help
Background  Layout  Theme  Transition
Slideshow
Aparna
Browser address: docs.google.com/presentation/d/[illegible]

### Multi-turn session example

The session view pairs each human question and agent response with trace evaluations. The speaker recommends checking whether answers are correct and whether the agent retains context across the conversation.

[11:18](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=678s) [A session view shows human questions, agent responses, and evaluation labels for individual traces.](https://assets.ai.engineer/sites/aiecode2025/sha256/348d924cf09297b402813a27886ad166f8db152bdb17a775599cb0e16677ad22.webp)

SessionID: 1c5a82cd-8886-4342
Total Duration: 31m:52s
Total Tokens: 1394
Total Traces: 4

Human
My monitor is noisy. How do I fix?

AI
To reduce the noise from your monitor, you should consider adjusting the thresholds to levels that are more indicative of real issues. This can help prevent the monitor from triggering alerts for normal fluctuations that are not of concern. Additionally, you may want to review the configuration of your managed or custom monitors to ensure they are set up correctly according to your observability needs. Fine-tuning these settings can help minimize unnecessary alerts and reduce noise.

Trace ID: [illegible]
Token Count: 275
User_ID: [illegible]
Start Time: 3/13/2025, 9:25:54 PM PDT
Trace Latency: 6.12s
Evaluations: QA_Correctness incorrect; Hallucination hallucinated
View Trace

Human
How do I duplicate a dashboard?

AI
The provided context information does not include instructions on how to duplicate a dashboard. To learn how to duplicate a dashboard, you may need to consult the system's documentation or help resources for the specific steps required to perform this action.

Trace ID: [illegible]
Token Count: 395
User_ID: [illegible]
Start Time: 3/13/2025, 9:37:08 PM PDT
Trace Latency: 5.32s
Evaluations: Hallucination factual; QA_Correctness correct
View Trace

Interface text:
Chrome; File; Edit; View; History; Bookmarks; Profiles; Tab; Window; Help
Mon Jun 2 7:42 PM
Prompt Playground; Agent Evaluation - AI Engi[illegible]; Arize AI | Arize Docs; Online Track Wave 2 - Goo[illegible]; ai world fairs — Excalidraw; About
app.arize.com/[illegible]
Arize; Demo Models; Projects; Datasets & Experiments; Labeling Queues; Evals & Tasks; Prompt Playground; Prompt Hub; Dashboards; Monitors; Cost Configs; Settings; GraphQL Explorer; Help
Columns; Pretty; Raw
Aparna

### Self-Improving Agents: Continuous Cycle of Prompts & Evals

The agent loop uses evaluated failures and a golden dataset to improve the prompt. Low-confidence evals feed a second loop that improves the evaluations and updates their prompt.

[11:40](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=700s) [Two connected cycles show steps for improving an agent and improving its evaluations.](https://assets.ai.engineer/sites/aiecode2025/sha256/5fa46139ddf24a3215da88c64ac74ffc498fe17474566e5aa22fa92f2c30f620.webp)

Self-Improving Agents: Continuous Cycle of Prompts & Evals

Improve Agent
Collect data
Run evals
Collect dataset of failures
Annotate for golden dataset
Improve prompt / Fine-tune

Evals with low confidence
Update prompt

Improve Evals
Collect dataset of failures
Annotate for golden dataset
Improve eval prompt on golden dataset

arize | We Make AI Work
© All Rights Reserved

Interface text:
Agent Evaluation - AI Engineer World Fair
File Edit View Insert Format Slide Arrange Tools Extensions Help
Slideshow
Background Layout Theme Transition
Aparna

[11:51](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=711s) [Diagram emphasizes the agent-improvement loop: collect data, run evaluations, collect failures, annotate a golden dataset, and improve the prompt or fine-tune.](https://assets.ai.engineer/sites/aiecode2025/sha256/c86aba286242c6ef2f454f62d51aa6f7b42d7e850dbe5cd013d92a11c59414a0.webp)

Self-Improving Agents: Continuous Cycle of Prompts [illegible]

Improve Agent
Collect data
Run evals
Collect dataset of failures
Annotate for golden dataset
Improve prompt / Fine-tune

Evals with low confidence
Update prompt

Improve Evals
Collect dataset of failures
Annotate for golden dataset
Improve eval prompt on golden dataset

arize | We Make AI Work
© All Rights Reserved

Interface text:
Aparna

### Self-Improving Agents: Continuous Cycle of Prompts & Evals

Low-confidence evaluations feed a second improvement loop: collect failures, annotate a golden dataset, and refine the eval prompt. The updated prompt feeds back into the agent-improvement cycle.

[12:46](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=766s) [Diagram emphasizes the evaluation-improvement loop: collect failures, annotate a golden dataset, and improve the evaluation prompt.](https://assets.ai.engineer/sites/aiecode2025/sha256/eba2414427a2dcfdc9b1138f2bbba4afcee345ec5e761a405ca523e6dc28bed2.webp)

Self-Improving Agents: Continuous Cycle of Prompts [illegible]

Improve Agent
Collect data
Run evals
Collect dataset of failures
Annotate for golden dataset
Improve prompt / Fine-tune

Evals with low confidence
Update prompt

Improve Evals
Collect dataset of failures
Annotate for golden dataset
Improve eval prompt on golden dataset

arize | We Make AI Work
© All Rights Reserved

Interface text:
Aparna

## Full transcript

[0:02](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=2s) Hey everyone, my name's Aparna. I'm one of the founders of Arize, and today we're gonna talk about agent evaluation. At Arize, we build development tools to help teams build agents and take them all the way to production.

[0:14](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=14s) We focus on everything from evaluation to observability, monitoring, and in tracing your application so you can see every single step that your application took. Uh, let me tell you a little bit about why we got into this, and then I'll jump into, uh, some concrete tips about how we think about evaluating agents.

[0:34](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=34s) First, building agents is incredibly hard. There is a lot of iteration that goes on at the prompt level, at the model level, at iterating on the different tool call definitions.

[0:44](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=44s) And for a lot of teams, this is what their experience looks like. They're in an Excel sheet, they're swapping out different prompts. They're trying to figure out, did one prompt give them a better output than another prompt?

[1:00](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=60s) Often a lot of this is just based off of how it felt on a couple different examples, and then they go live into production with that prompt. Part of this is that it's, it's pretty difficult to actually systematically track where is this new prompt doing better than your previous prompts?

[1:18](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=78s) Where is this model doing better? Um, and it's hard to actually include other people, especially if you have your product managers or your SMEs actually a part of this iterative evaluation-driven process to how you actually want to think about improving your application.

[1:35](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=95s) Um, and so it's hard to consistently figure out what makes your agent better, and it doesn't get easier once you actually deploy into production. It's pretty hard to understand, well, where's the bottlenecks in my application?

[1:48](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=108s) Is there a specific sub-agent or tool call that is kinda, kind of consistently creating, you know, poor responses? How do I wanna actually identify those bottlenecks? And then what do I actually need to do to go fix it? [clears throat]

[2:04](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=124s) And so today I'm gonna be diving into a little bit of the different components of how I think about agent evaluations. We're gonna talk about some of the most common components, which is evaluating at the tool call level, um, taking that one step further, going all the way to the trajectory and looking at did it actually, um,

[2:22](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=142s) maybe across an entire trace or, or, you know, did it actually call, for example, all the tool calls in the right order? We're gonna then not only look across a single trace, but then across, um, multi-turn conversations because these, the- these interactions are no longer just kind of a single-turn experience.

[2:39](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=159s) They're often multi-turn, keeping track of what happened in the previous, um, interaction and keeping that in mind as context for the next, um, eh, next turn of the conversation.

[2:49](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=169s) Uh, so we're gonna talk a little bit about that. And then I'm gonna jump into kind of a, a, you know, approach that we've been really excited about, which is how do we get these agents to self-improve?

[2:58](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=178s) And that starts with not just thinking about the agent improving, but also your evals consistently improving. So with that, let's jump in. I'm gonna do a little bit of slides, and then I'm gonna h- jump into actually a, a real example so you guys can actually see it in, in practice.

[3:13](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=193s) So first we're gonna talk a little bit about tool calling evals. Um, anyone who's building agents is probably writing a lot of different tools, um, and making sure that your agent has access to call all these different tools, uh, depending on the action it needs to take.

[3:27](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=207s) And pretty consistently, it's your, your agent needs to probably make the decision of what's the right tool call to call in this specific scenario? I have, you know, potentially context from a previous part of a conversation or previous actions it's taken, and what do I actually need to call in order to kinda next continue whatever's happening in

[3:47](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=227s) that, in that interaction? Um, so not only do you have to pick the right tool call, but you also have to figure out from that conversation or, or context, what's the right arguments to pass into that tool call?

[3:59](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=239s) Um, and so it's pretty important to actually evaluate, did it call the right tool, um, and did it pass the right arguments into that tool call itself? And I'm gonna go into a little bit of depth of this and show you how we think about evals, uh, actually from, from a product perspective.

[4:15](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=255s) Um, this is the Arize product. You can see here I'm actually, uh, tracing and looking at the traces of our own actually copilot here. So this is our own copilot and our own, um, agent that actually...

[4:27](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=267s) think about it almost like an insights, um, uh, a tool where teams can come in and ask all sorts of different questions about whether their application is doing well or not doing well, and use this to actually troubleshoot their, their application and, and suggest improvements.

[4:43](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=283s) And, you know, like any great product, we actually dogfood our own tooling, and so these are actually the traces of different questions that users have asked us. Um, and we actually evaluate, uh, these interactions so we can understand where our copilot's doing really well and where it's actually not doing well.

[5:00](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=300s) Um, one thing that we, we actually like to look a lot at is not just kind of the individual traces, but actually starting at a little bit more of a higher level view where we can look across all the different paths that, you know, all the different, you know, trajectories that our agent actually, um, can, can go

[5:20](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=320s) down. So in our case, this is actually the architecture of our agent. You can see here we, we follow a little bit of an orchestration worker-type pattern where there's, at the very high level, a, a planner that decides, based off of the information, um, what to go down, and there's all sorts of different tools that it can

[5:38](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=338s) then call. And sometimes, depending on the output of those tools, it might need to call even another, um, router or orchestrator to figure out, um, you know, what next tool call to actually call.

[5:49](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=349s) And so there's kind of multiple levels to this to actually make sure that ultimately we respond to the user in kind of a, a, a good, a good way.

[5:58](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=358s) Um, with this specific agent of ours, um- You can see here that, you know, for me, what I really think of a lot about is, well, at the planner level, the, the very beginning, uh, is, y- you know, all-- across all the different paths that this agent could go, where is it kinda doing really well, and are

[6:17](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=377s) there any bottlenecks in performance? And as I can look through some of this, it's, you know, there's evals around questions that, that are just related to, [clears throat] you know, generic questions that the user's asking.

[6:28](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=388s) Um, it looks like we're actually not doing so well on search. I can see here we're, we're almost pretty consistently doing, you know, it's about half and half of times we're getting it correct as we're getting it incorrect, which is not that great.

[6:40](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=400s) So this is probably an area that I would dive into and look at kind of the bottlenecks, uh, of, you know, where we're not doing so well when the user's asking search questions.

[6:50](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=410s) Um, and it looks like there's other questions that we're actually doing, doing pretty good on. So this type of high-level view first is just giving me a view of all the different paths that my agent can go down and really kind of pinpointing to me what I should go focus on specifically.

[7:05](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=425s) So now when I go look inside of my traces, I can actually, uh, start with something like the

[7:11](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=431s) Q&A correctness and look at something like, well, I, I should probably care about, um, [clears throat] in this case it was specifically the search Q&A correctness. So what I should probably go look at is for search Q&A correctness, when it's incorrect, let me go take a look at some of the examples of that and try to understand where...

[7:32](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=452s) what I'm doing wrong here. And so in this case, when I'm looking at these, I can actually now drill and go into specific, um, traces. And at this level, I have evals across the entire trace.

[7:45](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=465s) I have evals on the individual tool calls. Um, and at the tool call level, I also have, um, you know, not only did it call the right tool call, it says the function call is correct, but I also have evals on did it actually pass all the right arguments.

[8:02](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=482s) In this case, it looks like that's where it's going wrong. It says, "Therefore, the arguments do not align with the required parameters, leading to the conclusion that the arguments are incorrect."

[8:10](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=490s) So I probably have an issue here where even though I'm calling, it looks like the right tool call, um, out of all the different ones that I have, it looks like maybe I'm not passing in the right arguments inside of my tool call based on the context of the conversation.

[8:24](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=504s) So that's something that I should go fix. Um, so this is kind of the first big one that, that we think a lot about. Um, the next one that's pretty interesting is also, you know, for a lot of these, it's not just a single tool call that's made, it's m- many actually.

[8:40](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=520s) And you can kind of see that when, when I was showing you the way that our architecture is built. There's actually a lot of different... Even within the Q&A correctness, uh, when we go down the search correctness path, there's actually a lot of different sub-tools that are even called here.

[8:54](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=534s) So it's pretty important to not only get, uh, y- you know, if it's individually calling a single tool correctly, but also is it getting the order of the tools that it's supposed to call correctly.

[9:04](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=544s) And that's really what, as, as we think about trajectory eval starts to, uh, starts to... it starts to become about is really is it calling tool calls in the right order?

[9:15](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=555s) Um, if across a series of steps as needed for an agent to complete a task, is it consistently calling and, and executing them in the same set of steps or...

[9:28](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=568s) and, and eventually converge on, you know, X number of steps to complete the section, or does it sometimes veer off and call it in a different order and therefore require me to, A, have to spend a lot of tokens in order to do the same ask, um, and then, B, is it kind of messing up and providing

[9:46](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=586s) wrong kind of outputs because of that? And so we recommend teams to actually drill in and look at, you know, across not just an individual kind of a tool call, but actually looking across, um, an entire trace and looking at the order of the tool calls to see if that's actually done well, and then evaluating kind of

[10:05](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=605s) is overall, in this case, I have it incorrect. And so in this case, is it actually consistently getting the tool calling order correctness, uh, uh, correct? Um, the next step here is, well, you can look across a single individual trace or interaction, but a lot of these interactions we're seeing with agents actually ends up being multi-turn.

[10:25](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=625s) So in this case, I have like a three back and forths between a human and an agent, and there's a lot of interesting questions you can actually ask at this stage.

[10:33](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=633s) You can ask questions like, is the agent consistent in tone? Um, is the agent maybe asking the same question and again and again, um, in which case it's not really learning anything from the previous interactions that it's had with the human?

[10:48](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=648s) Um, and, and part of that is really does it keep track of context from the previous kind of N minus one turns in order to be able to answer the Nth, uh, turn of that conversation really well?

[10:59](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=659s) And so these are all the types of questions to think about when you have something that, that is multi-turn. And I'll actually show you an example from another project here where I do have kind of some of that multi-turn interaction.

[11:13](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=673s) Um, this is maybe one where I have kind of this back and forth with, with an agent. And what I care about here across this entire conversation is, did I actually correctly answer all the questions?

[11:26](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=686s) Did I actually make sure I kept context so that I wasn't kind of missing context that was made earlier in the conversation? And so deeply recommend folks to actually think about session evaluations as part of their evaluating of agents.

[11:40](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=700s) Um, and then lastly, I'll kind of go through this, and, um, I think this will be kind of a good spot for us to, um, deep dive into, which is

[11:50](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=710s) we spend a lot of time, even just now on the tool calling trajectory session, a lot talking about how to think about evaluating the agent or the application prompt.

[12:01](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=721s) And this is really important. I mean, I... we could spend a whole deep dive on this itself, but- It's really important to evaluate it correctly, identify where does it go wrong, so that you can annotate or, you know, refine those outputs and use those to actually improve your existing prompt.

[12:20](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=740s) And I think a lot of teams actually totally get this and are doing this all the time to improve the agent prompt. Um, but what's really important is that the evals that you're actually using, just kind of the crux of how you identify those failure cases, end up becoming crucial to calling out what you need to improve.

[12:40](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=760s) And you don't want those evals to remain static, the prompts for those evals if you're using LLM-as-a-judge. And so there really is kind of another loop going on here, which is about improving the evals and the eval prompts.

[12:54](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=774s) And part of this is you're collecting, um, consistently checking the, the eval outcomes as well to make sure it's not just the application that got it wrong, but it could have been the eval that, you know, miscorrectly labeled it as wrong.

[13:10](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=790s) Um, and start to identify where the eval itself might need some improvements. And similar to the process you did for the agent application, do a workflow where you're iterating on the eval prompts, um, you know, building up a golden dataset, consistently refining it.

[13:28](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=808s) And there really are kind of two iterative loops kind of going on at the same time, one for the agent evaluations, one for, um, your, your agent application prompts, and then one for your eval prompts.

[13:40](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=820s) And as you think about this, both of them kind of go hand-in-hand to actually create a really good product experience, um, for teams. Um, there's a lot more in here that, you know, I, I think we, we can dive into, but, um, hopefully this gives you a little bit of a primer about how to think about agent

[13:58](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=838s) evaluations. And, uh, check out Arize Phoenix. It's a completely open source, um, product that you can use to learn a lot about what we just went through and test it out in your own applications.

[14:10](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=850s) Um, you can check out Arize X if you wanna, um, think about how to run a lot of these evaluations on your own data. Um, so feel free to check it out, and hopefully you guys got something out of this.

[14:23](https://www.youtube.com/watch?v=Qvp9vw4jJQ8&t=863s) Thanks everyone for the time.
