# Engineering Better Evals: Scalable LLM Evaluation Pipelines That Work

Dat Ngo · Aman Khan

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/spvXj9tnWAQ-engineering-better-evals-scalable-llm-evaluation) · [Video](https://www.youtube.com/watch?v=spvXj9tnWAQ)

## Engineering Better Evals: Scalable LLM Evaluation Pipelines That Work

Scalable evaluation combines traces, human labels, model judgments and code checks, then uses that signal to improve applications, diagnose agent paths and refine the evaluators themselves.

Familiarity with LLM calls, retrieval-augmented generation and basic Python will help; traces, evaluation scopes and agent trajectories are explained as they arise.

### When every trace needs many evaluations

[0:42](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=42s)

What happens when a production application needs many evaluations for every trace? Dat Ngo introduces the problem from his work as an AI architect at Arize AI, observing teams such as Reddit and Duolingo. Dat reports that Duolingo runs about 20 evaluations per trace. In that example, running evaluations, understanding their results and optimizing their cost are substantial operational concerns; he does not specify the workload, sampling policy or evaluator mix.

That scale changes the engineering question. An evaluation pipeline must produce useful signals repeatedly and economically, while giving the people building the application enough visibility to act on them. Dat approaches this as an engineer working on observability and evaluations, rather than as a developer advocate.

### Observe the application at the level your team owns

[2:24](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=144s)

**Observability answers what the application is actually doing.** Engineers may inspect traces and spans; an AI product manager may inspect conversations. Those are different views of the same application, and evaluations can operate at the conversation level as well as on individual calls.

Analytics also reflects organizational ownership. Dat describes a hub-and-spoke structure: a central LLM platform team owns infrastructure, including the model gateway or router, while business teams build applications around it.

| Team | Typical responsibility | Main signals |
|---|---|---|
| Central platform | Infrastructure and model gateway | Cost and latency |
| Business application | Application behavior and user experience | Task-specific evaluation results |

The dashboard brings these concerns together, but the people using it need different views and different definitions of success.

![Dark dashboard with line charts, a latency histogram, cost and latency plots, and a Hallucination Eval Score panel.](https://ai.engineer/api/talk-essay-media/4634b2f8d4c6841a898c1bf63b3f8d3f2fa94f60c09eed11067db60586b2be42)

An observability dashboard brings evaluation, cost, and latency charts together.

### Choose the cheapest method that captures the signal

[4:08](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=248s)

Nobody can manually inspect every production trace. **Evaluations turn that volume of activity into signals about what is going well and what is failing.** An LLM judge is one way to produce those signals, but it is not the whole toolbox.

Consider retrieval-augmented generation: a user asks a question, the application retrieves context, and a model generates an answer. Each relationship offers something to evaluate. Retrieval relevance, for example, compares the retrieved context with the original query: would this material help answer that question? This isolates a retrieval problem before treating the final answer as one undifferentiated success or failure.

Judging an answer is also a different task from generating it. Writing a summary of *War and Peace* requires producing the summary; assessing an existing summary asks whether that candidate is good or bad. That distinction helps explain why an LLM can be useful as a judge without making its judgments automatically trustworthy.

Autoregressive generation is not required for every evaluation. Dat describes encoder-only **BERT**-type evaluators as about 10 times cheaper and 10–100 times faster than the larger-model alternative. He supplies no task, comparison model, hardware or quality threshold for those estimates, so they motivate testing a smaller evaluator rather than predicting its performance on a particular workload.

Human feedback supplies another kind of signal. Users can report whether an interaction was helpful, and developers or domain experts can label examples themselves. A **golden dataset** collects judgments the team trusts. Automated judges supply scale; human-graded examples provide a quality reference.

To connect the two, run the LLM judge on that golden dataset and compare its labels with the trusted labels. The question becomes concrete: can this evaluator approximate the judgments the team expects? Disagreements give you evidence for quantifying and tuning the judge, rather than accepting its first prompt as a reliable measuring instrument.

Some questions need neither a model nor a human. In the code-evaluation demonstration, Dat checks properties such as keyword containment, regex matching and whether an output parses as JSON. These are deterministic requirements, so ordinary code can capture the signal directly. A Python implementation of those checks looks like this:

```python
import json
import re


def evaluate_output(text: str) -> dict[str, bool]:
    try:
        json.loads(text)
        parses_as_json = True
    except json.JSONDecodeError:
        parses_as_json = False

    return {
        "contains_hotel": "hotel" in text.casefold(),
        "has_booking_id": bool(re.search(r"\bBK-\d{4}\b", text)),
        "parses_as_json": parses_as_json,
    }


output = '{"hotel": "Harbor Inn", "booking_id": "BK-1042"}'
scores = evaluate_output(output)
```

These checks establish syntax and string properties, not whether the hotel exists or the booking is correct. Choose code, human labels or a model according to the question the evaluator must answer.

### Improve the application and the evaluator separately

[8:31](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=511s)

The application-improvement loop is familiar:

1. Collect traces and other observations.
2. Run evaluations to locate successes and failures.
3. Inspect failures to identify causes—for example, a hallucinated answer caused by a faulty retrieval strategy or agent behavior.
4. Annotate examples to check the evaluation results.
5. Change the prompt, model or agent orchestration, then repeat.

The evaluation is the instrument that tells you where to intervene.

**The instrument needs its own improvement loop.** A judge may flag a correct answer as a hallucination, or approve an answer that is wrong. Periodically annotate evaluated examples, collect those mistaken judgments, and revise the evaluation prompt. Instructions that initially seemed adequate may turn out to be vague, obscure or insufficiently specific as the application changes.

These loops reinforce each other: better evaluations make application changes easier to assess, and new application behavior exposes weaknesses in the evaluations. Dat emphasizes iteration speed, contrasting four cycles in a month with two. His description of exponential improvement is an argument for faster learning, not a measured growth rate.

### Evaluate the component, workflow, route and session

[10:34](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=634s)

Evaluation structure should follow application structure. Dat uses [Booking.com’s Trip Planner](https://news.booking.com/bookingcom-launches-new-ai-trip-planner-to-enhance-travel-planning-experience/) as a travel-agent example, describing it as revenue-driving and discussing flights, hotels and itineraries. A routed application consists of multiple components, so its evaluation can be as complex as its orchestration.

Start with an individual component, such as one LLM call inside a trace. Then zoom out to the inputs and outputs of a whole workflow. In Dat’s travel example, that workflow combines LLM calls, API calls to find actual flights or hotel vacancies, and heuristics. The accompanying slide illustrates component composition with a separate Chat-to-Purchase Router: product comparison, product search, customer support and package tracking branches combine LLM calls, internal API calls and application code.

![Flowchart routing user intent through product comparison, product search, customer support, and package tracking branches, with a legend distinguishing three component types.](https://ai.engineer/api/talk-essay-media/29427bc892ee610d4b121547fe83e4a0f3183cbd797eff0dc5463c73bce4d0ec)

The Chat-to-Purchase Router combines LLM calls, internal API calls, and application code.

**Check control flow before spending money evaluating its consequences.** If the router sent a request down the wrong branch, downstream quality scores may add little diagnostic value. Conditional evaluation can first judge the routing decision and skip downstream checks when that decision fails. This makes the evaluator’s execution order reflect dependencies in the application.

Zoom out again and a session becomes a series of traces representing a back-and-forth conversation. A session-level evaluator can ask whether the customer became frustrated at any point, even when no single call captures the whole experience. There is no universal evaluation recipe: Dat recommends customizing evaluators heavily and first understanding how the application works.

### Find failure modes across agent paths

[13:12](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=792s)

Arize’s Copilot provides a concrete example of an agent built to observe, troubleshoot and create evaluations for other AI systems. Dat says it had been available for a year at the time of the talk. He frames it as an early step toward a future, five to ten years out, in which AI systems perform more of the work of evaluating other AI systems.

The demonstration moves into agent traces, which grow longer as agent calls become more involved. Dat opens a failed trace and explains how Copilot chooses specialized agents based on the user’s request and their current location in the platform. Each agent has its own set of tools. A trace lets an engineer inspect what happened in one execution, but the operational question is broader: which failure modes recur across the agent’s executions?

For an agent with ten available tools, useful questions include how often it calls each tool and how evaluation results vary by path. The aggregate agent graph is described as framework-agnostic, covering LangGraph, CrewAI and hand-written orchestration. Its purpose is to connect path frequency and path-specific evaluation results across many traces.

Dat’s dependency example makes the value concrete:

| Observed path | Evaluation pattern | Explanation |
|---|---|---|
| `1 → 2 → 3` | Looks good | Successful path in the example |
| `4 → 2 → 3` | Scores drop | Component `4` needs component `3` first |

The second path eventually reaches component `3`, but too late to satisfy component `4`’s dependency. Looking across executions exposes an ordering problem that a single overall good-or-bad score would obscure. The target is the distribution of behavior, not just one instantiation.

### Compare what the agent did with the path it needed

[16:33](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=993s)

A trajectory evaluation starts with a particular request and the components the agent should visit. For a Trip Planner hotel request, that might begin with a start agent followed by a tool agent. A golden trajectory dataset can record the expected path, or the three or four components an example must reach.

There are two straightforward ways to use that reference: give an LLM the actual and expected trajectories and ask it to grade the difference, or compare trajectory strings directly. The demonstrated strict comparison shows different actual and reference step sequences and returns `False`.

![Code and printed outputs show two different step sequences followed by a graph trajectory strict-match result with score False.](https://ai.engineer/api/talk-essay-media/ab49730d51dbd48d1c33e6702999f5f2357772045f42fcc86043b01656f0a0a1)

Actual and reference trajectories produce a false strict-match result.

Exact sequence equality and required-step coverage answer different questions. For a simplified hotel request, this Python example makes the difference explicit:

```python
request = "Find me hotels."
reference = ["start_agent", "tool_agent"]
actual = ["tool_agent", "start_agent"]

scores = {
    "strict_match": actual == reference,
    "required_steps_present": set(reference).issubset(actual),
}
# {"strict_match": False, "required_steps_present": True}
```

Both steps are present, but their order is wrong relative to the reference. Choose the comparison that expresses the application’s requirement; a presence check cannot enforce ordering.

An exact ground-truth path is not always necessary. Dat suggests judging whether the visited nodes and their descriptions match the expected process. He also describes an exploratory approach: provide the actual trajectory alongside possible paths, represented as nested key-value pairs, and ask a model to assess it. These alternatives move from literal sequence matching toward judging whether the path fulfills the intended process.

### Put checks in the execution path deliberately

[18:20](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1100s)

The first audience question turns from after-the-fact evaluation toward checks that can affect the current interaction. Dat prefers **in orchestration versus out of orchestration** to the less precise distinction between online and offline evaluation. An inline check can decide whether execution may continue; that makes it a guardrail, and puts its latency in the user’s path.

Think of two interacting systems. System one is the application: prompts, orchestration and other behavior. System two is the guardrail that checks it. The guardrail mitigates risk but adds work. Dat describes embedding-based guardrails as roughly 100 times lower-latency, without specifying the comparison system or benchmark conditions. A faster check can reduce the latency cost, but the second system still adds complexity.

When the guardrail repeatedly catches a problem, the first repair should often be in system one. Adjust the prompt or orchestration that produces the failure rather than immediately changing the protective check. Guardrails are fallible and resemble unit tests for known cases; observability and evaluations help discover behavior that was not anticipated before real users arrived.

The placement decision therefore depends on the application. An inline evaluator can prevent a bad continuation, while an evaluator outside orchestration can generate improvement signals without blocking the current response. Neither placement is universally correct; the question is which checks must influence execution now and which can inform the next iteration.

### Carry trace context across service boundaries

[20:39](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1239s)

A second question asks how to evaluate a complex, long-running asynchronous system when spans, timeouts and service boundaries limit the view. Dat points to [OpenTelemetry context propagation](https://opentelemetry.io/docs/concepts/context-propagation/). As LLM applications spread across services, Docker containers and Kubernetes pods, observing one process is no longer enough.

His example follows `application A → model router → application A → application B → application A`. Propagating trace context lets those service-level observations remain connected, so the engineer can inspect the complete operation instead of disconnected fragments. That requires instrumentation to carry the context, or explicit injection and extraction where automatic propagation is unavailable.

Dat says Arize committed to an OpenTelemetry-first approach two and a half years earlier. The answer addresses cross-service visibility: propagation preserves the relationship between work performed in different places. It does not itself define timeout policy or determine when an asynchronous workflow is complete.

### Treat evaluator confidence as another signal

[22:18](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1338s)

Asked how to obtain a confidence score for an evaluation, Dat starts with autoregressive judges. If the evaluator returns a single-token label and the provider exposes that token’s log probability, the value offers a pseudo-confidence signal. He names OpenAI as one provider with this capability; the [OpenAI logprobs example](https://developers.openai.com/cookbook/examples/using_logprobs) documents the historical approach and is now archived, so availability must be checked for the chosen model and endpoint.

A token’s log probability measures how likely that token is under the model’s context. It is not a calibrated probability that the evaluation is correct. Encoder-only classifiers can instead expose classification probabilities. Dat recommends using these signals together with the rest of the evaluation toolbox to understand where the system is doing well or badly.

### Shorten the loop from feedback to a revised prompt

[23:20](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1400s)

The final question asks how to reduce developer intervention between customer feedback and prompt improvement. Dat points to DSPy and its [MIPRO optimizer](https://arxiv.org/abs/2406.11695). He illustrates the idea with roughly 30 input-output pairs and describes the goal as less fragile prompts across models from providers such as OpenAI and Google, including **Gemini**. The example count is illustrative, not a fixed requirement or a guarantee of transfer across models; current [MIPROv2](https://dspy.ai/api/optimizers/MIPROv2/) uses configurable datasets and a task metric.

Dat then describes Arize’s meta-prompting approach, with prompt optimization available or being released at the time. The mechanism is to give an LLM the evidence needed to propose a repair:

1. Supply the original prompt.
2. Include a dataset of input-output pairs.
3. Attach evaluation results showing successes and failures.
4. Ask for a revised prompt that addresses the observed failures.

The output is a proposed prompt revision. This automates part of the improvement loop by turning examples and evaluation signals into a concrete change, rather than leaving a developer to translate every failure into new instructions manually.

## Resources

- [Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs](https://arxiv.org/abs/2406.11695) — The original MIPRO paper explains how to optimize instructions and few-shot demonstrations across a multi-stage language-model program.
- [DSPy MIPROv2 documentation](https://dspy.ai/api/optimizers/MIPROv2/) — API reference and examples for optimizing DSPy programs with task metrics, training examples and Bayesian search.
- [OpenTelemetry context propagation](https://opentelemetry.io/docs/concepts/context-propagation/) — How trace context crosses service boundaries through injection and extraction, including custom propagation.
- [Using logprobs](https://developers.openai.com/cookbook/examples/using_logprobs) — An archived OpenAI example demonstrating token log probabilities and classification thresholds in Chat Completions.
- [Arize Observe 2025 product releases](https://arize.com/blog/observe-2025-releases/) — June 2025 announcements covering Alyx, agent graphs, session and trajectory evaluation, and prompt optimization.
- [Booking.com's AI Trip Planner launch](https://news.booking.com/bookingcom-launches-new-ai-trip-planner-to-enhance-travel-planning-experience/) — The original announcement describes conversational destination, accommodation and itinerary planning and the initial US beta.

## Slides

### Engineering Better Evals: Scalable LLM Evaluation Pipelines That Work

Dat Ngo opens his talk on LLM evaluation pipelines by asking the audience who has built an agent or run an eval.

[0:23](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=23s) [Title slide for Dat Ngo’s talk on scalable LLM evaluation pipelines.](https://assets.ai.engineer/sites/aiecode2025/sha256/f3ce1d1d5957fb93f945022616d5f0085d088d787c0cd0e3532e134ed1babb62.webp)

Engineering Better Evals:
Scalable LLM Evaluation Pipelines That Work

Dat Ngo

arize
AI Engineer
World’s Fair
AIE
Microsoft
smol ai

### Dat Ngo: Introduction

Dat Ngo, an AI architect at Arize AI, builds observability and evals and works with AI teams across the industry. He introduces Mochi and Latte, his friends’ dogs.

[1:00](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=60s) [Dat Ngo introduces his work in AI observability and evaluation, alongside photos of two dogs.](https://assets.ai.engineer/sites/aiecode2025/sha256/2b85f96abf6535b96be4e7921d67795df6b86fefa6cc73b6e30d447956f516a6.webp)

Dat Ngo
Arize AI
AI Architect

- Building observability & evals since day 0
- Working with some of the world's best AI teams across the industry
- Huge passion for the AI community

@dat_attacked
@mochipomsky
@lattetotherescue

© All Rights Reserved

### Building GenAI is Hard, So How Are Teams Tackling This?

Observability reveals what an application is doing; evaluation measures how well it performs against criteria. Together, they point teams toward experiments and improvements.

[2:16](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=136s) [A diagram connects observability and evaluation to experimentation and improvement.](https://assets.ai.engineer/sites/aiecode2025/sha256/2dd985acc1a75ff7a182d19e60f2564192163f65d8b072157e74e76ce486fc24.webp)

Building GenAI is Hard, So How Are Teams Tackling This?

Observability
What is happening in my application? Can I root cause down into the problem?

+

Evaluation
How well is the AI product that I’ve built, actually performing according my criteria?

Experimentation & Improvement
The ultimate goal of observability and evaluation is to know where to iterate and know where to improve the system

[illegible]s Work
© All Rights Reserved

Interface text:
AIE
AI Engineer
World’s Fair

[2:25](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=145s) [Diagram linking observability and evaluation to experimentation and improvement.](https://assets.ai.engineer/sites/aiecode2025/sha256/3f82e276c2dd839e242ffed13d455775b17d45613e3fa33bb1b411ec20d5889e.webp)

Building GenAI is Hard, So How Are Teams Tackling This?

Observability
What is happening in my application? Can I root cause down into the problem?

+

Evaluation
How well is the AI product that I’ve built, actually performing according my criteria?

Experimentation & Improvement
The ultimate goal of observability and evaluation is to know where to iterate and know where to improve the system

[illegible]s Work
© All Rights Reserved

Interface text:
AIE
AI Engineer
World’s Fair
Microsoft
smol ai

### LLM tracing

Arize’s trace table puts questions and answers alongside latency, token counts, and evaluation results, helping engineers inspect what an AI application is doing.

[2:47](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=167s) [Arize tracing table with example queries, outputs, latency, and evaluation results.](https://assets.ai.engineer/sites/aiecode2025/sha256/d2e38645766af3e26f45d5987d184b0ced23a604a56473e92ca5617323b26d98.webp)

arize-demo-generative-llm-tracing

Last 7 Days
Total Spans 5.408k
Total Tokens 459.4k
Latency P50 3.01s
Latency P99 35.07s

Traces | Spans | Agents
Kind | Name | Input | Output | Start Time | Latency | Total Tokens | Status | Evaluations | Annotations | metadata | Session ID | attributes.user.id

CHAIN | query
Input: Can I copy a dashboard
Output: The information provided does not specify whether you can copy a dashboard. It only explains how to edit a dashboard by adding widgets and customizing their size and position.
Start Time: 6/4/2025, 03:31 PM
Latency: 3.52s
Total Tokens: 248
Evaluations: Hallucination factual; QA_Correctness correct

CHAIN | query
Input: Can I copy a dashboard to a new space?
Output: The provided information does not specify whether you can copy a dashboard to a new space. It only explains how to add a widget to a dashboard, resize it, and move it around, as well as...
Start Time: 6/4/2025, 03:22 PM
Latency: 5.27s
Total Tokens: 280
Evaluations: Hallucination factual; QA_Correctness correct

CHAIN | query
Input: How do I configure permissions for GBQ?
Output: To configure permissions for Google BigQuery (GBQ), you can follow these steps: 1. In the Google Cloud console, navigate to the BigQuery SQL Workspace. 2. Select the desired table or...
Start Time: 6/4/2025, 03:15 PM
Latency: 49.79s
Total Tokens: 1282
Evaluations: Hallucination factual; QA_Correctness correct

CHAIN | query
Input: How often does Arize query the table for table import jobs?
Output: [illegible]

Interface text:
arize
Demo Models / Demo Model Manager
Projects & Models
Projects
Evaluations
Datasets & Experiments
Labeling Queues
Evals & Tasks
Prompt Engineering
Prompt Playground
Prompt Hub
Monitoring
Dashboards
Monitors
LLM Tracing | Sessions | Embeddings | Monitor | Dashboard | Custom Metrics | Evals & Tasks | Config
Filter by: ‘span property’ = ‘dimension value’
Add to Dataset
Ask Copilot a question
AIE
AI Engineer World’s Fair
Microsoft
smol ai
Browser address: app.arize.com/[illegible]

### Conversation session

The session view groups human questions and AI responses; the speaker notes that evals can also run at the conversation level. Both visible answers say the supplied context does not resolve the question.

[3:00](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=180s) [Arize session view showing example human questions and AI responses.](https://assets.ai.engineer/sites/aiecode2025/sha256/cf62ab0dff79a15e915862d45fe0c9e8ef5bef0adcc69c2edce611c7165560e4.webp)

Total Duration: 41m:14s
Total Tokens: 2043
Total Traces: 5

Human
Does Arize support Microsoft Teams alerting?

AI
The provided context does not mention support for Microsoft Teams alerting. It only specifies native integration with PagerDuty and Slack for monitoring workflows and alerts.

Human
Do we have the ability to resolve a monitor?

AI
The provided information does not specify the ability to resolve a monitor. It only details the configuration options for setting up and customizing monitors.

Interface text:
LLM Tracing
Last 7 Days
Session ID
Pretty
Raw
Trace ID:
Token Cou[illegible]
Start Time:
Trace Late[illegible]
Evaluations
AIE
AI Engineer World's Fair
aws
app.arize.com/[illegible]

### Monitoring dashboard

The dashboard places hallucination and summarization evals beside cost and latency charts. The speaker connects these metrics to different teams: business-side teams may focus on evals, while platform teams track cost and latency.

[3:14](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=194s) [Arize dashboard showing evaluation, latency, and cost charts.](https://assets.ai.engineer/sites/aiecode2025/sha256/ed5d2256dacf3087a60364c5f05bc7ffab2d6e1a1b37ea4d292836dc53879270.webp)

Hakan - Single Pane of Glass Demo

Model(s) on this dashboard: arize-demo-CV-image-classification-quality-drift, arize-demo-fraud-detection-use-case, arize-demo-generative-llm-search-and-retrieval-debug-prompts-with-corpus, arize-demo-generative-llm-tracing

RAG App - Hallucination Monitor
View Monitor
Y-axis: Custom Metric
Y-axis ticks: 0.1, 0.2, 0.3
Timeline: May 28, May 29, May 30, May 31, Jun 1, Jun 2, Jun 3, Jun 4, Jun 5
Legend: Threshold; Evaluation Window (72 hours)

Current Hallucination Rate
Hallucination Eval Score
0.181

Latency Distribution
Y-axis: Count
Y-axis ticks: 8%, 18%, 28%
X-axis: latency_ms
X-axis bins: -1k to 0, 0 to 1k, 1k to 3k, 3k to 4k, 4k to 6k, 6k to 7k, 7k to 9k, 9k to 10k, 10k to 12k, 12k to 49.79[illegible]
Legend: plot_1

Cost over Time
Y-axis: Metrics
Y-axis ticks: 0.025, 0.05, 0.075, 0.1, 0.125
Timeline: May 28, 3 PM; May 29, 11 AM; May 30, 7 AM; May 31, 2 AM; May 31, 9 PM; Jun 1, 4 PM; Jun 2, 10 AM; Jun 3, 4 AM; Jun 3, 10 PM; Jun 4, 11 PM
Legend: Cost

Summarization App - Summarization Monitor
View Monitor
Y-axis: Custom Metric
Y-axis ticks: 0.2, 0.4, 0.6, 0.8, 1
Timeline: May 28, May 29, May 30, May 31, Jun 1, Jun 2, Jun 3, Jun 4, Jun 5
Legend: Threshold; Evaluation Window (72 hours)

Latency over Time
Y-axis: Metrics
Y-axis ticks: 0, 15k, 30k, 45k, 60k
Timeline: May 28, 3 PM; May 29, 11 AM; May 30, 7 AM; May 31, 2 AM; May 31, 9 PM; Jun 1, 4 PM; Jun 2, 10 AM; Jun 3, 4 AM; Jun 3, 10 PM; Jun 4, 11 PM

[illegible]ssions
Prediction Drift Over Time

Interface text:
arize
Demo Models / Demo Model Manager
05/28/2025 12 AM – 06/05/2025 12 AM
PDT
Auto, hour, day, week, month
Projects & Models
Projects
Evaluations
Datasets & Experiments
Labeling Queues
Evals & Tasks
Prompt Engineering
Prompt Playground
Prompt Hub
Monitoring
Dashboards
Monitors
Filters
Select Project
Use natural language to create a dashboard widget
app.arize.com/organizations/[illegible]
AIE
AI Engineer World's Fair
Microsoft
smol ai

[3:22](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=202s) [A dashboard shows hallucination and summarization scores alongside cost and latency charts.](https://assets.ai.engineer/sites/aiecode2025/sha256/374a382e825417bfe4d3e2511c30082bbfcaad0006dcae54e8d1e9df46ec1020.webp)

Hakan - Single Pane of Glass Demo

Model(s) on this dashboard: arize-demo-CV-image-classification-quality-drift, arize-demo-fraud-detection-use-case, arize-demo-generative-llm-search-and-retrieval-debug-prompts-with-corpus, arize-demo-generative-llm-tracing

RAG App - Hallucination Monitor
View Monitor
Y-axis: Custom Metric
Y-axis ticks: 0.1, 0.2, 0.3
Timeline: May 28, May 29, May 30, May 31, Jun 1, Jun 2, Jun 3, Jun 4, Jun 5
Legend: Threshold; Evaluation Window (72 hours)

Current Hallucination Rate
Hallucination Eval Score
0.181

Latency Distribution
Y-axis: Count
Y-axis ticks: 8%, 18%, 28%
X-axis: latency_ms
X-axis bins: -1k to 0, 0 to 1k, 1k to 3k, 3k to 4k, 4k to 6k, 6k to 7k, 7k to 9k, 9k to 10k, 10k to 12k, 12k to 49.79[illegible]
Legend: plot_1

Cost over Time
Y-axis: Metrics
Y-axis ticks: 0.025, 0.05, 0.075, 0.1, 0.125
Timeline: May 28, 3 PM; May 29, 11 AM; May 30, 7 AM; May 31, 2 AM; May 31, 9 PM; Jun 1, 4 PM; Jun 2, 10 AM; Jun 3, 4 AM; Jun 3, 10 PM; Jun 4, 11 PM
Legend: Cost

Summarization App - Summarization Monitor
View Monitor
Y-axis: Custom Metric
Y-axis ticks: 0.2, 0.4, 0.6, 0.8, 1
Timeline: May 28, May 29, May 30, May 31, Jun 1, Jun 2, Jun 3, Jun 4, Jun 5
Legend: Threshold; Evaluation Window (72 hours)

Latency over Time
Y-axis: Metrics
Y-axis ticks: 0, 15k, 30k, 45k, 60k
Timeline: May 28, 3 PM; May 29, 11 AM; May 30, 7 AM; May 31, 2 AM; May 31, 9 PM; Jun 1, 4 PM; Jun 2, 10 AM; Jun 3, 4 AM; Jun 3, 10 PM; Jun 4, 11 PM

[illegible]ssions
Prediction Drift Over Time

Interface text:
arize
Demo Models / Demo Model Manager
05/28/2025 12 AM – 06/05/2025 12 AM
PDT
Auto, hour, day, week, month
Projects & Models
Projects
Evaluations
Datasets & Experiments
Labeling Queues
Evals & Tasks
Prompt Engineering
Prompt Playground
Prompt Hub
Monitoring
Dashboards
Monitors
Filters
Select Project
Use natural language to create a dashboard widget
app.arize.com/organizations/[illegible]
AIE
AI Engineer World's Fair

### Observability, evaluation, and improvement

Observability shows what is happening; evaluation signals how well the AI product performs. Together, they guide improvement. The speaker notes that inspecting every trace manually does not scale.

[4:10](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=250s) [Observability and evaluation lead to experimentation and improvement.](https://assets.ai.engineer/sites/aiecode2025/sha256/39b6f52f5dcf0db6ade10b0db2978671a94f48b24453f76ffb65aa86718a9abd.webp)

Building GenAI is Hard, So How Are Teams Tackling This?

Observability
What is happening in my application? Can I root cause down into the problem?

+

Evaluation
How well is the AI product that I’ve built, actually performing according my criteria?

Experimentation & Improvement
The ultimate goal of observability and evaluation is to know where to iterate and know where to improve the system

arize | We Make Models Work
© All Rights Reserved

Interface text:
Engineering Better Evals: Scalable LLM Evaluation Pipelines That Work
File  Edit  View  Insert  Format  Slide  Arrange  Tools  Extensions  Help
Menus  Fit  Background  Layout  Theme  Transition
Slideshow  Share
Address bar: docs.google.com/presentation/d/[illegible]/edit?slide=id.g3430a956046_0_204#slide=id.g3430a956046_0_204

### Agent trace

The trace follows an agent through a SQL query tool and back to the agent. The speaker uses it to explain why manually inspecting every trace does not scale, and why evals provide signals about what is going well or poorly.

[4:28](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=268s) [An agent trace shows execution steps and message input and output.](https://assets.ai.engineer/sites/aiecode2025/sha256/9f1630456b7da3073ab43628adbd19750c19b457211176e9f32fe76d078c3385.webp)

Trace
LangGraph 2.82s
  __start__ 1.00ms
  agent 0.77s
    ChatOpenAI 272 0.77s
    ChannelWrite<agent,messages> 0.00ms
    should_continue 0.00ms
  tools 1.48s
    generate_and_run_sql_query 1.47s
      ChatOpenAI 917 0.61s
      ChatOpenAI 960 0.76s
    ChannelWrite<tools,messages> 0.00ms
  agent 0.56s
    ChatOpenAI 299 0.55s
    ChannelWrite<agent,messages> 0.00ms
    should_continue 1.00ms

chain LangGraph
2.82s at 5/28/2025 17:19:09 PM
Info  Annotations 0  Attributes  Events 0

Input
{
  "messages": [
    "content=\"How many of my traces ask about the author's employment?\" additional_kwargs={} response_metadata={} id='[illegible]'",
    "content=\"\\nYou are a helpful assistant that choses a tool to call based on the user's request.\\n\\nAll of your responses should be a tool call or text. Only generate tool calls or text.\\nIf you generate a tool call, be sure you include the original prompt as is in the parameters.\\n\\nOnce you receive the results from all of your skills, generate a response to the\\nuser that incorporates all of the results.\\n\" additional_kwargs={} response_metadata={} id='[illegible]'"
  ]
}

Output
{
  "messages": [
    "content=\"How many of my traces ask about the author's employment?\" additional_kwargs={} response_metadata={} id='[illegible]'",
    "content=\"\\nYou are a helpful assistant that choses a tool to call based on the user's request.\\n\\nAll of your responses should be a tool call or text. Only generate tool calls or text.\\nIf you generate a tool call, be sure you include the original prompt as is in the parameters.\\n\\nOnce you receive the results from all of your skills, generate a response to the\\nuser that incorporates all of the results.\\n\" additional_kwargs={} response_metadata={} id='[illegible]'",
    [illegible]

Interface text:
phoenix-demo.arize.com/projects/[illegible]
Spans
AIE
aws
AI Engineer World's Fair

### Application evaluation metrics

LLM-as-a-judge is one of several evaluation tools. The slide places code checks and annotations alongside it for development, then extends the metrics to user feedback and business outcomes in production.

[4:43](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=283s) [Five columns compare model-judged, code-based, annotation, user feedback, and business metrics.](https://assets.ai.engineer/sites/aiecode2025/sha256/47d3ffa5fdd00ec17f35a6e32dff511b3d31868981e6b70f4961649fb03ca521.webp)

What Metrics Should You Think About For Your Application

LLM as a Judge
- Different templates for different tasks
- Question answering
- Summarization
- Encoder only BERT based token classifiers (SLMs) - LibreEvals

Code/Logic Based Evals
- Unstructured -> Structured Verification
- CodeGen
- Code Checks / Validation
- Successful Python / TypeScript runs
- Rouge/Bleu

Annotations
- Golden Datasets
- Label (e.g. good / bad, hallucinated / factual)
- Score (e.g. 0 - 1)
- Notes (text)
- Can be generated synthetically

User Based Feedback
- Thumbs up / Thumbs down
- Accept / reject response

Business Metrics
- # of times that customer is kicked to human
- Conversation ended unexpectedly
- Recommend item CTR
- Adoption in terms of customers

Development Metrics
Production Metrics

Interface text:
Engineering Better Evals: Scalable LLM Evaluation Pipelines That Work
File Edit View Insert Format Slide Arrange Tools Extensions Help
Menus
Background Layout Theme Transition
Slideshow Share

### RAG evaluation metrics

RAG relevance compares retrieved context with the query. Other evaluations compare the generated answer with the ground truth for correctness and with the retrieved context for faithfulness.

[5:03](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=303s) [A diagram relates a query, ground truth, generated answer, and retrieved context to evaluation measures.](https://assets.ai.engineer/sites/aiecode2025/sha256/c5c66bc5f45d98d3045355f6d022164708a3a97a3f310800a062b00ed56bf4a1.webp)

Ground Truth Answer
Query
Generated Answer
Retrieved Context

Human or AI Generated
Q/A Correctness
Context Recall
Answer Relevance
RAG Relevance
Grounded-ness/ Faithful-ness

Q/A Correctness: The accuracy of the generated answer when compared to the ground truth
Answer Relevance: Assesses how pertinent the generated answer is to the query
Context Recall: Measures the extent to which the retrieved context aligns with the ground truth (no tru-lens support yet)
RAG Relevance: Gauges the relevancy of the retrieved context to the query
Grounded-ness / Faithful-ness: Assess the factual consistency of the generated answer against the retrieved context

Interface text:
Engineering Better Evals: Scalable LLM Evaluation Pipelines That Work
File Edit View Insert Format Slide Arrange Tools Extensions Help
Menus
149%
Format options
Animate
Slideshow
Share

[5:31](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=331s) [Diagram connects a query, ground truth answer, generated answer and retrieved context; a table defines the associated evaluation metrics.](https://assets.ai.engineer/sites/aiecode2025/sha256/9eab58541b7d4ad85b7f774b4fedcace58eb65b03ae93e34f52adcd178c1cb78.webp)

Ground Truth Answer
Query
Generated Answer
Retrieved Context

Human or AI Generated
Q/A Correctness
Context Recall
Answer Relevance
RAG Relevance
Grounded-ness/ Faithful-ness

Q/A Correctness: The accuracy of the generated answer when compared to the ground truth
Answer Relevance: Assesses how pertinent the generated answer is to the query
Context Recall: Measures the extent to which the retrieved context aligns with the ground truth (no tru-lens support yet)
RAG Relevance: Gauges the relevancy of the retrieved context to the query
Grounded-ness / Faithful-ness: Assess the factual consistency of the generated answer against the retrieved context

Interface text:
Engineering Better Evals: Scalable LLM Evaluation Pipelines That Work
File Edit View Insert Format Slide Arrange Tools Extensions Help
Menus
152%
Format options
Animate
Slideshow
Share

### Why LLM Evals Are Good Indicators

The slide argues that comparing a written summary with its source is easier than generating a thoughtful, concise, accurate summary—a distinction behind using LLMs as judges.

[5:40](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=340s) [Slide says generating a good summary is harder than assessing one, illustrated by a light bulb and balance scales.](https://assets.ai.engineer/sites/aiecode2025/sha256/8ca494a1c859e487016352b85bbf25d3aa6e099764812b77d6175248e64a2a62.webp)

Why LLM Evals Are Good Indicators

From a cognitive perspective, it’s much harder to produce a thoughtful, concise and accurate summary, than it is to do a comparative analysis between an already written summary to its original corpra.

Generate a good summary
Generate IDEAs

Harder Than

Is the summary a good one?

Interface text:
Engineering Better Evals: Scalable LLM Evaluation Pipelines That Work
File  Edit  View  Insert  Format  Slide  Arrange  Tools  Extensions  Help
Menus  143%  Background  Layout  Theme  Transition
Slideshow  Share

### What Metrics Should You Think About For Your Application

Pair scalable LLM judges with a human-graded golden dataset to check and tune their judgments. Code checks, user feedback and business metrics add signals across development and production.

[5:58](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=358s) [Five columns compare LLM judges, code-based evaluations, annotations, user feedback and business metrics.](https://assets.ai.engineer/sites/aiecode2025/sha256/cbf6fa3aaeb84ea2651291c6adfac76ff05e1a7866b3cf187b4cee339f4c1463.webp)

What Metrics Should You Think About For Your Application

LLM as a Judge
- Different templates for different tasks
- Question answering
- Summarization
- Encoder only BERT based token classifiers (SLMs) - LibreEvals

Code/Logic Based Evals
- Unstructured -> Structured Verification
- CodeGen
- Code Checks / Validation
- Successful Python / TypeScript runs
- Rouge/Bleu

Annotations
- Golden Datasets
- Label (e.g. good / bad, hallucinated / factual)
- Score (e.g. 0 - 1)
- Notes (text)
- Can be generated synthetically

User Based Feedback
- Thumbs up / Thumbs down
- Accept / reject response

Business Metrics
- # of times that customer is kicked to human
- Conversation ended unexpectedly
- Recommend item CTR
- Adoption in terms of customers

Development Metrics
Production Metrics

Interface text:
Engineering Better Evals: Scalable LLM Evaluation Pipelines That Work
File  Edit  View  Insert  Format  Slide  Arrange  Tools  Extensions  Help
Menus  143%  Background  Layout  Theme  Transition  Slideshow  Share
docs.google.com/presentation/d/[illegible]/edit?slide=[illegible]

### Creating a code evaluator task

The demo configures a continuously running code evaluator, then shows a keyword check that tests output with code instead of an LLM or human.

[7:49](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=469s) [An evaluation platform shows a new task form with the code evaluator option selected and a continuous run schedule.](https://assets.ai.engineer/sites/aiecode2025/sha256/c9a130982037de55e73f7844e3247e63abeb51b79e19aa47d3515f091085b3e8.webp)

New Task
Task Name*
test

Run On*
Project
arize-demo-generative-llm-tracing

Sampling Rate (%)
100
"attributes.openinference.span.kind" = 'LLM'

Task Type:
LLM Evals
Evaluate LLM data with an LLM-as-a-Judge
Code Evaluator
Evaluate LLM data with code

Schedule Run*
Run continuously on new incoming data

Next
Cancel

Interface text:
arize
Demo Models / Demo Model Manager
Evals & Tasks
Create and run automated evals & actions on your projects
Projects & Models
Projects
Evaluations
Datasets & Experiments
Labeling Queues
Evals & Tasks
Prompt Engineering
Prompt Playground
Prompt Hub
Monitoring
Dashboards
Monitors

Name | Source | Type
Agent Tool Selection_ExactMatch | Agent Tool Selection, Dataset | LLM Eval
Agent Tool Selection_Tool Correctness | Agent Tool Selection, Dataset | LLM Eval
inventions_dataset_Exact Match | inventions_dataset, Dataset | LLM Eval
movies_exact_match | movies, Dataset | LLM Eval
movies with both outputs_exact_match | movies with both outputs, Dataset | LLM Eval
test dataset_test | test dataset, Dataset | LLM Eval
Movies Dataset_Exact Match | Movies Dataset, Dataset | LLM Eval
hallucinateed dataset 1_hallucination | hallucinateed dataset 1, Dataset | LLM Eval
Hallucinated dataset_hallucination | Hallucinated dataset, Dataset | LLM Eval
Hallucinated dataset_qa | Hallucinated dataset, Dataset | LLM Eval
Movies Dataset_exact_match_round_2 | Movies Dataset, Dataset | LLM Eval
Movies Dataset_user_frustration | Movies Dataset, Dataset | LLM Eval
annie_movies_exact_match | annie_movies, Dataset | LLM Eval
tool-selection-production_tool-selection | tool-selection-production, Dataset | LLM Eval

app.arize.com/[illegible]/tasks/create
AIE
aws
AI Engineer World's Fair

[7:56](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=476s) [A code evaluator checks whether output contains any specified keywords; a menu covers part of the screen.](https://assets.ai.engineer/sites/aiecode2025/sha256/8aa85595a45adef527669454ea725e325bb8ac31a247bfa6ff692d2b2c0c80c2.webp)

2 Code Evaluator Class: ContainsAnyKey[illegible]

An evaluator that checks if any specified keywords a[illegible]
useful for identifying if the output contains specific t[illegible]
validation or analysis.

class ContainsAnyKeyword(CodeEvaluator):
    """
    An evaluator that checks if any of the given keywords are present in the
        output.
    """

    def __init__(self, keywords: list[str], name: Optional[str] = None) -> None:
        self.keywords = keywords
        self._name = name or f"ContainsAny({keywords})"

    def evaluate(
        self,
        *,
        output: Optional[TaskOutput] = None,
        **kwargs: Any,
    ) -> EvaluationResult:
        assert isinstance(output, str), "Output must be a string"
        found = [keyword for keyword in self.keywords if keyword in output]
        if found:
            explanation = f"the keywords {found} were found in the output"

Interface text:
Contains all Keywords
CustomArizeEvaluator
new

### Self-Improving Agents: Continuous Cycle of Prompts & Evals

The agent loop turns failures into a golden dataset and prompt or fine-tuning changes. Low-confidence evals feed a second loop that uses annotated failures to improve the eval prompt and returns the updated prompt.

[9:26](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=566s) [Two linked cycles show collecting data, running evaluations, annotating failures, and improving agent and evaluation prompts.](https://assets.ai.engineer/sites/aiecode2025/sha256/7e37852163331e940783fd7bcda91178c0f6530532774a070f285f9f25f86b81.webp)

Self-Improving Agents: Continuous Cycle of Prompts & Evals

Improve Agent
Collect data
Run evals
Collect dataset of failures
Annotate for golden dataset
Improve prompt / Fine-tune

Evals with low confidence
Update prompt

Improve Evals
Collect dataset of failures
Annotate for golden dataset
Improve eval prompt on golden dataset

Interface text:
Engineering Better Evals: Scalable LLM Evaluation Pipelines That Work
File  Edit  View  Insert  Format  Slide  Arrange  Tools  Extensions  Help
Menus  168%  Background  Layout  Theme  Transition  Slideshow  Share
Click to add speaker notes
Browser address: docs.google.com/presentation/d/[illegible]

[10:14](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=614s) [Connected cycles show collecting data, running evaluations, annotating failures, and improving agent and evaluation prompts.](https://assets.ai.engineer/sites/aiecode2025/sha256/7129a876592875e6f29b0e249a696f4e775e4cded5acbb81db1549fcfdc3f0f8.webp)

Self-Improving Agents: Continuous Cycle of Prompts & Evals

Improve Agent
Collect data
Run evals
Collect dataset of failures
Annotate for golden dataset
Improve prompt / Fine-tune

Evals with low confidence
Update prompt

Improve Evals
Collect dataset of failures
Annotate for golden dataset
Improve eval prompt on golden dataset

Interface text:
Engineering Better Evals: Scalable LLM Evaluation Pipelines That Work
File  Edit  View  Insert  Format  Slide  Arrange  Tools  Extensions  Help
Menus  164%  Background  Layout  Theme  Transition  Slideshow  Share
Click to add speaker notes
Browser address: docs.google.com/presentation/d/[illegible]

### Trace evaluation detail

The selected synthesize span shows its input and output beside the nested trace. The speaker contrasts evaluating an individual component with evaluating the larger workflow.

[11:20](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=680s) [Arize trace view showing a workflow, its input and output, and evaluation results.](https://assets.ai.engineer/sites/aiecode2025/sha256/2c1fd43c70467d90ec6c7b4572ebae1a93116bae81b436673fda1a0582dc8839.webp)

trace_id: 8d006d34-7161-45a8-af92-f8e0404580fc
Trace Status: OK
Latency: 20.04s
Start Time: 6/4/2025, 4:56:14 PM
Evaluations: Hallucination hallucinated; QA_Correctness incorrect

query — 2 evals — 20.04s
  synthesize — 19.47s
    llm — 503 — 19.47s
  retrieve — 0.57s
    embedding — 0.37s

synthesize
Input / Output
Input — gpt-3.5-turbo
How do I use the SDK to upload a ranking model?

Output
To upload a ranking model using the SDK, you would typically follow these steps:
1. Install the SDK: Ensure that the SDK is installed in your development environment. This is usually done via a package manager specific to your programming language.
2. Import the SDK: In your code, import the necessary classes or methods from the SDK that are required to interact with the service.
3. Initialize the SDK: Create an instance of the SDK client, often requiring you to provide credentials or an API key that authorizes you to upload models.
4. Prepare the Model: Make sure your ranking model is in the correct format expected by the SDK. This may involve serializing the model into a specific file format.
5. Upload the Model: Use the appropriate method provided by the SDK to upload your ranking model. This method will require you to specify the model file and may also require additional metadata about the model.
6. Handle Responses: After the upload request, handle any responses or exceptions provided by the SDK to ensure the model [illegible]
7. Verify Upload: Check if the model has been uploaded successfully, which may involve using another SDK method to list [illegible]

Workflow diagram: Start → Query → Retrieve → Embedding → End; Query → Synthesize → Llm → End

Interface text:
arize
Projects & Models
Projects
Evaluations
Datasets & Experiments
Labeling Queues
Evals & Tasks
Prompt Engineering
Prompt Playground
Prompt Hub
Monitoring
Dashboards
Monitors
LLM Tracing
Last 7 Days
Total Spans 5.935k
Traces
Spans
Agents
Search by span name
Add to Dataset
Evaluations
Attributes
Events
Annotations
Ask Copilot about this span
AIE
Microsoft
smol ai
AI Engineer World's Fair

### Chat-to-Purchase Router

The router sends user intent into comparison, search, support or tracking workflows. The speaker recommends evaluating control flow first: if the route is wrong, downstream evaluations may waste effort.

[11:55](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=715s) [A router diagram branches from user intent into product comparison, product search, customer support and package tracking workflows; the bottom is cropped.](https://assets.ai.engineer/sites/aiecode2025/sha256/6d7ff980b6d8bcf4fbccdb70387a3877ddeebae4a93c32ad0507a39c94f04a00.webp)

Chat-to-Purchase Router

Legend:
LLM Call
Internal API Call
Application Code

Entry Point: Customer Input → User Intent
router_function

Product Comparison → Chatbot Query: Product, Price, Quality → Unstructured to Structured → Search Ranking API → Comparison UI

Product Search → Unstructured to Structured → Search API → Rank Products → Summarize Product Descriptions → Add to Wishlist / Checkout

Customer Support → Extract Query → RAG on Customer Support Docs → Return to Menu / Chat with Live Support Agent

Track Package → Call Order Details API → Tracking UI

Interface text:
Engineering Better Evals: Scalable LLM Evaluation Pipelines That Work
File  Edit  View  Insert  Format  Slide  Arrange  Tools  Extensions  Help
Menus  167%  Background  Layout  Theme  Transition
Slideshow  Share
docs.google.com/presentation/d/[illegible]

### Sessions list

Arize lists conversations with message previews, duration, trace counts and token counts. The speaker describes a session as a series of traces, allowing evaluation of a back-and-forth with an agent.

[12:29](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=749s) [Arize sessions table with message previews, durations, trace counts and token counts.](https://assets.ai.engineer/sites/aiecode2025/sha256/ada30a9acc9947cdb1b8efbd90ae047a9505de3c866b0118b0a1845a31245ca9.webp)

arize-demo-generative-llm-tracing
Sessions
Last 7 Days
Filter by: “span property” = ‘dimension value’

Session ID | First Message | Last Message | Duration | User ID | Total Traces | Total Tokens | Start Time
[illegible] | How do I configure permissions for GBQ? | The information provided does not specify whether you can copy a dashb... | 16m:54s | [illegible] | 3 | 1810 | 6/4/2025, 03:15 PM
[illegible] | Do you need to have a prediction label for classification models? | The frequency at which Arize queries the table for table import jobs i... | 18m:49s | [illegible] | 3 | 1082 | 6/4/2025, 02:45 PM
[illegible] | How do you set up pagerduty alerts? | Yes, a prediction ID is required for the training set to connect predi... | 9m:24s | [illegible] | 2 | 678 | 6/4/2025, 02:26 PM
[illegible] | Does Arize store the individual records (rows) somewhere, or does it o... | The ingestion job for GBQ tables should be set up to detect changes in... | 28m:9s | [illegible] | 4 | 1729 | 6/4/2025, 01:48 PM
[illegible] | Does Arize support Microsoft Teams alerting? | If you upload actuals twice, especially with the same prediction IDs, ... | 41m:14s | [illegible] | 5 | 2043 | 6/4/2025, 01:02 PM
[illegible] | Can I export a dashboard as pdf? | If you have sent in duplicate prediction IDs, you should revise your d... | 18m:49s | [illegible] | 3 | 918 | 6/4/2025, 12:34 PM
[illegible] | Does the ingestion job for GBQ tables detect changes in schema? | The context information provided does not include specific details on ... | 35m:40s | [illegible] | 5 | 2516 | 6/4/2025, 11:49 AM
[illegible] | about how long should it take for delayed actuals to link to predictio... | The provided information does not specify whether you can configure th... | 11m:21s | [illegible] | 2 | 648 | 6/4/2025, 11:28 AM
[illegible] | How do I pass in delayed ground truth? | Yes, you can convert strings to numeric values in Arize. This is a com... | 18m:47s | [illegible] | 3 | 1133 | 6/4/2025, 11:04 AM
[illegible] | How large should my file sizes be when uploading data? | Yes, you can pass in your own metrics within Arize. These are referred... | 13m:10s | [illegible] | 2 | 830 | 6/4/2025, 10:41 AM
[illegible] | How do I edit the frequency that my table import job runs? | The time it takes to ingest your data is not specified. However, you c... | 11m:21s | [illegible] | 2 | 734 | 6/4/2025, 10:24 AM
[illegible] | How do I grant permissions to import my GBQ table? | To import your files from cloud storage, you must grant Arize access t... | 11m:16s | [illegible] | 2 | 1569 | 6/4/2025, 10:04 AM
[illegible] | Which file or dataframe columns can be null? | Yes, Arize ingests null values within prediction and actual columns, p... | 13m:12s | [illegible] | 2 | 1148 | 6/4/2025, 09:45 AM
[illegible] | How do I need to format timestamps? | The provided context information does not specify the file types that... | 20m:39s | [illegible] | 3 | 854 | 6/4/2025, 09:17 [illegible]

Interface text:
arize
Demo Models / Demo Model Manager
Projects & Models
Projects
Evaluations
Datasets & Experiments
Labeling Queues
Evals & Tasks
Prompt Engineering
Prompt Playground
Prompt Hub
Monitoring
Dashboards
Monitors
LLM Tracing
Embeddings
Monitor
Dashboard
Custom Metrics
Evals & Tasks
Config
app.arize.com/[illegible]
AIE
AI Engineer World's Fair
Microsoft
smol ai

### Session conversation and evaluations

A session groups a back-and-forth into five traces. The visible exchanges pair AI responses with correctness and hallucination evaluations; the speaker also suggests evaluating customer frustration across a session.

[12:36](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=756s) [A session view displays human questions, AI responses and evaluation results for individual traces.](https://assets.ai.engineer/sites/aiecode2025/sha256/ecb8ac45bd145823926c87ca76135889cf5772747ef1415bbbbcd13eba80c65c.webp)

Total Duration: 41m:14s
Total Tokens: 2043
Total Traces: 5

Human: Does Arize support Microsoft Teams alerting?
AI: The provided context does not mention support for Microsoft Teams alerting. It only specifies native integration with PagerDuty and Slack for monitoring workflows and alerts.
Trace ID: b922a3f7-3905-46...
Token Count: 338
Start Time: 6/4/2025, 1:02:10 PM PDT
Trace Latency: 3.03s
Evaluations: QA_Correctness correct; Hallucination factual

Human: Do we have the ability to resolve a monitor?
AI: The provided information does not specify the ability to resolve a monitor. It only details the configuration options for setting up and customizing monitors.
Trace ID: 774925a2-6755-4e...
Token Count: 240
Start Time: 6/4/2025, 1:15:16 PM PDT
Trace Latency: 3.76s
Evaluations: QA_Correctness correct; Hallucination factual

Trace ID: dfac3254-ab9d-48...
Token Count: 776

Interface text:
Columns
Pretty
Raw
View Trace
AIE
AI Engineer World's Fair

[12:46](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=766s) [Arize session view showing questions, AI responses, and evaluation labels.](https://assets.ai.engineer/sites/aiecode2025/sha256/a4793ba89ba9b31d28a67483ea586bc80782be40fe4c5f46b23e86addfe51b5c.webp)

SessionID: 4bfeb0c6-f1cd-4df1
Total Duration: 41m:14s
Total Tokens: 2043
Total Traces: 5

Human: Does Arize support Microsoft Teams alerting?
AI: The provided context does not mention support for Microsoft Teams alerting. It only specifies native integration with PagerDuty and Slack for monitoring workflows and alerts.
Trace ID: b922a3f7-3905-46...
Token Count: 338
Start Time: 6/4/2025, 1:02:10 PM PDT
Trace Latency: 3.03s
Evaluations: QA_Correctness correct; Hallucination factual

Human: Do we have the ability to resolve a monitor?
AI: The provided information does not specify the ability to resolve a monitor. It only details the configuration options for setting up and customizing monitors.
Trace ID: 774925a2-6755-4e...
Token Count: 240
Start Time: 6/4/2025, 1:15:16 PM PDT
Trace Latency: 3.76s
Evaluations: QA_Correctness correct; Hallucination factual

Human: How often do the monitors run and evalaute?
AI: The frequency at which the monitors run and evaluate is not explicitly stated in the provided context. The context does mention [illegible]
Trace ID: dfac3254-ab9d-48...
Token Count: 776
Start Time: 6/4/2025, 1:20:53 PM PDT
Trace Latency: 5.21s
Evaluations: QA_Correctness correct; Hallucination factual

Interface text:
arize
Projects & Models
Projects
Evaluations
Datasets & Experiments
Labeling Queues
Evals & Tasks
Prompt Engineering
Prompt Playground
Prompt Hub
Monitoring
Dashboards
Monitors
LLM Tracing
Last 7 Days
Share link
Columns
Pretty
Raw
View Trace
AIE
Microsoft
smol ai
AI Engineer World's Fair

### Arize Copilot Agent Architecture: v1

A user question enters Arize AI Copilot, where six agents have tools for tasks from model analysis and evaluations to search and prompt optimization. The copilot then produces a response.

[13:15](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=795s) [A user question enters Arize AI Copilot, which routes work among six agents and their tools before producing a response.](https://assets.ai.engineer/sites/aiecode2025/sha256/094dd5e6ef1da81bcaccdf28b6befee786aa3b6ddaa07d6232ea6571140422d5.webp)

Arize Copilot Agent Architecture: v1

USER QUESTION

ARIZE AI COPILOT

Tabular Agent
Model insights
Drift insight
Data quality checks
Cohort performance
Assess features

Eval Agent
Create custom eval

Analysis Agent
Diagnose RAG
Dashboard generator
ArizeQL generator
Span chat

Search Agent
Column search

Prompt Agent
Optimize prompts

Computer Vision Agent
Embedding summarization
Cluster analysis

Tools

COPILOT RESPONSE

Interface text:
Engineering Better Evals: Scalable LLM Evaluation Pipelines That Work
File  Edit  View  Insert  Format  Slide  Arrange  Tools  Extensions  Help
Menus
147%
Background  Layout  Theme  Transition
Slideshow  Share

## Full transcript

[0:00](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=0s) [upbeat music] All right.

[0:15](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=15s) Well, let's get started. Uh, we don't have much time, but, um, I hope your conference is going well. Uh, welcome to AI Engineering World Fair. Uh, my name is Dat Ngo.

[0:24](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=24s) Uh, today's talk is about LLM Eval Pipelines. Um, so I never know what I wanna talk about till I get into the room, so I don't prep too hard.

[0:31](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=31s) But by show of hands, who here, uh, who here has built, uh, an agent? Just raise your hand. Okay. Uh, who here, um, has run an eval?

[0:42](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=42s) Right. Who here has productionized an AI product? Nice. Okay. Some technical builders. Um, let's get technical then. So, uh, my name is, uh, Dat. Um, I'm an AI architect at Arize AI.

[0:56](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=56s) Um, this is Mochi and Lati- uh, Latte. They're, uh, dogs of my friends. I figured let's, let's keep it spicy and interesting. Um, but I've been building observability and evals since, since day zero.

[1:07](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=67s) So, uh, since the first... Uh, I don't know if you guys know what Arize AI is, but we are the largest, uh, AI evals player, uh, in the space, so observability, evals, kinda beyond.

[1:18](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=78s) Uh, we work really heavily, uh, with, with real use cases, so folks like Reddit, folks like Duolingo. Um, so we work across the best AI teams, and we have a really unique business.

[1:29](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=89s) Being on the observability side, we get to see what everyone is building,

[1:34](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=94s) how they're tackling those problems, what are their biggest pains, uh, and what are the kind of the tips and tricks that they use to, to really productionize these things.

[1:41](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=101s) And just to give you a s- a hint, um, you know, Duolingo has massive eval scale. They tend to run about twenty evals por- per, like, um, trace. Um, so they end up spending, you know, quite a fair amount, uh, doing evals, understanding their evals, optimizing them.

[1:57](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=117s) Uh, and the last thing about me, I have a huge passion for the AI community. Um, when I was in SF the last five years, uh, really loved to go to pretty much every single event that I could.

[2:05](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=125s) Uh, I'm not a developer advocate. I'm an engineering-- uh, an engineer by, by trade, but I just love the community. So, um, yeah, this is a little bit about Arize, but I think I don't wanna keep it too salesy.

[2:15](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=135s) I just wanna keep it pretty technical. So really three concepts that I think everybody should be familiar with and where evals really sit in the space is, is really as simple as this.

[2:24](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=144s) Um, this is what I teach all my customers. Really, the first thing is observability. I think you guys have kind of seen this before. Observability really just answers the question of, what is the thing that I built actually doing, right?

[2:35](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=155s) To some people, it may be traces, uh, traces and, and spans. Um, I'll show you a little bit. Um, it, you know... Platform agnostic, just, just think about the concepts.

[2:45](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=165s) Um, but traces might be one, one area for people. So traces represent, hey, what's happening? Can I look at things? Uh, to an AI engineer, makes a lot of sense.

[2:54](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=174s) Um, let's say you're an AIPM, maybe not super technical, or maybe, uh, you wanna think about things differently. Maybe you wanna look at, hey, what are the conversations that are happening?

[3:03](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=183s) Turns out you can run evals at these levels. Uh, we'll get into that in depth kind of later. Um, you know, signal comes and observability comes in different kind of flavors and forms.

[3:12](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=192s) Maybe it's, it's analytics. Uh, what we're starting to really realize is that LLM teams are getting split into two special niches. There's platform teams, right? And they, they own things like the, the infrastructure.

[3:25](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=205s) So who here has heard of a, a model gateway router? Uh, it's like an interface pattern. Behind it are all the models, right? Well, it turns out the central LLM platform team tends to own that.

[3:34](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=214s) They care about cost, latency, things like that. And then you have the other LLM teams. These t- these LLM teams sin- team, uh, sit on the, like, the outer side of the business, so like a hub and spoke.

[3:45](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=225s) They work for the business side. So these are like, uh, the people building the applications to help the business. So, um, if anyone here comes from, like, the, the ML or, or data science space, it's actually not far from that.

[3:56](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=236s) Um, and so different teams care about different metrics. Uh, so maybe if you're an AIPM sitting on the business side, you care about evals. If you care about, you know, the platform, maybe you care about costs, latency, things like that.

[4:08](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=248s) But TLDR, uh, observability... Oops. Uh, observability, um, represents what's happening. And now evals are really important in this space because the reality of the fact is, if you've ever seen a trace or something like that, um, you're not gonna inspect every single trace manually, right?

[4:26](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=266s) It is not scalable for you, an AI engineer, or you, the AIPM, to look through these things. So what is eval's use for? It's actually just a really clever word for signal.

[4:36](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=276s) You're just trying to understand what's going well and what's not going well. So I'm not here to sell you on, like, evals. I think everybody knows how important they are.

[4:43](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=283s) But if you think evals, evals are LLM-as-a-Judge only, uh, there's actually a lot of other tools that you're missing. So LLM-as-a-Judge, raise your hand if you used LLM-as-a-Judge. Okay, about half the room.

[4:55](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=295s) Um, it's super great. Uh, you use an LLM to give you feedback or process on, on any process, including an LLM process. So if you're doing RAG, this is a really good way to, to think about RAG in terms of evals.

[5:07](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=307s) Um, RAG would be like, hey, user has a question. Um, we retrieve some context to be able to, you know, possibly answer that question, and then we generate an answer.

[5:17](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=317s) Turns out every arrow on this is actually an eval you can run. So, hey, I retrieved some context, and I wanna compare that to the query being asked. Well, that's RAG relevance.

[5:26](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=326s) It's like, is the thing that I returned even helpful in answering the question? And so LLM-as-a-Judge is great. It's super helpful. Um, you know, I think most people understand why it works, but there's a whole, there's a whole research area on why they're really good indicators.

[5:41](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=341s) Um, the original task is not the eval task, right? So if I asked you, a human, "Hey, generate me a summary on, uh, something long and complex, like the book War and Peace," uh, that's a very different task than say, "Hey, I wrote this summary for you.

[5:56](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=356s) Is it a good one or is it a bad one?" But LLM-as-a-Judge is, is a small part. Um, it doesn't always have to be a large language model or autoregressive model.

[6:05](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=365s) Um, things like encoder-only BERT-type architectures are super helpful. They're about ten times cheaper, um, about one or two orders of magnitudes faster, uh, to run that eval. But, you know, you don't just have LLMs at your disposal.

[6:20](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=380s) You, a human, also are a really good way to discern signal. So it turns out evals can also come in the form of, is your user having a good or bad experience?

[6:27](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=387s) So for those people who've productionized, uh, some sort of LLM application, do you guys have user feedback? Raise your hand if you've implemented user feedback. And okay, about thirty percent.

[6:37](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=397s) Uh, it's actually incredible signal. Um, so that comes from a human. Obviously, you yourself can also generate labels on stuff. So has anyone here heard of a, a golden dataset?

[6:46](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=406s) Raise your hand. Okay, most of the room.

[6:50](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=410s) Um, and the way I encourage folks to think about this is a pro tip, actually. If the first column represents scale... So LLM-as-a-Judge is valuable because I don't have to grade it myself, right?

[7:00](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=420s) But let's say I don't necessarily trust it off the bat. Use the third column to help you out. So a golden dataset represents quality. So you yourself graded it.

[7:09](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=429s) You know that it's, it's what you expected, you know. Um, well, it turns out you can run your LLM-as-a-Judge on a golden dataset. Um, what you're trying to do is say, "Hey, can the LLM approximate the thing that I trust?"

[7:21](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=441s) Right? And what that allows you to do is to actually quantify and tune your LLM-as-a-Judge. So we'll go over that in a second. But strong pro tip, uh, most really strong LLM teams in the world kind of do this today.

[7:33](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=453s) Uh, and it turns out you don't always have to use an, an LLM, uh, or a human. You can use what are called, like, heuristics or code-based logic. So I'm gonna take you into the platform a little bit to talk through it.

[7:42](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=462s) But, um, in our platform, you have a way to, to run evals, great. Uh, what code evals actually are, um, are just, um... it's much cheaper, so I'll just run a little test here.

[7:53](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=473s) But, you know, let's say you wanna say, "Hey, does this output contain any keywords?" I don't need to use an LLM or a human for that. I can just use code.

[8:01](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=481s) It's infinitely cheaper, um, faster to run. Um, does this match this regex pattern XYZ? Is this a parsable JSON? So the reality of it is you have this kind of, uh, large toolbox in your kind of, uh, eval set.

[8:15](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=495s) So when you say evals, don't just think of LLM-as-a-Judge. There's a whole other set of smarter things that you can use that are actually more cost-effective. And so really, you know, this is a really good way to, to emphasize really the value of, like, the AI engineer evals and observability.

[8:31](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=511s) Most people understand this, this left-hand circle, this purple one. Um, it actually represents what we all wanna do, and it's like, hey, build a better AI system, right? So what you do is you collect data, observability, traces, things like that.

[8:43](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=523s) Then you run some evals to say, "Hey, did this process go well, or did this process not go well?" So you're discerning signal from that, you know, mass of data.

[8:52](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=532s) You'll actually collect where areas of things went right or wrong, right? And you'll say, "Hey, turns out we hallucinated on this. It's because our RAG strategy is off," or the agent is, is off, for example.

[9:03](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=543s) Um, you'll also annotate datasets as well just to double-check that those evals are correct. And then, of course, you always come back in, into your platform, and you, you know, you update the, the prompt template, right?

[9:14](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=554s) You change the model 'cause it's, it's wasn't good enough, or you update the agent orchestration. So everybody understands that left-hand circle. Now, a lot of people actually forget about the right-hand circle.

[9:24](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=564s) And so it turns out the first time you run evals, what you'll qu- quickly realize is that, um, they're not perfect, right? You actually have to tune those evals over time.

[9:35](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=575s) So the way you collect signal actually adjusts as your application also, you know, gets better. And so what I mean by that is that process of running evals, what you might notice if you annotate some of them, is that the eval said something hallucinated or wasn't correct, and it actually was, or vice versa.

[9:52](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=592s) Um, so what you actually need to do is collect a set of those failures, right? Say, "Hey, this is where the eval was wrong." And you'll know that by annotating some data every now and again.

[10:02](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=602s) Um, and then you'll want to improve the eval prompt template, right? 'Cause the way you collect signal at first, um, you'll quickly realize it's either too obscure, too vague, uh, not specific enough.

[10:12](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=612s) So these are the kind of two virtuous cycles that you really wanna get through very quickly. And the way I describe it to AI engineering teams is if you wanna build an, like, a quality AI product, uh, think about velocity.

[10:26](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=626s) So the faster you iterate through stuff, if you... I can get through four iterations in a month rather than two, you're gonna exponentially have a better AI product as you build.

[10:34](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=634s) And so when we talk about architectures and things like that, um, when the industry first started, this was state-of-the-art, um, routers, right? Um, am I right? Um, so routers are made up of, like, components.

[10:46](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=646s) Um, this is a really dumb example of Booking.com's Trip Planner. Um, so Booking, you know, they're one of our largest customers. Trip Planner is basically a, a travel agent in LLM form.

[10:56](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=656s) It drives revenue for that company. It helps you book, you know, it'll book your flights, your hotels. It'll give you an itinerary. And so, um, you know, when we think about evals, evals can be as complex as the applicat- application itself.

[11:10](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=670s) So in kind of older architectures, um, where there's things like routing, um, you can, you can eval individual components. I think most people get this when you're looking inside of a, a trace, for example.

[11:22](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=682s) Um, maybe I wanna eval a specific, uh, component or trace, so this one LLM call, right? But remember... Oops, I'll come up here. But remember that you can, um, okay.

[11:33](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=693s) Remember that you can zoom out too, so it doesn't have to just be this one specific component. Let's say this one component is part of an agent or a workflow.

[11:40](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=700s) Uh, maybe I just wanna evaluate the input/output of that larger workflow. So that larger workflow is made up of LLM calls, API calls, right? I have to find actual flights, actual hotels that have vacancy, um, maybe some heuristics.

[11:54](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=714s) And then you can zoom out a little bit more. Uh, maybe you want to eval things like the way control flow happens. It's a really important component. If you have components in your AI agents that have control flow in them, uh, it actually makes way more sense to eval your control flow first.

[12:09](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=729s) And you have conditional evals, meaning if you didn't get the control flow right, why eval anything down the line? 'Cause it's probably wrong, right? So save yourse- save yourself some, some money and some, um, some costs.

[12:21](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=741s) So you can think about conditional evals as well. And then of course, we have things like, um, people wanna run evals at, at the highest level. So imagine you have a back and forth, so we call this a session in, in our platform.

[12:33](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=753s) But, uh, the whole idea is, you know, uh, a session is made up of a series of traces. So you can imagine there's a back and forth between your...

[12:40](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=760s) you and your agent. I just wanna understand, hey, at any point, was the customer frustrated? Was the customer XYZ? So when you start to think about evals, there's no one-stop shop.

[12:50](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=770s) Uh, if anybody t- like, says, "This is how you should do evals," and they never asked you about how your application works, you probably shouldn't trust them. Also, I have a hot take, and my hot take is that don't use out-of-the-box evals.

[13:01](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=781s) If you get out-of-the-box... If you use out-of-the-box evals, you'll get out-of-the-box results. Um, so really customize them very heavily. It's something that we've learned really from some of the best teams in the world.

[13:12](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=792s) Um, okay, let me come here. Um, then you have complexity. Um, this is our own architecture for, uh, our AI copilot. Uh, we built an AI whose one purpose is to troubleshoot, observe, build evals for your AI system.

[13:25](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=805s) It obviously takes advantage of our platform, but you know, the reason why we go this route is take us forward five, 10 years from now. Do you guys really think that you, a human, are gonna be the ones who are evaluating all these AI systems, like, manually? [laughs]

[13:39](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=819s) Um, what do you think would actually take your place? Um, it's probably gonna be an AI that evaluates future AI. So this is our first iteration on this stuff.

[13:46](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=826s) Um, we're super excited about it. Um, you know, it's been out for a year. It's getting better and better. But maybe I'll show you a little bit of the workflows that we have in, in our platform really quickly.

[13:56](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=836s) Um, who here is, is working with agents? Okay, who here is interested in, like, agent evaluation? Okay, let's cover that then. Let's see. I'll show you... We'll show you how the industry is doing agent evals.

[14:08](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=848s) So with agent evals, things get, like, way more complex, right? The calls are longer. So when you look at your traces, they're much longer. Um, I'll actually show you, uh, our agent, um, traces.

[14:20](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=860s) So this is one that kinda failed. But our agent traces kinda works like this. So Copilot works like this. It basically, based off what you say and where you're at in the platform, there's e- there's agents that kind of can do things.

[14:32](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=872s) So, um, and it has tools. Each agent has access to a set of tools that it's particularly good at. Um, so the whole idea is that, yes, we can see what each individual, um, trace is doing, right?

[14:44](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=884s) We can say, "Hey, what's happening in this particular area?" We can look at the traces. But the reality of, of what people are actually asking in the space is not, "Is my AI agent good or bad?"

[14:56](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=896s) What they're actually asking is, "What are the failure modes in which my agent fails?" Right? And so what I mean by that is you can look at one individual, uh, trace and the graph view of it, but the reality is you wanna understand and discern the signal with your entirety of your AI agent.

[15:13](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=913s) So what is the d- like, what does the pathing look like across all of that particular AI agent's, um, calls? So for instance, if it had access to, to 10 tools, maybe you wanna answer questions like, how often did it call a specific tool, right?

[15:29](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=929s) Um, what were the evals in a specific path? So in our agent graph, for example, um, it's framework agnostic. So whether you use LangGraph, whether you use CrewAI, whether you use your hand-rolled code, um, this is an agnostic way to look at, um, you know, how, how an agent's pathing performs across the aggregate traces.

[15:50](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=950s) Um, and so this helps you understand, okay, if my agent that hits component one, then two, then three, uh, my evals look great. But for some reason, when we hit component four, then two, then three, our evals are dropping.

[16:03](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=963s) And the reason is why? Well, oh, it turns out component four had a dependency, right, on component three, and it needs that dependency in order to perform. And so when you think about the complexity of agent evals, um, you need kind of the ability to see across not one instantiation, but all of them.

[16:20](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=980s) You need to understand the distribution of what's happening. And so when we think about evals across agents, um, that's one way you can think about it. And then maybe a, an easier way to kinda think about it too is, um, you know, trajectory.

[16:33](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=993s) Uh, we're thinking about trajectory evals. So imagine for a second, um, you have this specific input, right? And the input is like, you know, "Hey, find me these hotels at, at Trip Planner."

[16:44](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1004s) And you know you should hit this component, then that component, this other component. So in this case, it's like start agent, tool agent. You might have a golden dataset, like very similar to how we have golden datasets for LLM-as-a-Judge, but this is for trajectories.

[16:57](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1017s) So I expect us to be able to hit at least three or four of these components, for example. So the reference trajectory is kind of mentioned, like I need to hit these components.

[17:05](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1025s) Then you get to do, do two, two things, one of two things. Either one, you can pass in like, here's what we did, here's what we expected into an LLM, and then an LLM can actually grade the trajectory.

[17:16](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1036s) Um, you can also just say, "Hey, did we explicitly hit these exact like trajectory strings?" Great. Um, but you don't always need a ground truth for that. You, you can start to get creative here.

[17:25](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1045s) You can say, "Hey, you know, here's this process that I expected to hit. Do, do the, does these nodes and the description of their nodes match the correct, correct trajectory," for instance.

[17:35](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1055s) Um, and then maybe you could do things like we're kind of playing around with this, but maybe here's the trajectory that I hit. Uh, here's the possible paths that are just possible.

[17:45](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1065s) Did I do well in these specific areas, right? And you can pass in the, the pathing as a series of like nested key value pairs, for example. LLMs are pretty good at that.

[17:53](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1073s) But we start to think about, um, you know, agent evals. You know, the eval space is already complex enough, and what we're seeing is even more complexity. Um, but hopefully that makes sense.

[18:04](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1084s) Um, I'll pause here. Um, hopefully that makes sense, but usually I like to make time for questions at the end to keep this pretty interactive. Um, hope that's okay, team.

[18:12](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1092s) But any questions? Does this make sense? Cool. No questions. Oh, yeah. Go ahead.

[18:20](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1100s) Most of the evals you're talking about are kind of like after the fact, right? Is there a way that you can-

[18:28](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1108s) Clearly a hallucination [door closing]

[18:31](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1111s) Yeah, incredible question. So a lot of people-- So there's evals that can be, um, you know, some people call them offline or online. Um, I like to say is, like, is it in the path-- Is it in orchestration or out of orchestration?

[18:43](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1123s) So for some people, there's a cost to in, in orchestration evals, and the cost is things like latency, right? Some people might call those a guardrail too. Like, "Hey, can I continue or not continue?"

[18:54](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1134s) Um, and so there's pros and cons to everything. I think when it comes to guardrails, this is something I, I kinda coach my customers. The way to think about guardrails, um, in general is you have system one.

[19:06](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1146s) System one is your orchestration system. It's what you built. It's your prompts. It's everything else. System two is your guardrail system, right? Guardrails are really nice because they, they mitigate risk, right?

[19:17](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1157s) But there is a cost, and the cost is maybe it's latency in your user's experience. You can get around that by doing smart things like maybe embeddings guardrails. They're, you know, two orders of magnitude shorter.

[19:29](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1169s) Um, but a lot of people don't think about the other two cons here. The other con is complexity. Two systems is complex, especially when one system checks in with the first.

[19:38](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1178s) The third thing is that a lot of people mistake guardrails as, like, the thing that needs to be adjusted. A lot of people will go to their guardrails first, like, "Oh, I need to adjust my guardrails."

[19:48](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1188s) The reality is you need to adjust system one. That's the root cause, right? Your guardrails are really there to protect you. And then maybe the last thing I'll say too is guardrails are not infa-fallible.

[19:59](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1199s) They're-- They kinda act like unit tests. They're for known knowns, right? Whereas observability plus evals... 'Cause the reality is you don't know the distribution of what you're gonna see until you get there, right?

[20:10](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1210s) Um, ask anybody who's built in the L-LLM space. Uh, their users are just crazy. [laughs] And so, um, that, that's the difference, and I, and I really caution people 'cause people are like, "Oh, I need to fix my guardrail."

[20:21](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1221s) No, no. Go fix the prompt [laughs] first and then worry about your guardrails. But yeah, inline, we call those inline, uh, evals. Some people call them guardrails. But really, do you do the evals in the orchestration or outside of it?

[20:33](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1233s) Um, and so you can-- there's pros and cons, so there's no right or wrong answer there, but good question.

[20:39](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1239s) Yeah. So, uh, when we have a complex system that is typically taking a long time to run and you have timeouts in this, and I know, uh, uh, you'll have something called a span that limits what, uh, uh, kind of, uh, view we're taking to, uh, a complex agent.

[20:57](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1257s) So if we have like a complex system that's really gonna take time, and then there's an asynchronous way, is there support to manage something like that and eval it across the whole system that we have modeled?

[21:08](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1268s) Oh, yeah. Amazing question. So, um, who here has ever heard of OTel? OpenTelemetry. Okay, even less than... Okay. Um, one of the most important things to our enterprise customers is, is being on OpenTelemetry.

[21:21](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1281s) Um, so you know how I said LLM teams are being split into two? Well, it turns out LLM services are also being split across services. And so the idea is, like, people wanna understand, um, so maybe an asynchronous process in one service or one Docker container or, you know, one, one Docker or one Kubernetes pod.

[21:38](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1298s) Um, OTel propagation is a great way to get around that, meaning you can have a process like application A sends data to my model router, right? And then that comes back to application A.

[21:50](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1310s) Then application A hits application B for some reason, and then it comes back to A. When you're actually creating those traces, you wanna be able to see all that work, right?

[21:58](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1318s) You don't wanna just instrument one particular thing. You wanna see it across, work across. And so OTel is an incredible pattern for that. It's a solved problem. So that's why we at Arize two and a half years ago when this crazy time started for all of us, we made a bet to be OTel first, and it's, it's

[22:14](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1334s) really paid off. Um, yeah.

[22:16](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1336s) Awesome.

[22:17](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1337s) Yeah.

[22:17](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1337s) That's all.

[22:18](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1338s) So, uh, I just wanted to ask, like in your experience, is, uh, is there a good way to get the kind of confidence score on the like model eval?

[22:26](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1346s) Because I imagine if I, if you have that, you can, you know, identify the value, values like

[22:32](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1352s) Okay. So confidence scores on evals, right? Um, yeah, I think it depends where you're getting your eval, if it's from an autoregressive model. Um, companies like OpenAI have actually exposed the log prob.

[22:42](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1362s) So the log probability is pseudo like confidence of like... And since you're returning only one token and that token is like the, the eval label, log prob is a really good way for those autoregressive models.

[22:54](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1374s) If you're using things like, um, small language models, encoder-only models, they come with a probability, uh, of the classification. Um, but really, yeah, it's tough, but you have a bunch of tools in your toolbox, and you generally use them together to discern where things go well or not well.

[23:09](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1389s) But log prob, if you're using, uh, a model provider that exposes the log prob is, is a really good way to start for autoregressive models.

[23:18](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1398s) Okay. Last question, and then we time up. Yeah.

[23:20](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1400s) Hey. Uh, are, do you have anything in your plans like going forward, uh, like how to shorten the loop between customer feedback and automatically improving the prompts as opposed to having like, you know, the development team have to look at it?

[23:32](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1412s) Oh, good question. Yeah. We wanna automate in that area definitely. So who here has heard of DSPy?

[23:38](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1418s) All right. Okay. If you, if you didn't raise your hand on any of this, I hope you learned a bunch. Um, DSPy obviously has something like MIPRO. Uh, MIPRO is an optimizer.

[23:46](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1426s) You get like thirty inputs, thirty outputs, and then, um, it basically creates less fragile prompts that span across different models. Like, so it does-- it works for OpenAI, and then it works for Gemini, et cetera.

[23:56](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1436s) Um, in terms of like auto optimization, yeah, I think we have the ability to... or we're releasing the ability to run, uh, prompt optim-optimization. So some people call it-- We call it meta-prompting.

[24:08](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1448s) But basically, we feed it a dataset. We said, "Here's the input-output pairs. Here's the evals on those things, uh, and there's where things failed and didn't fail. Look at the original prompts.

[24:18](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1458s) Look at this dataset. Can you give me a new prompt that fixes this dataset?" Yeah. So we, we have-- we call that meta-prompting, but it's basically use an LLM to just automate so you don't have to.

[24:29](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1469s) Yeah. But good question. But really appreciate the time. Uh, we're over at the booth. Uh, feel free to come grab me if you want to talk architecture or, or anything.

[24:36](https://www.youtube.com/watch?v=spvXj9tnWAQ&t=1476s) But really nice to see you all. [audience applauding] [upbeat music]
