# The Future of Evals: From LLM as a Judge to Agent as a Judge — Aparna Dhinakaran, Arize AI

Aparna Dhinakaran, Arize AI

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/q2JrUKBMf0w-future-evals-from-llm-as-judge-agent) · [Video](https://www.youtube.com/watch?v=q2JrUKBMf0w)

## From LLM as a Judge to Agent as a Judge

As agents gain tools, memory and longer trajectories, evaluation must examine how they work—not just whether their final answers satisfy a rubric.

Familiarity with tool-using agents and basic LLM evaluation will help; a trace is a record of the steps an agent takes during a task.

### What do production evaluations actually catch?

[0:35](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=35s)

How do you know whether an agent is working once real users start interacting with it? Pre-release tests provide one view; **production traces** provide another, exposing what the agent actually did. Aparna Dhinakaran, Arize AI’s cofounder, draws on teams evaluating both agents under development and agents already serving users.

Dhinakaran reports that Arize runs over 100 million evaluations per month, with the average team running about 12 evaluation jobs and top teams running over 3,800 evaluators. These are reported operational figures; the talk does not specify the reporting window, team population or counting definitions.

Offline and online evaluations each have a place. The focus here is evaluation over traces: using recorded agent behavior to identify what works, catch failures and supply evidence for continual learning loops.

Dhinakaran invokes enthusiasm for evaluations from leaders associated with Anthropic and OpenAI, along with Greg Brockman and Gary Tan. But that consensus leaves a practical question unanswered: does adding evaluations mean you now catch every failure?

![Slide titled “Evals were going to solve everything,” with four quotation cards and a four-panel meme questioning whether added evals catch failures.](https://ai.engineer/api/talk-essay-media/3bbdc2fb343099c7c3c94a03de28f6b02c222d55eb3ae276bce762628d067d19)

Evals were going to solve everything—but would they catch the failures?

### The evaluation target changed

[2:00](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=120s)

While teams were building their first generation of evaluations, the systems being evaluated changed underneath them. Dhinakaran frames the progression as three stages:

| Stage | What the system does |
| --- | --- |
| 2023 | Answers a prompt |
| 2024 | Adds tool calls, reasoning and deep research |
| The agents described here | Runs loops over real-world data and launches sub-agents for long-horizon tasks |

This is her framing of the capability shift, rather than a precise chronology of individual model releases.

Each stage adds more than complexity. It changes the kinds of failure an evaluator must recognize. An answer can be assessed as an output; an agent also has a **trajectory**—the sequence of actions through which it attempts a task. As that sequence becomes longer and more variable, failures can arise in the process itself.

### Alyx: more capability, new failure modes

[2:47](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=167s)

Arize encountered this shift in [Alyx](https://community.arize.com/x/arize-news/1eueb8d2lutm/introducing-alyx-20-the-adaptive-ai-engineering-ag), its agent embedded in the product UI. As frontier labs introduced new functionality, the team incorporated it. Alyx gained longer memory, the ability to create dynamic interfaces and search across an enormous volume of traces. Yet it still forgot context, failed to recognize when a task was done and sometimes became stuck in loops.

These failures exposed gaps in Arize’s classical **LLM-as-a-Judge** evaluations. A rubric that covered an expected behavior did not necessarily catch every failure the expanded agent could exhibit. The accompanying slide broadens the picture with failure categories including model–harness fit and false completion, beside an Alyx trace interface.

![“Failures got more complex” slide listing nine failure categories beside an Alyx trace interface, including model–harness fit and false completion.](https://ai.engineer/api/talk-essay-media/5b26faed474c0afc7c96cdc5bccfc42bb1d43ae6b3d50748f967d015e338e8f4)

Alyx’s failure modes span context, tools, execution, permissions, recovery, loops, and trajectories.

The departure from a deterministic flow was especially visible in the UI: each user interaction could produce a new interface and a different trajectory. The evaluator therefore had to account for behavior that varied with the task and interaction. That led to the central proposal: **evaluate an agent with another agent**, capable of adapting its analysis to the path taken.

### Three complementary evaluation methods

[3:54](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=234s)

Agent judging adds another evaluation tool; deterministic checks and classical LLM judges retain their roles. The slide maps the three methods to different failure types:

| Method | Focus in the comparison |
| --- | --- |
| Deterministic code evaluators | Deterministic code behavior |
| Classical LLM-as-a-Judge | Prompt-level failures such as hallucinations |
| Agent-as-a-Judge | Trajectory and harness failures |

The useful distinction is what each evaluation method can investigate and how it performs that investigation.

Dhinakaran contrasts a classical judge applying a predefined rubric and returning fixed scores with **Agent-as-a-Judge** performing adaptive, dynamic analysis. When user inputs lead to different trajectories, the investigation can vary with the behavior encountered. This is a comparison of evaluation workflows, not a claim that an LLM judge is inherently unable to inspect a trajectory.

Her expectation is that teams will combine all three methods. Most teams, in her account, already use the first two; agent judging extends their coverage to failures that require a more adaptive investigation.

![Three-column diagram maps deterministic code to code evaluators, prompt hallucinations to classic LLM-as-Judge, and trajectory and harness failures to Agent-as-Judge.](https://ai.engineer/api/talk-essay-media/bd765572bfeea891e91a7d90c76f0ba2a7cc68ed85a76e74f28680c85ebbd28a)

Different failures call for code evaluators, classic LLM judges, and agent judges.

### Signal: from traces to proposed fixes

[4:35](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=275s)

Dhinakaran announces Arize’s release of Agent-as-a-Judge and introduces [Signal](https://arize.com/blog/building-ai-factory-self-improving-agents-arize-ax/) as a long-running agent that reads submitted traces and discovers patterns of issues. The aim is to find problems beyond those anticipated by a predefined judging rubric.

She reports that Signal helped uncover subtle failures: loops that continued repeatedly, prolonged calls to the same tool and inefficient trajectories. These are properties of an execution path. Recognizing them requires examining how the agent progressed through its work, including where it repeated actions without useful progress.

The accumulated analysis can then support a proposed repair. Dhinakaran says the agent can open a pull request with a fix, connecting failure discovery to a concrete change for review. The dashboard makes that handoff visible through its detected issues and the selected issue’s Overview, Evidence and Proposed fix headings. A proposed fix is the endpoint shown here; no completed merge or deployment is demonstrated.

![Arize Signal dashboard with an issues donut chart, issues-over-time bars, a detected-issues list, and a selected issue showing Overview, Evidence, and Proposed fix headings.](https://ai.engineer/api/talk-essay-media/c3253898205518242748654367172b31fdf980c9e32a43d6fbc4e0c3819208fd)

Arize Signal shows detected issues and evidence for a selected retry-loop failure.

## Resources

- [Signal and the Arize AX agent improvement loop](https://arize.com/blog/building-ai-factory-self-improving-agents-arize-ax/) — Aparna Dhinakaran and Jason Lopatecki explain Signal, managed investigations, agent judging and experiments in this June 2026 product overview.
- [Introducing Alyx 2.0](https://community.arize.com/x/arize-news/1eueb8d2lutm/introducing-alyx-20-the-adaptive-ai-engineering-ag) — The Alyx 2.0 announcement describes adaptive planning and workflows across traces, datasets, experiments and evaluation.
- [Build evaluators in Arize AX](https://arize.com/docs/ax/evaluate/create-evaluators) — Current guidance for choosing evaluator types, defining scoring scope and building checks over agent traces.
- [Improve an agent harness with traces and evals](https://arize.com/blog/improve-ai-agents-traces-evals-harness/) — A worked PM-agent example connects tracing, evaluator criticism and targeted changes to prompts, tools and context.

## Slides

### Evals are growing quickly

Arize reports 100M evals every month alongside 3,800+ eval jobs for top AI teams and 12.3 for the average team. The talk focuses on evals run on traces.

[1:16](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=76s) [Customer logos alongside statistics for monthly evaluations and evaluation jobs.](https://assets.ai.engineer/sites/aiecode2025/sha256/4aa6da85f166088204036bbff0ac5e3a382a348f4896536799b2c2c8478b2c73.webp)

Evals are growing quickly

reddit
Booking.com
Uber
Tripadvisor
ATLASSIAN
wayfair
DOORDASH
klaviyo
PagerDuty
Typeform
priceline
Handshake

100M evals run every month
3,800+ eval jobs run by the top AI teams
12.3 eval jobs the average team runs

arize
02 / 10

### Evals were going to solve everything

Industry leaders praise evals, while a meme questions whether adding them will catch failures.

[1:41](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=101s) [Four quotations about evals beside a meme questioning whether evals catch failures.](https://assets.ai.engineer/sites/aiecode2025/sha256/6c622c5657fb22f4c82629f01aa1da0d334c31863eceab5062b2e10d0a08fb6a.webp)

Evals were going to solve everything

Kevin Weil
OpenAI CPO
Writing evals is going to become a core skill for product managers.

Mike Krieger
Anthropic CPO
Writing evals is probably the most important thing we can teach people.

Garry Tan
@garrytan
Evals are emerging as the real moat for AI startups.

Greg Brockman
@gdb
evals are surprisingly often all you need

We added evals
So they'll catch the failures, right?
They'll catch the failures, right?

arize

### Agents got more complex

The chart traces rising task complexity from answering prompts in 2023 to running sub-agents over long horizons in 2026, passing through tool use, reasoning, computer use, and independent research and coding.

[1:59](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=119s) [A rising timeline progresses from answering prompts to running sub-agents over long horizons.](https://assets.ai.engineer/sites/aiecode2025/sha256/cb3e60a617bb1a72ef2957ba30c05565bf270159e8e9d6dccd4b9228e4898faf.webp)

Agents got more complex

Y-axis: TASK COMPLEXITY ↑
Timeline: 2023, 2024, 2025, 2026
X-axis: TIME →

Answer a prompt
GPT-4

Call functions & run code
Function calling · Code Interpreter

Use tools & browse the web
Claude tool use · GPT-4o

Reason step-by-step
o1 · MCP

Operate a computer
Operator · computer use

Research & code on their own
Deep Research · Claude Code

Run sub-agents over long horizons
GPT-5.6 · Dynamic Workflows

arize

### Failures got more complex

Arize found Alyx could lose context, get stuck in loops or misjudge completion. Its existing LLM-as-a-Judge evaluations did not catch every failure the team experienced.

[2:38](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=158s) [Nine agent failure modes, including context, tools, loops and false completion, beside a trace interface.](https://assets.ai.engineer/sites/aiecode2025/sha256/304d9144aa85f12959d9d479def15ec3fe8dffeccae96e009244afe3ac6e0a76.webp)

Failures got more complex

We saw this with our own agent Alyx.

1 Context — seen late or lost in long run
2 Tool — wrong tool, bad args, stale schema
3 Execution — sandbox ≠ prod; deps, timeouts
4 Permission — blocked, over-permitted, unsafe
5 Recovery — repeats errors, false success
6 Loops — Repeats the same step
7 Model–harness fit — differs across models
8 Trajectory — ran, but not effectively
9 False Done — Claims success midway

arize

Interface text:
Alyx
copilot-prod
Signal 27
Evals & Metrics 5
Arize Default
Traces 4.13k
Spans 52.92k
4.13k traces
Status
Kind
AGENT

Trace: c969d7d3 ... caa33138
Session: 48825
Status OK
Cost $0.364943
Start Time 6/26/2026, 11:51:26 AM
Trace Tree 41
Timeline
Agent Graph
Search by span name
ROUTER-TRACE_AGENT
1 eval 62.83s
one_alyx_agent
62.54s
Orchestrator - RunIteration 1
3.1s, 30,229 tk, $0.105158
get_trace_data
0.15s
GQL query GqlSpanRecords...
0.08s
Orchestrator - RunIteration 2
2.69s, 31,116 tk, $0.012054
jq
0.01s
Orchestrator - RunIteration 3
3s, 31,525 tk, $0.012624
grep_json
0.02s
Orchestrator - RunIteration 4
1.89s, 31,700 tk, $0.011739
jq
0.01s
Orchestrator - RunIteration 5
2.13s, 32,071 tk, $0.012114
grep_json
0.01s
Orchestrator - RunIteration 6
1.7s, 32,236 tk, $0.011619
jq
0.01s
Orchestrator - RunIteration 7

AGENT ROUTER-TRA[illegible]
Jailbreak Detection: saf[illegible]
Attributes
Input / Out[illegible]
Search
input 1 item
value 3 items
type
question
input_context 7 [illegible]
trace_ids [illegible]
[0]
span_ids [illegible]
[0]
highlighted_c[illegible]
tracing_page_[illegible]
page
project_id
view
dataset 5 [illegible]
startTime
endTime
timeZone
queryFilt[illegible]
subQuery
columnVisib[illegible]
attributes
name
input_val[illegible]
output_v[illegible]

### The best way to evaluate an agent is another agent

Dhinakaran proposes using another agent to evaluate Alyx after classical LLM-as-a-judge evals proved insufficient for its changing trajectories.

[3:43](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=223s) [A statement proposing another agent as the best way to evaluate an agent.](https://assets.ai.engineer/sites/aiecode2025/sha256/3fc74cbb2174e570138bc3d7d3d5d1313a59d83507af4282903099626cfbc037.webp)

The best way to evaluate an agent is another agent

arize

### Different Evals for Different Failures

Code evaluators check deterministic behavior; an LLM judge scores a known, well-defined failure. Agent-as-Judge reviews the whole trajectory, using traces, tools and other context to identify failure modes.

[3:52](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=232s) [Three columns compare code evaluators, classic LLM-as-judge and agent-as-judge for different failures.](https://assets.ai.engineer/sites/aiecode2025/sha256/8c2dfb430844cd31a8cf2c2f98a393817c07ec316713c9715efe4ffb9e4cf37e.webp)

Different Evals for Different Failures

Deterministic code
Code Evaluators
Unit tests, assertions, rules.

✓ valid_json(output)
✓ status == 200
✓ has_required_fields()
✓ no_pii_leaked(text)
PASS 4 / 4 assertions · 0.02s

Prompt hallucinations
Classic LLM-as-Judge
Score a known, well-defined failure.

input
output
criteria
LLM
label · score · explanation

Trajectory & harness failures
Agent-as-Judge
Another agent reviews the whole trajectory.

traces
filesystem
skills
tools
Agent
subagents
insights
failure modes
evidence

arize

### Arize Signal

Signal reads traces to find issue patterns. Its dashboard shows 27 issues over the last 30 days, including a retry loop that cancels the parent stream.

[4:33](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=273s) [Signal dashboard showing issue counts, a timeline, and details of a detected retry loop.](https://assets.ai.engineer/sites/aiecode2025/sha256/b657c3e3e84d981d5a7c6ae8e6cc86e45a02cc2ee1804141a7ce818ffee7c80d.webp)

Arize Signal

Issues Found
27 issues
Entity | Count
ROUTER-HOME_PAGE | 4
run_mapping_fix_sub_agent | 3
playground_agent | 3
search_agent | 2

Issues Over Time
Y-axis ticks: 0, 4, 8
Timeline: Jun 9, Jun 10, Jun 14, Jun 15, Jun 16, Jun 17, Jun 19, Jun 22, Jun 24, Jun 27, Jun 28, Jun 29, Jun 30

Detected Issues
#1 todo_update retry loop on already-in-progress todo causes stream cancellation
A ‘ROUTER-PROMPT_OPTIMIZATION’ session ended with ‘status_code=ERROR’ and...
6 matching traces

#2 ROUTER-HOME_PAGE generates over-verbose responses causing 20–75s latency
Reconfirmed: a ROUTER-HOME_PAGE session took 83.57s end-to-end. The elevated latency in this run...
33 matching traces

todo_update retry loop on already-in-progress todo causes stream cancellation
todo_update
6/17/2026, 6:24 AM

Overview
A ROUTER-PROMPT_OPTIMIZATION session ended with status_code=ERROR and status_message=stream_cancelled. The export contains 10 todo_update spans across the batch, and this is the same pattern previously identified where a todo_update retry loop on an already-in-progress item cancels the parent stream.

Evidence
• trace [illegible] — ROUTER-PROMPT_OPTIMIZATION, ERROR, stream_cancelled. User was working on prompt integration (partial message visible in attributes).

Proposed fix

Interface text:
copilot-prod
ID
Jun 19 - Jun 26
Live
Online Evaluators 2
Signal 27
Evals & Metrics 5
Traces
Spans
Sessions
Agent Graph
Agent Path
Signal Issues
Active
arize-ai/arize
Viewing Signal’s last 30 days of issues
Mute
Mark as fixed
Next Steps
Add to Dataset
Create Evaluator
Open PR
arize

### Trace. Eval. Learn

Aparna Dhinakaran invites attendees to the Arize booth for a demo and to the Evals track in room 2005.

[5:19](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=319s) [Arize closing slide with a QR code and speaker name.](https://assets.ai.engineer/sites/aiecode2025/sha256/6ee52abde1a0acff6d9ca4f313832654b241458ae5e67b6f96594749536296e6.webp)

arize

Trace. Eval. Learn

Aparna Dhinakaran    @aparnadhinak

bitly

## Full transcript

[0:00](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=0s) [on-hold jingle] Awesome. Well, hey, everyone. My name's Aparna, one of the founders of Arize.

[0:16](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=16s) We work with some amazing teams to help them build evals. Um, and we have an incredible lineup of talks for you all today at the Evals Track. Um, it's happening in room two zero f- zero five, and there's gonna be amazing speakers from Termbench and Uber and Snorkel kind of all happening after this.

[0:35](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=35s) Um, but today I'm here to talk to you about the future of evals. Evals have gone from the new skill that every PM and every AI engineer has to learn to the thing that every serious AI team is betting on.

[0:50](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=50s) We've been really fortunate to get to work with some of the best AI teams in the world, so we get a front row seat into not just what's happening when they're building their actual agents and before they actually ship, but actually the evals that teams are running on their live production agent via their traces.

[1:09](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=69s) Little bit of some stats for you guys. We run over a hundred million evals every month. The average team runs about twelve different eval jobs with the top teams running over thirty-eight hundred different evaluators.

[1:22](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=82s) And offline evals, online evals, they each have their own place, but today what I'm actually gonna talk to you about is the teams that are running evals on their traces.

[1:32](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=92s) This is actually what's helping teams figure out what's working, catch their failures, and that's the type of data you need to fuel your continual learning loops.

[1:43](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=103s) And the industry kind of agrees. I mean, all the CPOs of Anthropic, OpenAI, all, all, you know, GDB, you have Gary Tan saying, "Evals are everything you need," and the whole industry kind of agrees.

[1:55](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=115s) So we added evals, they catch all the failures, right?

[2:00](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=120s) Here's the problem. While we were building all of these first gen evals, the thing that we were actually evaluating has changed underneath us. In twenty twenty-three, it was about just answering a prompt.

[2:12](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=132s) In twenty twenty-four, we started to see all different tier models. They've added tool calls, they've added reasoning, they've added deep research. Now what we have is teams running loops on real-world data with sub-agents kicked off on, um, long horizon tasks.

[2:30](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=150s) Every one of these was actually a massive jump in complexity, and we didn't just make the problem harder, we actually got a fundamentally different type of problem. What that meant is that as these systems got more complex, so did the way that they actually fail.

[2:47](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=167s) We're really lucky 'cause we have our own agent that we've built, Alyx, that lives in our UI, and we get a f- k- kinda get to feel this pain ourselves.

[2:54](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=174s) Every time the frontier labs added new functionality, we added it to our agent. And now Alyx can-- has much longer memory. It has the ability to create dynamic UIs.

[3:05](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=185s) It can go search across an enormous volume of traces. But we also realized that it would forget context. It wouldn't know when something was done. Um, sometimes it would just get stuck in these loops.

[3:17](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=197s) And the key thing here is that the classical LLM-as-a-Judge evals that probably many of you have written in this room just weren't enough for us to be able to catch all the types of failures that we were experiencing.

[3:32](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=212s) I mean, it's just fundamentally different, right? You have a deterministic flow, and now what we have is literally every time a user interacted with Alyx, it would create a new UI.

[3:42](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=222s) That's a fundamentally different trajectory. So this led to our really big revelation. What if the best way to evaluate an agent was actually with an agent?

[3:54](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=234s) Doesn't mean that all of the ways that we did evals with deterministic evals, with LLM-as-a-Judge, classic evals doesn't matter anymore, but it just means that we have a different type of tool to solve a different type of problem.

[4:07](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=247s) Agent as a Judge is about adaptive dynamic analysis. LLM as a Judge just gives you a fixed rubric with these fixed scores. It's what everyone's doing. But when your agent's doing completely different trajectories every time a user puts in data, it just means that you need a fundamentally different type of eval.

[4:27](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=267s) My take is that most teams today are doing the first two, but the future of evals is actually having all three.

[4:35](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=275s) And today I'm actually excited to share we've released Agent as a Judge, um, to help our teams on their eval journey. We've released Signal. Signal is actually a long-running agent that can read traces sent in, discover patterns of issues, um, it can figure out types of problems that a classical LLM-as-a-Judge eval just would never be able to

[4:58](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=298s) do with these deterministic rubrics. It's helped us figure out, um, very subtle failures that you wouldn't even think of doing, such as something going on in a loop for multiple times, it was calling the same tool, uh, for a repeatedly long time, the trajectory was inefficient.

[5:13](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=313s) And actually what this does is, because it has all that analysis, it can go put up a PR and put up a fix. So if you wanna learn more, come to our, come to our booth.

[5:23](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=323s) We're right by the OpenAI booth. We'll give you a demo. We'll show you a bit more about it. Um, we're also, like I said, taking over the Evals track, so come to room two zero zero five.

[5:33](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=333s) We're gonna be talking a lot about the future of evals and what they look like. And if you just wanna hang out with our team, we're throwing a viewing party for the USA World Cup, uh, game tonight.

[5:44](https://www.youtube.com/watch?v=q2JrUKBMf0w&t=344s) So, uh, check out the Luma and register to come join us. Awesome. Thank you all so much. [applause] [upbeat music]
