AI Engineer World's Fair 2026
The Future of Evals: From LLM as a Judge to Agent as a Judge — Aparna Dhinakaran, Arize AI
Read the talk
From LLM as a Judge to Agent as a Judge
As agents gain tools, memory and longer trajectories, evaluation must examine how they work—not just whether their final answers satisfy a rubric.
From a talk by Aparna Dhinakaran
Before you start: Familiarity with tool-using agents and basic LLM evaluation will help; a trace is a record of the steps an agent takes during a task.
What do production evaluations actually catch?
How do you know whether an agent is working once real users start interacting with it? Pre-release tests provide one view; production traces provide another, exposing what the agent actually did. Aparna Dhinakaran, Arize AI’s cofounder, draws on teams evaluating both agents under development and agents already serving users.
Dhinakaran reports that Arize runs over 100 million evaluations per month, with the average team running about 12 evaluation jobs and top teams running over 3,800 evaluators. These are reported operational figures; the talk does not specify the reporting window, team population or counting definitions.
Offline and online evaluations each have a place. The focus here is evaluation over traces: using recorded agent behavior to identify what works, catch failures and supply evidence for continual learning loops.
Dhinakaran invokes enthusiasm for evaluations from leaders associated with Anthropic and OpenAI, along with Greg Brockman and Gary Tan. But that consensus leaves a practical question unanswered: does adding evaluations mean you now catch every failure?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The evaluation target changed
While teams were building their first generation of evaluations, the systems being evaluated changed underneath them. Dhinakaran frames the progression as three stages:
| Stage | What the system does |
|---|---|
| 2023 | Answers a prompt |
| 2024 | Adds tool calls, reasoning and deep research |
| The agents described here | Runs loops over real-world data and launches sub-agents for long-horizon tasks |
This is her framing of the capability shift, rather than a precise chronology of individual model releases.
Each stage adds more than complexity. It changes the kinds of failure an evaluator must recognize. An answer can be assessed as an output; an agent also has a trajectory—the sequence of actions through which it attempts a task. As that sequence becomes longer and more variable, failures can arise in the process itself.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Alyx: more capability, new failure modes
Arize encountered this shift in Alyx, its agent embedded in the product UI. As frontier labs introduced new functionality, the team incorporated it. Alyx gained longer memory, the ability to create dynamic interfaces and search across an enormous volume of traces. Yet it still forgot context, failed to recognize when a task was done and sometimes became stuck in loops.
These failures exposed gaps in Arize’s classical LLM-as-a-Judge evaluations. A rubric that covered an expected behavior did not necessarily catch every failure the expanded agent could exhibit. The accompanying slide broadens the picture with failure categories including model–harness fit and false completion, beside an Alyx trace interface.
The departure from a deterministic flow was especially visible in the UI: each user interaction could produce a new interface and a different trajectory. The evaluator therefore had to account for behavior that varied with the task and interaction. That led to the central proposal: evaluate an agent with another agent, capable of adapting its analysis to the path taken.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Three complementary evaluation methods
Agent judging adds another evaluation tool; deterministic checks and classical LLM judges retain their roles. The slide maps the three methods to different failure types:
| Method | Focus in the comparison |
|---|---|
| Deterministic code evaluators | Deterministic code behavior |
| Classical LLM-as-a-Judge | Prompt-level failures such as hallucinations |
| Agent-as-a-Judge | Trajectory and harness failures |
The useful distinction is what each evaluation method can investigate and how it performs that investigation.
Dhinakaran contrasts a classical judge applying a predefined rubric and returning fixed scores with Agent-as-a-Judge performing adaptive, dynamic analysis. When user inputs lead to different trajectories, the investigation can vary with the behavior encountered. This is a comparison of evaluation workflows, not a claim that an LLM judge is inherently unable to inspect a trajectory.
Her expectation is that teams will combine all three methods. Most teams, in her account, already use the first two; agent judging extends their coverage to failures that require a more adaptive investigation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Signal: from traces to proposed fixes
Dhinakaran announces Arize’s release of Agent-as-a-Judge and introduces Signal as a long-running agent that reads submitted traces and discovers patterns of issues. The aim is to find problems beyond those anticipated by a predefined judging rubric.
She reports that Signal helped uncover subtle failures: loops that continued repeatedly, prolonged calls to the same tool and inefficient trajectories. These are properties of an execution path. Recognizing them requires examining how the agent progressed through its work, including where it repeated actions without useful progress.
The accumulated analysis can then support a proposed repair. Dhinakaran says the agent can open a pull request with a fix, connecting failure discovery to a concrete change for review. The dashboard makes that handoff visible through its detected issues and the selected issue’s Overview, Evidence and Proposed fix headings. A proposed fix is the endpoint shown here; no completed merge or deployment is demonstrated.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Aparna Dhinakaran and Jason Lopatecki explain Signal, managed investigations, agent judging and experiments in this June 2026 product overview.
The Alyx 2.0 announcement describes adaptive planning and workflows across traces, datasets, experiments and evaluation.
Further reading
A worked PM-agent example connects tracing, evaluator criticism and targeted changes to prompts, tools and context.
Updates since the talk
Current guidance for choosing evaluator types, defining scoring scope and building checks over agent traces.
Read the complete timestamped transcript
- 0:00
[on-hold jingle] Awesome. Well, hey, everyone. My name's Aparna, one of the founders of Arize.
- 0:16
We work with some amazing teams to help them build evals. Um, and we have an incredible lineup of talks for you all today at the Evals Track. Um, it's happening in room two zero f- zero five, and there's gonna be amazing speakers from Termbench and Uber and Snorkel kind of all happening after this.
- 0:35
Um, but today I'm here to talk to you about the future of evals. Evals have gone from the new skill that every PM and every AI engineer has to learn to the thing that every serious AI team is betting on.
- 0:50
We've been really fortunate to get to work with some of the best AI teams in the world, so we get a front row seat into not just what's happening when they're building their actual agents and before they actually ship, but actually the evals that teams are running on their live production agent via their traces.
- 1:09
Little bit of some stats for you guys. We run over a hundred million evals every month. The average team runs about twelve different eval jobs with the top teams running over thirty-eight hundred different evaluators.
- 1:22
And offline evals, online evals, they each have their own place, but today what I'm actually gonna talk to you about is the teams that are running evals on their traces.
- 1:32
This is actually what's helping teams figure out what's working, catch their failures, and that's the type of data you need to fuel your continual learning loops.
- 1:43
And the industry kind of agrees. I mean, all the CPOs of Anthropic, OpenAI, all, all, you know, GDB, you have Gary Tan saying, "Evals are everything you need," and the whole industry kind of agrees.
- 1:55
So we added evals, they catch all the failures, right?
- 2:00
Here's the problem. While we were building all of these first gen evals, the thing that we were actually evaluating has changed underneath us. In twenty twenty-three, it was about just answering a prompt.
- 2:12
In twenty twenty-four, we started to see all different tier models. They've added tool calls, they've added reasoning, they've added deep research. Now what we have is teams running loops on real-world data with sub-agents kicked off on, um, long horizon tasks.
- 2:30
Every one of these was actually a massive jump in complexity, and we didn't just make the problem harder, we actually got a fundamentally different type of problem. What that meant is that as these systems got more complex, so did the way that they actually fail.
- 2:47
We're really lucky 'cause we have our own agent that we've built, Alyx, that lives in our UI, and we get a f- k- kinda get to feel this pain ourselves.
- 2:54
Every time the Frontier Labs added new functionality, we added it to our agent. And now Alyx can-- has much longer memory. It has the ability to create dynamic UIs.
- 3:05
It can go search across an enormous volume of traces. But we also realized that it would forget context. It wouldn't know when something was done. Um, sometimes it would just get stuck in these loops.
- 3:17
And the key thing here is that the classical LLM-as-a-Judge evals that probably many of you have written in this room just weren't enough for us to be able to catch all the types of failures that we were experiencing.
- 3:32
I mean, it's just fundamentally different, right? You have a deterministic flow, and now what we have is literally every time a user interacted with Alyx, it would create a new UI.
- 3:42
That's a fundamentally different trajectory. So this led to our really big revelation. What if the best way to evaluate an agent was actually with an agent?
- 3:54
Doesn't mean that all of the ways that we did evals with deterministic evals, with LLM-as-a-Judge, classic evals doesn't matter anymore, but it just means that we have a different type of tool to solve a different type of problem.
- 4:07
Agent as a Judge is about adaptive dynamic analysis. LLM as a Judge just gives you a fixed rubric with these fixed scores. It's what everyone's doing. But when your agent's doing completely different trajectories every time a user puts in data, it just means that you need a fundamentally different type of eval.
- 4:27
My take is that most teams today are doing the first two, but the future of evals is actually having all three.
- 4:35
And today I'm actually excited to share we've released Agent as a Judge, um, to help our teams on their eval journey. We've released Signal. Signal is actually a long-running agent that can read traces sent in, discover patterns of issues, um, it can figure out types of problems that a classical LLM-as-a-Judge eval just would never be able to
- 4:58
do with these deterministic rubrics. It's helped us figure out, um, very subtle failures that you wouldn't even think of doing, such as something going on in a loop for multiple times, it was calling the same tool, uh, for a repeatedly long time, the trajectory was inefficient.
- 5:13
And actually what this does is, because it has all that analysis, it can go put up a PR and put up a fix. So if you wanna learn more, come to our, come to our booth.
- 5:23
We're right by the OpenAI booth. We'll give you a demo. We'll show you a bit more about it. Um, we're also, like I said, taking over the Evals track, so come to room two zero zero five.
- 5:33
We're gonna be talking a lot about the future of evals and what they look like. And if you just wanna hang out with our team, we're throwing a viewing party for the USA World Cup, uh, game tonight.
- 5:44
So, uh, check out the Luma and register to come join us. Awesome. Thank you all so much. [applause] [upbeat music]