AI Engineer World's Fair 2025
How to Run Evals at Scale: Thinking Beyond Accuracy or Similarity
Read the talk
How to Run Evals at Scale: Beyond Accuracy and Similarity
Reliable evaluation starts with application-specific data and extends to code behavior, agent trajectories, tool calls, and a repeatable process for balancing speed with human judgment.
From a talk by Muktesh Mishra
Before you start: Familiarity with LLM applications, retrieval-augmented generation (RAG), and basic software testing will help you follow the examples.
Testing an application whose answers change
How do you test an AI application when the same input can produce different answers—and deciding whether an answer is good requires judgment? Muktesh Mishra approaches this problem from his work leading applied AI engineering for developer platforms at Adobe, alongside coauthoring CI/CD Design Patterns and contributing to open source. After invoking a Twitter snapshot as a sign of growing interest in evaluations, he turns to the practical difficulty shared by RAG applications, chatbots, and agents: ordinary expectations of repeatable output no longer suffice.
A prompt edit makes the problem immediate: “If I am changing a prompt, what is gonna break?” You need a way to detect regressions, not just inspect whether the new answer looks plausible. As model capabilities evolve, that same evaluation process must help you choose tools, metrics, and models suited to the application.
Evals are test cases for measuring application behavior. Their purpose extends beyond assigning an accuracy score: they connect system outputs to business consequences and application goals, establish whether successive changes improve the product, and support the trust and accountability customers need. Without that measurement, an application can change without anyone knowing whether it has become better.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start small, then refine the data
The first practical step is to assemble evaluation data. Start with a small synthetic dataset: artificial inputs against which you can validate application outputs. That gives you an initial testing surface before you have accumulated a mature collection of examples.
The dataset should then change as you observe the system. Generated outputs reveal where the existing examples are insufficient; those observations feed back into refinement. Label cases by the aspects of behavior and application flows they exercise, so the collection expresses what it covers rather than becoming an undifferentiated pile of prompts.
For example, a small support application could separate factual lookup from a clarification flow using a TypeScript data structure like this:
typescript
type EvalCase = {
id: string;
flow: "policy-lookup" | "clarification";
input: string;
expectedBehavior: string;
};
const cases: EvalCase[] = [
{
id: "return-policy",
flow: "policy-lookup",
input: "Where can I find the return policy?",
expectedBehavior: "Point to the supplied return-policy source.",
},
{
id: "unspecified-change",
flow: "clarification",
input: "Can I change it?",
expectedBehavior: "Ask what the customer wants to change.",
},
];
function casesForFlow(flow: EvalCase["flow"]): EvalCase[] {
return cases.filter((testCase) => testCase.flow === flow);
}
The labels make it possible to select cases by flow; expectedBehavior records what a later evaluator must assess. Mishra’s experience is that one dataset is never sufficient: different applications, flows, and intended outcomes require different collections of cases.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn “evaluate everything” into explicit coverage
Wanting to evaluate everything is an ambition, not yet a testing strategy. Make that ambition concrete in three steps:
- Define goals and objectives. Specify which system behaviors you want to evaluate.
- Separate components into modules. Organize evaluation around those components and handle the data so different flows receive appropriate datasets.
- Test flows, outputs, and paths. When the application can follow several routes, cover those routes rather than checking only a single final response.
This connects the dataset design to the application’s structure: coverage means exercising the behaviors the system can actually perform.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Match the evaluation to the task
There is no universal evaluation. A typical RAG question-answering application and a code generator produce different kinds of artifacts, so they need different evidence of success.
| Application | Evaluation focus |
|---|---|
| RAG question answering | Accuracy, similarity, usefulness |
| Code generation | Functional correctness, robustness |
For question answering, the listed criteria examine qualities of the response. For code generation, Mishra calls for testing the generated code against the actual codebase. A plausible-looking implementation is insufficient: evaluation needs to establish whether it functions correctly and behaves robustly in that setting.
Agents add another dimension: the path taken through the task. Trajectory evaluation examines which route an agent follows while executing a flow, because agents can take different paths. Multi-turn simulation then extends the evaluation to conversations, where behavior unfolds across exchanges rather than within one answer. Tool use adds checks for call correctness, associated test suites, and the data being generated. These are separate evaluation targets that a final-answer score alone does not describe.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Scale the execution and the feedback loop
Once the evaluation targets are clear, scaling becomes an execution problem. Mishra points to caching intermediate results, regression evaluation, and deliberate orchestration and parallelism. These are the operational concerns behind running a collection of evaluations repeatedly: deciding how work is coordinated, what can run concurrently, and which intermediate results can be reused.
Execution must lead to analysis. Aggregate the results, run evaluations frequently, and use the findings to improve the application. Mishra describes the cycle as “measure, monitor, analyze and repeat.” The metrics and methodologies still depend on the use case; running more evaluations does not remove the need to decide what is worth measuring.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make evaluation a development process
There is also no fixed division between human judgment and automation that suits every application.
- Human-in-the-loop evaluation: Give human judgment precedence where the task requires it.
- Automated tests and evals: Use automation where repeatable, frequent assessment serves the goal.
Mishra frames this as a balance between speed and fidelity. The right balance depends on what the application is trying to achieve, rather than on a preference for automating every check.
Establish the process before relying on the tools. Not everything can be automated, so the evaluation strategy must define how assessments happen, including where human judgment belongs. A tool can participate in that process; it cannot substitute for deciding how the application should be evaluated.
This leads to eval-driven development, analogous to test-driven development in conventional software. Define evaluations around the use case, and include negative cases as well as positive ones: the dataset must cover situations where the application should not succeed in the ordinary way, not just examples of desired output. Data remains central as development continues. Measuring, monitoring, analyzing, and iterating become a recurring engineering practice, with the balance between fidelity and speed revisited as the application changes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Further reading
Introduces reference-free evaluation across retrieval relevance, faithful use of retrieved context, and answer quality.
A repository-level evaluation framework that asks models to resolve real GitHub issues by editing existing codebases.
Current Python examples for evaluating agent trajectories with deterministic matching, tool-argument checks, and LLM judges.
Read the complete timestamped transcript
- 0:00
[on hold music] Hey, everyone.
- 0:16
Um, hope you are, uh, having a great conference. Um, so I'm gonna talk about, uh, how to run evals at scale and thinking beyond accuracy or similarity. Uh, so in the last, uh, presentation, we, we learned about like how to ar- uh, architect the AI applications, um, and then why evals are important.
- 0:39
In this presentation, I'm gonna talk about like the importance of evals as well as what type of evals we have to choose when we are crafting an application.
- 0:49
This is a bit about me. Um, so I work as a lead engineer for applied AI for developer platforms at Adobe. Um, I have also co-authored, um, uh, CI/CD Design Patterns book, um, and also involved in a lot of open source work, uh, across the communities.
- 1:07
So let's get started. So how many of you have seen this or are active on the Twitter right now? Like, have you seen these kind of patterns emerging? Um, I think this morning there was a talk where this snapshot was again surfaced.
- 1:23
So the one of the most important trends in AI application development is evals, because without evals, we can't, uh, we can't craft any AI application.
- 1:35
Then, um, how many of you are, uh, developing an AI application, be it a RAG, chatbot, agents, anything? So if you are working on that, you often have the, uh-- come across these kind of questions.
- 1:47
Like, how do I te-test applications when outputs are non-deterministic and re-require subjective judgment? Because we all know in LLM world, uh, you can have the different output for the same set of input.
- 2:00
LLMs are non-deterministic. Or how many times you wondering like, "If I am changing a prompt, what is gonna break?" Or, "How am I gonna test that?" And then most importantly, when you are developing an application, in order to, uh, measure the performance or accuracy, you need to find out what tools to use, what metrics to me-- uh,
- 2:19
use, or what models are best, because models are getting capable day by day.
- 2:25
And the answer is evals. So evals is the fundamental approach where you are, uh, writing sort of test cases to measure your AI applications. And why do we-- why do they matter?
- 2:38
Uh, because without measuring something, uh, it can have various impacts. It can, uh, impact your business. Uh, you need to measure whatever s- system output is being produced. How do you align your application with system goals?
- 2:53
Or one of the important aspect is how do you keep getting better? Because applications are-- you are developing applications day by day, and you need to make sure it is getting better.
- 3:02
And then trust and accountability, this is one of the aspects, um, uh, which is very important because whenever you are developing something for a customer, uh, you need to make sure, uh, they trust your application, whatever output is being generated.
- 3:18
Now, when we talk about evals, one of the important aspect to focus on is data. So when we think about evals, when we think about the tests, how do we start?
- 3:29
So the very first step is starting with the data. Now, how do you get the data? So there are a couple of approaches to get the data. One is you start small, and you start with the synthetic data.
- 3:41
Synthetic data means you can generate the, um, you can generate the ancillary data, you can generate the artificial data, and start validating, uh, your a-applications output against that data.
- 3:53
Then it's eval-- Uh, when you think about the data in evals, uh, it's a continuous improvement process. It means every time you generate some output, you need to observe the system, and then you need to keep on refining that data set, whatever data set you are procuring.
- 4:10
And another aspect is you need to label your data accordingly. So because data is fundamental to writing evals, so you need to-- When generating the data, you need to define your data set in, in a way where it is labeled into different aspects.
- 4:24
It is covering multiple flows or application prospects, so things like that. And then you need to continuously refine that. Another, another, uh, approach which I have learned from my experience is you-- one data set is never sufficient.
- 4:39
So when you are thinking about eval, you need to think about multiple data sets based on the flows, based on the applications and, uh, whatever you are trying to achieve.
- 4:50
Now, when we think about evaluation, uh, what do we think about? So what do we want to evaluate? The answer is everything, but what does that mean? So you need to defi-- start by defining your goals and objectives, and what do you want to evaluate in your system.
- 5:05
Then you need to design in a way where you have module-- modules defined for each of the components. You need to optimize your data handling. Um, and I, I noticed I'm mentioning data again and again, but the point is you need to have different data sets for different flows.
- 5:22
Uh, you need to test your flows, outputs, uh, and paths. So if your application involves multiple flows, multiple path, you need to evaluate i-in, in all, all paths.
- 5:33
Now, adaptive evals. So one of the, uh, previous presentation talked about, like, there is no universal eval. And that's-- that is again, most important thing because your evals depend upon what you wa-- what type of application you want to evaluate.
- 5:47
For example, evaluating a RAG application, a typical RAG application is different from code generation.
- 5:53
Uh, if you are dealing with a RAG, typical Q&A type of ac-- uh, application, you can define your eval such as accuracy or similarity or usefulness. Versus when you are generating a code, uh, you want to gen, uh, test the generated code against the actual code base.
- 6:09
So that is where you need to defi-- uh, measure your functional correctness of the code generated or how robust that code is generated.
- 6:18
Then, uh, when you are trying to evaluate, uh, um, agents. So one of the important aspect for evaluating agents is trajectory evaluation, because agents, uh, can take a different path, and oftentimes you need to define which path they are taking in order to execute a flow.
- 6:35
There is also multi-turn simul-- uh, simulation where most of these agents are complex and you, you need to check, like, w-when you are having a conversation, like, how do you evaluate that?
- 6:47
Then if you are doing the tool call, then you also need to check the correctness or test suite or, like, how they are-- how the data is being generated.
- 6:57
Now, another aspect is how do you scale eval? So one, one strategy is you can cache the intermediate results and regression. You need to focus on orchestration and parallelism, like how you are we-- how you are ge-- uh, running your evals, how you are orchestrating them, how you are parallelizing them.
- 7:15
You need to aggregate the results, and then you need-- The important aspect here is you need to run them frequently and then improve upon. So one of the, uh, term which is being u-used in industry is measure, monitor, analyze and repeat.
- 7:28
So you need to often measure it, you need to, uh, analyze that and iterate on that. Then you need to strategize what you want to measure. So again, depending upon the use case, there are different type of matrices or different type of methodologies you need to adapt to.
- 7:44
And then again, use-- There is no fixed strategy to run your eval, so use wha-what fits best. In some cases, uh, you want your humans in the loop to be taking precedence.
- 7:57
In some cases, you have, um, uh, automation tests, automation evals running in. There is a fine balance or trade-off between, uh, human-in-the-loop versus automation, like whether you want the high speed versus high fidelity.
- 8:12
So again, depending upon what you want to achieve, um, you want to take-- give a fine balance on that. And rely on process over tools. Reason is because tools, again, you cannot automate everything.
- 8:25
So you need to define and establish the process, how do you want to run the evals?
- 8:31
So these are some of the key takeaways we just talked about. Um, so one is evals are the most important aspect for AI application. Uh, there is a term being coined now, eval-driven development, uh, which is if, if you think about typical software like test-driven development, this is the eval-driven development.
- 8:50
Define evals based on the use cases. Uh, you need to focus on positive as well as negative cases. Then focus on the data. That is-- I cannot emphasize enough on, uh, on that.
- 9:01
And then remember to measure, monitor, analyze, iterate in a loop continuously. And always take a balanced approach in fidelity versus speed. Uh, if you have any questions, uh, there's a barcode.
- 9:13
You can come later and chat with me. Uh, happy to chat more, and that's all from now. [outro music]