# How to Run Evals at Scale: Thinking Beyond Accuracy or Similarity

Muktesh Mishra, Adobe

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/coKKKKh8Vns-run-evals-at-scale-thinking-beyond-accuracy) · [Video](https://www.youtube.com/watch?v=coKKKKh8Vns)

## How to Run Evals at Scale: Beyond Accuracy and Similarity

Reliable evaluation starts with application-specific data and extends to code behavior, agent trajectories, tool calls, and a repeatable process for balancing speed with human judgment.

Familiarity with LLM applications, retrieval-augmented generation (RAG), and basic software testing will help you follow the examples.

### Testing an application whose answers change

[0:39](https://www.youtube.com/watch?v=coKKKKh8Vns&t=39s)

How do you test an AI application when the same input can produce different answers—and deciding whether an answer is good requires judgment? Muktesh Mishra approaches this problem from his work leading applied AI engineering for developer platforms at Adobe, alongside coauthoring *CI/CD Design Patterns* and contributing to open source. After invoking a Twitter snapshot as a sign of growing interest in evaluations, he turns to the practical difficulty shared by RAG applications, chatbots, and agents: ordinary expectations of repeatable output no longer suffice.

A prompt edit makes the problem immediate: “If I am changing a prompt, what is gonna break?” You need a way to detect regressions, not just inspect whether the new answer looks plausible. As model capabilities evolve, that same evaluation process must help you choose tools, metrics, and models suited to the application.

![Slide titled “Have you seen these questions before?” with three colored boxes describing testing challenges, prompt changes, and evaluation choices.](https://ai.engineer/api/talk-essay-media/8d2904646748b96e45f64f406469019b5e9d35898e75956938502d61d8f47fdd)

Three evaluation questions: testing nondeterministic outputs, checking prompt changes, and choosing metrics, tools, and models.

**Evals are test cases for measuring application behavior.** Their purpose extends beyond assigning an accuracy score: they connect system outputs to business consequences and application goals, establish whether successive changes improve the product, and support the trust and accountability customers need. Without that measurement, an application can change without anyone knowing whether it has become better.

### Start small, then refine the data

[3:18](https://www.youtube.com/watch?v=coKKKKh8Vns&t=198s)

The first practical step is to assemble evaluation data. Start with a small synthetic dataset: artificial inputs against which you can validate application outputs. That gives you an initial testing surface before you have accumulated a mature collection of examples.

The dataset should then change as you observe the system. Generated outputs reveal where the existing examples are insufficient; those observations feed back into refinement. Label cases by the aspects of behavior and application flows they exercise, so the collection expresses what it covers rather than becoming an undifferentiated pile of prompts.

For example, a small support application could separate factual lookup from a clarification flow using a TypeScript data structure like this:

```typescript
type EvalCase = {
  id: string;
  flow: "policy-lookup" | "clarification";
  input: string;
  expectedBehavior: string;
};

const cases: EvalCase[] = [
  {
    id: "return-policy",
    flow: "policy-lookup",
    input: "Where can I find the return policy?",
    expectedBehavior: "Point to the supplied return-policy source.",
  },
  {
    id: "unspecified-change",
    flow: "clarification",
    input: "Can I change it?",
    expectedBehavior: "Ask what the customer wants to change.",
  },
];

function casesForFlow(flow: EvalCase["flow"]): EvalCase[] {
  return cases.filter((testCase) => testCase.flow === flow);
}
```

The labels make it possible to select cases by flow; `expectedBehavior` records what a later evaluator must assess. Mishra’s experience is that one dataset is never sufficient: different applications, flows, and intended outcomes require different collections of cases.

### Turn “evaluate everything” into explicit coverage

[4:50](https://www.youtube.com/watch?v=coKKKKh8Vns&t=290s)

Wanting to evaluate everything is an ambition, not yet a testing strategy. Make that ambition concrete in three steps:

1. **Define goals and objectives.** Specify which system behaviors you want to evaluate.
2. **Separate components into modules.** Organize evaluation around those components and handle the data so different flows receive appropriate datasets.
3. **Test flows, outputs, and paths.** When the application can follow several routes, cover those routes rather than checking only a single final response.

This connects the dataset design to the application’s structure: coverage means exercising the behaviors the system can actually perform.

### Match the evaluation to the task

[5:33](https://www.youtube.com/watch?v=coKKKKh8Vns&t=333s)

**There is no universal evaluation.** A typical RAG question-answering application and a code generator produce different kinds of artifacts, so they need different evidence of success.

| Application | Evaluation focus |
| --- | --- |
| RAG question answering | Accuracy, similarity, usefulness |
| Code generation | Functional correctness, robustness |

For question answering, the listed criteria examine qualities of the response. For code generation, Mishra calls for testing the generated code against the actual codebase. A plausible-looking implementation is insufficient: evaluation needs to establish whether it functions correctly and behaves robustly in that setting.

Agents add another dimension: **the path taken through the task**. Trajectory evaluation examines which route an agent follows while executing a flow, because agents can take different paths. Multi-turn simulation then extends the evaluation to conversations, where behavior unfolds across exchanges rather than within one answer. Tool use adds checks for call correctness, associated test suites, and the data being generated. These are separate evaluation targets that a final-answer score alone does not describe.

![Adaptive Evals slide with four colored rows: RAG accuracy, similarity, usefulness and conciseness; code correctness, robustness, efficiency, quality and HITL; agent trajectories and multiturn simulation; tool-call correctness, test suites and Pass@K.](https://ai.engineer/api/talk-essay-media/2e555b2dff0585e0b49570a82815eb3c84ec15c6b22ec503d9f205e8b52803ff)

Adaptive Evals lists different evaluation criteria for RAG, code generation, agents, and tool calls.

### Scale the execution and the feedback loop

[6:57](https://www.youtube.com/watch?v=coKKKKh8Vns&t=417s)

Once the evaluation targets are clear, scaling becomes an execution problem. Mishra points to caching intermediate results, regression evaluation, and deliberate orchestration and parallelism. These are the operational concerns behind running a collection of evaluations repeatedly: deciding how work is coordinated, what can run concurrently, and which intermediate results can be reused.

Execution must lead to analysis. Aggregate the results, run evaluations frequently, and use the findings to improve the application. Mishra describes the cycle as “measure, monitor, analyze and repeat.” The metrics and methodologies still depend on the use case; running more evaluations does not remove the need to decide what is worth measuring.

### Make evaluation a development process

[7:44](https://www.youtube.com/watch?v=coKKKKh8Vns&t=464s)

There is also no fixed division between human judgment and automation that suits every application.

- **Human-in-the-loop evaluation:** Give human judgment precedence where the task requires it.
- **Automated tests and evals:** Use automation where repeatable, frequent assessment serves the goal.

Mishra frames this as a balance between speed and fidelity. The right balance depends on what the application is trying to achieve, rather than on a preference for automating every check.

**Establish the process before relying on the tools.** Not everything can be automated, so the evaluation strategy must define how assessments happen, including where human judgment belongs. A tool can participate in that process; it cannot substitute for deciding how the application should be evaluated.

This leads to **eval-driven development**, analogous to test-driven development in conventional software. Define evaluations around the use case, and include negative cases as well as positive ones: the dataset must cover situations where the application should not succeed in the ordinary way, not just examples of desired output. Data remains central as development continues. Measuring, monitoring, analyzing, and iterating become a recurring engineering practice, with the balance between fidelity and speed revisited as the application changes.

## Resources

- [Ragas: Automated Evaluation of Retrieval Augmented Generation](https://arxiv.org/abs/2309.15217) — Introduces reference-free evaluation across retrieval relevance, faithful use of retrieved context, and answer quality.
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://arxiv.org/abs/2310.06770) — A repository-level evaluation framework that asks models to resolve real GitHub issues by editing existing codebases.
- [Agent Evals](https://docs.langchain.com/oss/python/langchain/test/evals) — Current Python examples for evaluating agent trajectories with deterministic matching, tool-argument checks, and LLM judges.

## Slides

### How to run evals at scale

The talk explores why evals matter and how to choose evaluation types when building an application, looking beyond accuracy and similarity.

[0:21](https://www.youtube.com/watch?v=coKKKKh8Vns&t=21s) [Title slide about running evaluations at scale and thinking beyond accuracy or similarity.](https://assets.ai.engineer/sites/aiecode2025/sha256/20278b24e40a6628537fb6ff620b2b4fa2a6d25e7103dce431c1c24364209d63.webp)

How to run evals at scale? Thinking beyond accuracy or similarity

AIE

AI Engineer
World’s Fair
June 3-5 2025 • San Francisco

### Who am I

The speaker is a lead engineer working on applied AI for developer platforms at Adobe, a co-author of CICD Design Patterns, and a contributor to open-source and developer communities.

[0:48](https://www.youtube.com/watch?v=coKKKKh8Vns&t=48s) [Speaker introduction listing engineering, authorship, open-source and community roles.](https://assets.ai.engineer/sites/aiecode2025/sha256/dfca41b0c6bfd452efaaa7c2641fccb2f18758c577fe1f26a3ca75f0f2086f6a.webp)

Who am I

Lead Engineer, Applied AI for Developer Platforms at Adobe
Co-Author or Best selling CICD Design Patterns Book
Open-Source Contributor to 20+ Projects
Continuous Delivery Foundation Ambassador
Founder Sunnyvale Java User Group
MLOps Community Ambassador
AISummit Ambassador

Interface text:
AIE
AI Engineer World's Fair
aws

### Have you seen this before?

Posts from Garry Tan, Andrew Ng and Greg Brockman highlight evals in AI development. Tan also shares Anjney Midha’s observation that a product leader uses ruthless evals behind the scenes.

[1:13](https://www.youtube.com/watch?v=coKKKKh8Vns&t=73s) [Public posts discuss the importance of evaluations for AI applications.](https://assets.ai.engineer/sites/aiecode2025/sha256/876af300d9a2df01a49eb57086268e0e3d9a9ff201fddecd04aba4ac5a2a949b.webp)

Have you seen this before?

Garry Tan @garrytan · Feb 21
Evals are emerging as the real moat for AI startups

Hard won insights about customers and their business logic discovered by founders acting almost as ethnographers spelunking in the underserved slices of the GDP pie chart

Anjney Midha @AnjneyMidha · Feb 21
The best AI product leader I know makes it a habit of saying ‘taste’ is his differentiator publicly

But behind the scenes, it’s all ruthless evals
...

Andrew Ng @AndrewYNg · Feb 19
New short course: Evaluating AI Agents! Evals are important for driving AI system improvements, and in this course you'll learn to systematically assess and improve an AI agent’s performance. This is built in partnership with @arizeai and taught by @JohnGilhuly, Head of Developer Relations, and @_amankhan, Director of Product.

I've often found evals to be a critical tool in the agent development process - they can be the difference between picking the right thing to work on vs. wasting weeks of effort. Whether you’re building a shopping assistant, coding agent, or research assistant, having a structured evaluation process helps you refine its performance systematically, rather than relying on random trial and error.

Greg Brockman @gdb · Dec 9, 2023
[illegible] are surprisingly often all you need

Interface text:
Garry Tan post: 77, 128, 1.3K, 319K
Greg Brockman post: 133, 1.2K, 351K
AIE
AI Engineer World's Fair
Microsoft
smol ai

### Have you seen these questions before?

AI application developers face variable outputs, subjective judgments, and prompt regressions. The speaker presents evals as test cases for measuring applications and guiding choices of metrics, tools, and models.

[1:33](https://www.youtube.com/watch?v=coKKKKh8Vns&t=93s) [Questions about testing nondeterministic outputs, prompt regressions, and choosing metrics, tools, and models.](https://assets.ai.engineer/sites/aiecode2025/sha256/9faffedec06048e161b109dbc32ab052bc988fb22f469917c07be2c597d6634a.webp)

Have you seen these questions before?

How do I test applications when the outputs are non-deterministic and require subjective judgements?

If I change the prompt, how do I know I'm not breaking something else?

What metrics should I track? What tools should I use? Which models are best?

Interface text:
AIE
Microsoft
smol ai
AI Engineer World's Fair

### Data is your friend

Start with synthetic or manually curated data, then refine and label it as outputs arrive. Use multiple datasets to cover different flows; no single dataset covers everything.

[3:16](https://www.youtube.com/watch?v=coKKKKh8Vns&t=196s) [A workflow recommends starting with data, refining and labeling it, iterating continuously, and using multiple datasets.](https://assets.ai.engineer/sites/aiecode2025/sha256/a9a5cc7beeb513c50e17c615383e4705d388862a8e12a4c7b6440569cdeecb49.webp)

Data is your friend

Start from somewhere (synthetic data, manually curated)
Keep refining and add a pipeline
Label data for different cases, for success and failure
Continuous process flow and iteration (real world is not stopping)
Multiple data sets: No data is perfect or covers everything

Interface text:
AIE
AI Engineer
World's Fair

### Evaluate Everything

Define goals first, then evaluate components using different datasets for different flows. Test every path and its outputs.

[4:48](https://www.youtube.com/watch?v=coKKKKh8Vns&t=288s) [Evaluation should cover goals, modular design, data handling, flows, paths, and outputs.](https://assets.ai.engineer/sites/aiecode2025/sha256/53d18fe6470e7eae7e1e3208aa3b77ec525ab355365ec950221da0b8a0184b7f.webp)

Evaluate everything

Define goals and objectives
Modular design
Optimize data handling
Flows
Paths
Outputs

Event branding:
AIE
AI Engineer
World’s Fair

### Adaptive Evals

Evaluation criteria depend on the application: RAG emphasizes answer quality, code generation needs functional checks, and agents require attention to the paths they take.

[5:31](https://www.youtube.com/watch?v=coKKKKh8Vns&t=331s) [Evaluation criteria for RAG, code generation, agents, and tool calls.](https://assets.ai.engineer/sites/aiecode2025/sha256/978278d8bca40547998b3cc534f857384ce3f8a7da112140503c4d0d97f1d002.webp)

Adaptive Evals

RAG: Accuracy, Similarity, Usefulness, Conciseness

Code generation: Functional Correctness, Robustness (Adversarial Code Evals), Efficiency, Code Quality, HITL

Agents: Trajectory, Multiturn Simulation

Tool calls: Correctness, Test Suites, Pass@K

AIE
AI Engineer
World’s Fair

### Scaling Evals

Scale evals with caching, parallel runs and frequent iteration. Choose human review, automation or a combination based on the trade-off between fidelity and speed, and establish a process for running evals.

[6:54](https://www.youtube.com/watch?v=coKKKKh8Vns&t=414s) [Strategies include caching, parallelism, aggregating results, frequent experiments, standardized metrics, and choosing human or automated evaluation processes.](https://assets.ai.engineer/sites/aiecode2025/sha256/e637e9a88a3509d222a536e751c695af819f83d7323d4074e7f4171ff5e32597.webp)

Scaling Evals

Cache intermediate results, regressions
Orchestration and parallelism
Aggregate Results
Run frequent experimentations and improve
Standardize Metrics and iterate often
Use what fits best (one or combination of human in the loop, Automated pipeline with refinement)
Rely on process over tools

Interface text:
AIE
AI Engineer
World’s Fair

### Takeaways

Define evals around each use case, cover positive and negative cases, and improve the data. Measure, monitor, analyze, and iterate continuously while balancing fidelity with speed.

[8:29](https://www.youtube.com/watch?v=coKKKKh8Vns&t=509s) [Takeaways on defining evaluations, testing positive and negative cases, improving data, iterating on measurements, and balancing fidelity with speed.](https://assets.ai.engineer/sites/aiecode2025/sha256/9014148ccac8f09f4eadecce886f4b18cb67eb89effc8ac2499f852baeff9c8b.webp)

Takeaways

Evals are the most important aspect for AI Applications (Eval driven development is real)
Define Evals based on the use cases
Focus on Positive and Negative cases
Focus on data (synthetic and continuously improved)
Remember to measure, monitor, analyze and iterate in a loop
Always take a balance approach Fidelity v/s Speed

Questions? Let’s chat:

Interface text:
AIE
AI Engineer
World’s Fair
Microsoft
smol ai

## Full transcript

[0:00](https://www.youtube.com/watch?v=coKKKKh8Vns&t=0s) [on hold music] Hey, everyone.

[0:16](https://www.youtube.com/watch?v=coKKKKh8Vns&t=16s) Um, hope you are, uh, having a great conference. Um, so I'm gonna talk about, uh, how to run evals at scale and thinking beyond accuracy or similarity. Uh, so in the last, uh, presentation, we, we learned about like how to ar- uh, architect the AI applications, um, and then why evals are important.

[0:39](https://www.youtube.com/watch?v=coKKKKh8Vns&t=39s) In this presentation, I'm gonna talk about like the importance of evals as well as what type of evals we have to choose when we are crafting an application.

[0:49](https://www.youtube.com/watch?v=coKKKKh8Vns&t=49s) This is a bit about me. Um, so I work as a lead engineer for applied AI for developer platforms at Adobe. Um, I have also co-authored, um, uh, CI/CD Design Patterns book, um, and also involved in a lot of open source work, uh, across the communities.

[1:07](https://www.youtube.com/watch?v=coKKKKh8Vns&t=67s) So let's get started. So how many of you have seen this or are active on the Twitter right now? Like, have you seen these kind of patterns emerging? Um, I think this morning there was a talk where this snapshot was again surfaced.

[1:23](https://www.youtube.com/watch?v=coKKKKh8Vns&t=83s) So the one of the most important trends in AI application development is evals, because without evals, we can't, uh, we can't craft any AI application.

[1:35](https://www.youtube.com/watch?v=coKKKKh8Vns&t=95s) Then, um, how many of you are, uh, developing an AI application, be it a RAG, chatbot, agents, anything? So if you are working on that, you often have the, uh-- come across these kind of questions.

[1:47](https://www.youtube.com/watch?v=coKKKKh8Vns&t=107s) Like, how do I te-test applications when outputs are non-deterministic and re-require subjective judgment? Because we all know in LLM world, uh, you can have the different output for the same set of input.

[2:00](https://www.youtube.com/watch?v=coKKKKh8Vns&t=120s) LLMs are non-deterministic. Or how many times you wondering like, "If I am changing a prompt, what is gonna break?" Or, "How am I gonna test that?" And then most importantly, when you are developing an application, in order to, uh, measure the performance or accuracy, you need to find out what tools to use, what metrics to me-- uh,

[2:19](https://www.youtube.com/watch?v=coKKKKh8Vns&t=139s) use, or what models are best, because models are getting capable day by day.

[2:25](https://www.youtube.com/watch?v=coKKKKh8Vns&t=145s) And the answer is evals. So evals is the fundamental approach where you are, uh, writing sort of test cases to measure your AI applications. And why do we-- why do they matter?

[2:38](https://www.youtube.com/watch?v=coKKKKh8Vns&t=158s) Uh, because without measuring something, uh, it can have various impacts. It can, uh, impact your business. Uh, you need to measure whatever s- system output is being produced. How do you align your application with system goals?

[2:53](https://www.youtube.com/watch?v=coKKKKh8Vns&t=173s) Or one of the important aspect is how do you keep getting better? Because applications are-- you are developing applications day by day, and you need to make sure it is getting better.

[3:02](https://www.youtube.com/watch?v=coKKKKh8Vns&t=182s) And then trust and accountability, this is one of the aspects, um, uh, which is very important because whenever you are developing something for a customer, uh, you need to make sure, uh, they trust your application, whatever output is being generated.

[3:18](https://www.youtube.com/watch?v=coKKKKh8Vns&t=198s) Now, when we talk about evals, one of the important aspect to focus on is data. So when we think about evals, when we think about the tests, how do we start?

[3:29](https://www.youtube.com/watch?v=coKKKKh8Vns&t=209s) So the very first step is starting with the data. Now, how do you get the data? So there are a couple of approaches to get the data. One is you start small, and you start with the synthetic data.

[3:41](https://www.youtube.com/watch?v=coKKKKh8Vns&t=221s) Synthetic data means you can generate the, um, you can generate the ancillary data, you can generate the artificial data, and start validating, uh, your a-applications output against that data.

[3:53](https://www.youtube.com/watch?v=coKKKKh8Vns&t=233s) Then it's eval-- Uh, when you think about the data in evals, uh, it's a continuous improvement process. It means every time you generate some output, you need to observe the system, and then you need to keep on refining that data set, whatever data set you are procuring.

[4:10](https://www.youtube.com/watch?v=coKKKKh8Vns&t=250s) And another aspect is you need to label your data accordingly. So because data is fundamental to writing evals, so you need to-- When generating the data, you need to define your data set in, in a way where it is labeled into different aspects.

[4:24](https://www.youtube.com/watch?v=coKKKKh8Vns&t=264s) It is covering multiple flows or application prospects, so things like that. And then you need to continuously refine that. Another, another, uh, approach which I have learned from my experience is you-- one data set is never sufficient.

[4:39](https://www.youtube.com/watch?v=coKKKKh8Vns&t=279s) So when you are thinking about eval, you need to think about multiple data sets based on the flows, based on the applications and, uh, whatever you are trying to achieve.

[4:50](https://www.youtube.com/watch?v=coKKKKh8Vns&t=290s) Now, when we think about evaluation, uh, what do we think about? So what do we want to evaluate? The answer is everything, but what does that mean? So you need to defi-- start by defining your goals and objectives, and what do you want to evaluate in your system.

[5:05](https://www.youtube.com/watch?v=coKKKKh8Vns&t=305s) Then you need to design in a way where you have module-- modules defined for each of the components. You need to optimize your data handling. Um, and I, I noticed I'm mentioning data again and again, but the point is you need to have different data sets for different flows.

[5:22](https://www.youtube.com/watch?v=coKKKKh8Vns&t=322s) Uh, you need to test your flows, outputs, uh, and paths. So if your application involves multiple flows, multiple path, you need to evaluate i-in, in all, all paths.

[5:33](https://www.youtube.com/watch?v=coKKKKh8Vns&t=333s) Now, adaptive evals. So one of the, uh, previous presentation talked about, like, there is no universal eval. And that's-- that is again, most important thing because your evals depend upon what you wa-- what type of application you want to evaluate.

[5:47](https://www.youtube.com/watch?v=coKKKKh8Vns&t=347s) For example, evaluating a RAG application, a typical RAG application is different from code generation.

[5:53](https://www.youtube.com/watch?v=coKKKKh8Vns&t=353s) Uh, if you are dealing with a RAG, typical Q&A type of ac-- uh, application, you can define your eval such as accuracy or similarity or usefulness. Versus when you are generating a code, uh, you want to gen, uh, test the generated code against the actual code base.

[6:09](https://www.youtube.com/watch?v=coKKKKh8Vns&t=369s) So that is where you need to defi-- uh, measure your functional correctness of the code generated or how robust that code is generated.

[6:18](https://www.youtube.com/watch?v=coKKKKh8Vns&t=378s) Then, uh, when you are trying to evaluate, uh, um, agents. So one of the important aspect for evaluating agents is trajectory evaluation, because agents, uh, can take a different path, and oftentimes you need to define which path they are taking in order to execute a flow.

[6:35](https://www.youtube.com/watch?v=coKKKKh8Vns&t=395s) There is also multi-turn simul-- uh, simulation where most of these agents are complex and you, you need to check, like, w-when you are having a conversation, like, how do you evaluate that?

[6:47](https://www.youtube.com/watch?v=coKKKKh8Vns&t=407s) Then if you are doing the tool call, then you also need to check the correctness or test suite or, like, how they are-- how the data is being generated.

[6:57](https://www.youtube.com/watch?v=coKKKKh8Vns&t=417s) Now, another aspect is how do you scale eval? So one, one strategy is you can cache the intermediate results and regression. You need to focus on orchestration and parallelism, like how you are we-- how you are ge-- uh, running your evals, how you are orchestrating them, how you are parallelizing them.

[7:15](https://www.youtube.com/watch?v=coKKKKh8Vns&t=435s) You need to aggregate the results, and then you need-- The important aspect here is you need to run them frequently and then improve upon. So one of the, uh, term which is being u-used in industry is measure, monitor, analyze and repeat.

[7:28](https://www.youtube.com/watch?v=coKKKKh8Vns&t=448s) So you need to often measure it, you need to, uh, analyze that and iterate on that. Then you need to strategize what you want to measure. So again, depending upon the use case, there are different type of metrics or different type of methodologies you need to adapt to.

[7:44](https://www.youtube.com/watch?v=coKKKKh8Vns&t=464s) And then again, use-- There is no fixed strategy to run your eval, so use wha-what fits best. In some cases, uh, you want your humans in the loop to be taking precedence.

[7:57](https://www.youtube.com/watch?v=coKKKKh8Vns&t=477s) In some cases, you have, um, uh, automation tests, automation evals running in. There is a fine balance or trade-off between, uh, human-in-the-loop versus automation, like whether you want the high speed versus high fidelity.

[8:12](https://www.youtube.com/watch?v=coKKKKh8Vns&t=492s) So again, depending upon what you want to achieve, um, you want to take-- give a fine balance on that. And rely on process over tools. Reason is because tools, again, you cannot automate everything.

[8:25](https://www.youtube.com/watch?v=coKKKKh8Vns&t=505s) So you need to define and establish the process, how do you want to run the evals?

[8:31](https://www.youtube.com/watch?v=coKKKKh8Vns&t=511s) So these are some of the key takeaways we just talked about. Um, so one is evals are the most important aspect for AI application. Uh, there is a term being coined now, eval-driven development, uh, which is if, if you think about typical software like test-driven development, this is the eval-driven development.

[8:50](https://www.youtube.com/watch?v=coKKKKh8Vns&t=530s) Define evals based on the use cases. Uh, you need to focus on positive as well as negative cases. Then focus on the data. That is-- I cannot emphasize enough on, uh, on that.

[9:01](https://www.youtube.com/watch?v=coKKKKh8Vns&t=541s) And then remember to measure, monitor, analyze, iterate in a loop continuously. And always take a balanced approach in fidelity versus speed. Uh, if you have any questions, uh, there's a barcode.

[9:13](https://www.youtube.com/watch?v=coKKKKh8Vns&t=553s) You can come later and chat with me. Uh, happy to chat more, and that's all from now. [outro music]
