# Why should anyone care about Evals?

Manu Goyal, Braintrust

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/jJ45Yz1lJao-why-should-anyone-care-about-evals) · [Video](https://www.youtube.com/watch?v=jJ45Yz1lJao)

## Why should anyone care about Evals?

A higher model score does not establish that an application works. Evals connect model changes to real tasks, support experiments before deployment, and turn production feedback into the next iteration.

### From Nintendo to adaptive software

[0:24](https://www.youtube.com/watch?v=jJ45Yz1lJao&t=24s)

A boy absorbed in a Nintendo 64 game looks like an unlikely starting point for a discussion of AI evaluation. Manu Goyal, an engineer at [Braintrust](https://www.braintrust.dev/blog/reliable-ai), offers a comic reinterpretation of his childhood photograph: here is someone already disappointed by technology’s limitations. Rule-based systems repeat the same behavior; he wants technology that grows, adapts, and becomes a thought partner. The childhood career revelation is a joke, but the ambition sets up a practical question: how do we know whether adaptive software actually works?

### A better classifier is not a road-ready car

[1:29](https://www.youtube.com/watch?v=jJ45Yz1lJao&t=89s)

In Goyal’s accelerated origin story, he drops the Nintendo, starts grinding LeetCode, and lands in the self-driving car industry. That experience supplies the central engineering problem. You can spend all day tuning a model, changing its architecture, or adjusting its loss function without acquiring enough evidence to ship. In his hypothetical example, improving image classification from 98% to 99% does not establish that a car is ready for the road.

![A two-panel cartoon shows buttons labeled “blame the model” and “blame the prompt,” followed by a sweating man struggling to choose.](https://ai.engineer/api/talk-essay-media/0f0980deeecbcad5406742c2c96f8ccfa59c8fae7d03844e5a5ddbed3dee2ae3)

Blame the model or blame the prompt?

The missing evidence concerns the application, not just the model’s component task:

| Question | What it tests |
| --- | --- |
| Does it avoid pedestrians? | Behavior around people |
| Does it negotiate traffic appropriately? | Behavior in driving scenarios |
| Does it obey the law? | Compliance with driving rules |

**Evals put model performance in the context of the real-world application.** An aggregate classification score cannot answer all of these questions. Calling evals unit tests for AI captures only part of their purpose.

### Build a laboratory before going to production

[2:36](https://www.youtube.com/watch?v=jJ45Yz1lJao&t=156s)

Regression detection is useful, but evals also provide a place to experiment. Without them, Goyal argues, getting feedback on a change means shipping it to production and observing what happens. That loop is expensive, slow, and risky: the real world becomes the testing environment.

Investing in good evals creates a laboratory where developers can repeatedly test changes before exposing users to them. Goyal estimates that good evals can move 90% of the product iteration loop before production. This is an advocacy estimate, with no measurement protocol supplied; the practical claim is that earlier feedback makes iteration faster and deployment more confident.

The laboratory should remain connected to production. Apply the same metrics used offline to online production data, and those measurements can identify which real examples will be most useful for the next iteration. **Production feedback helps decide what to evaluate next.** The loop therefore does more than check a release: it helps improve the evidence available for subsequent changes.

### The enthusiasm around evals

[3:32](https://www.youtube.com/watch?v=jJ45Yz1lJao&t=212s)

Goyal closes his personal journey with a before-and-after transformation gag: understanding evals completes the transformation. He then turns to industry enthusiasm, naming Kevin Weil, Gary Tan, Mike Krieger, and Greg Brockman as advocates. His joking inference is that, with everyone endorsing evals, there must be something worth investigating—it cannot all be a scam. The appeal to recognizable names follows the concrete reason to care: a safer, faster way to learn whether changes help an application.

### Connect experimentation, evaluation, and observation

[4:18](https://www.youtube.com/watch?v=jJ45Yz1lJao&t=258s)

At Braintrust, Goyal describes the goal as building a development platform around this loop. Evals sit alongside prompt tweaking and playground experiments, while logging supplies data and observability makes application behavior visible. Connecting those activities creates the data flywheel: experiments produce changes, observation supplies feedback, and that feedback informs further evaluation and experimentation. The accompanying slide groups the workflow into Play, Eval, Log, Review, and Monitor.

![A slide lists Play, Eval, Log, Review, and Monitor beside a mint-green panel with the Braintrust symbol.](https://ai.engineer/api/talk-essay-media/7a58c94083d1265f58b874ef856037ebf683e569cb262da818e5d4d17ebb8c32)

Braintrust’s workflow: Play, Eval, Log, Review, Monitor.

After joking that this has been a dense, content-heavy presentation, Goyal distills his pitch into a broad assertion: evals are the key to industry transformation and success. His concrete invitation is to continue the discussion in the Evals track, in Golden Gate Ballroom B.

## Resources

- [It's time to build reliable AI](https://www.braintrust.dev/blog/reliable-ai) — Braintrust's original introduction connects evaluations, production examples, datasets and prompt experimentation.
- [Braintrust workflow](https://www.braintrust.dev/docs/workflow) — Current guide to tracing applications, reviewing production data, evaluating changes and monitoring deployments.
- [Score production traces](https://www.braintrust.dev/docs/evaluate/score-online) — Configure production scoring to monitor deployed changes and discover examples for future evaluations.
- [Iterate in playgrounds](https://www.braintrust.dev/docs/evaluate/playgrounds) — Experiment with prompts and models across datasets, inspect scores and save evaluation results.

## Slides

### Manu Goyal

Manu Goyal introduces his evals journey with a childhood gaming photo, joking that even then he wanted technology to grow, adapt, and become a thought partner.

[0:51](https://www.youtube.com/watch?v=jJ45Yz1lJao&t=51s) [Manu Goyal, founding engineer at Braintrust, alongside a childhood photograph of a boy holding a game controller.](https://assets.ai.engineer/sites/aiecode2025/sha256/3ef858677e52dffecc65c53ff84b1a5b9a3a48b9118265119df25ade3b0a2ecb.webp)

Manu Goyal

Founding Engineer

Braintrust

Interface text:
AI Engineer
World’s Fair
aws
WHY SHOULD ANYONE CARE ABOUT EVALS?

### Blame the model or the prompt

A meme contrasts blaming the model with blaming the prompt. Goyal says a self-driving car needs real-world evaluation: improving image classification from 98% to 99% alone isn't enough to put it on the road.

[1:57](https://www.youtube.com/watch?v=jJ45Yz1lJao&t=117s) [A two-button meme shows a person choosing between blaming the model and blaming the prompt.](https://assets.ai.engineer/sites/aiecode2025/sha256/dae0ce1ec01c0122456a755a55954c983a370b768d707fe1fee67e94b9a03cfb.webp)

blame the model
blame the prompt

JAKE-CLARK.TUMBLR

Interface text:
AI Engineer World's Fair
braintrust
aws
WHY SHOULD ANYONE CARE ABOUT EVALS?

### Play, eval, log, review, monitor

Braintrust aims to connect prompt experiments, evals, and logged data in a development workflow, with review and monitoring alongside them.

[4:18](https://www.youtube.com/watch?v=jJ45Yz1lJao&t=258s) [A slide lists Play, Eval, Log, Review, and Monitor next to a Braintrust demo invitation.](https://assets.ai.engineer/sites/aiecode2025/sha256/ac525b577d2e903a79be1c6d325336c31b27b6650730c6aa53d3585a3580c6dc.webp)

Play
Eval
Log
Review
Monitor

Book a demo today
braintrust.dev

### Join the eval track

Braintrust’s eval track covers adapting development for the AI era in Golden Gate Ballroom B.

[4:58](https://www.youtube.com/watch?v=jJ45Yz1lJao&t=298s) [Slide invites attendees to the Braintrust eval track in Golden Gate Ballroom B to learn about adapting development for the AI era.](https://assets.ai.engineer/sites/aiecode2025/sha256/2533efcf9b2051f827a4361cba0bd2dc11cbd17facf09fbb125f82ab7b8cea67.webp)

Join our eval track to learn more about adapting your development for the AI era.

Eval track presented by Braintrust
Golden Gate Ballroom B

## Full transcript

[0:00](https://www.youtube.com/watch?v=jJ45Yz1lJao&t=0s) [upbeat music] Woo. [clapping]

[0:17](https://www.youtube.com/watch?v=jJ45Yz1lJao&t=17s) Woohoo. All right. Who's excited about evals?

[0:23](https://www.youtube.com/watch?v=jJ45Yz1lJao&t=23s) Woo.

[0:24](https://www.youtube.com/watch?v=jJ45Yz1lJao&t=24s) Woo. All right, what can I do to get those juices flowing? Uh, I'm Manu, and, uh, I work at Braintrust, where we build a platform to do evals and a whole bunch of other stuff.

[0:36](https://www.youtube.com/watch?v=jJ45Yz1lJao&t=36s) Um, so I was thinking we could just start by, uh, talking a little bit my, about my own personal evals journey. Now, you might see this picture and say, "Ah, what an adorable little boy absorbed in his Nintendo 64 video game."

[0:52](https://www.youtube.com/watch?v=jJ45Yz1lJao&t=52s) But if you look a little closer, you'll see a boy who's deeply disappointed with the state of technology in his society. [laughs] 'Cause this boy, he knows that technology is not meant to be shackled to the constraints of rule-based systems doomed to do the same thing over and over and over.

[1:12](https://www.youtube.com/watch?v=jJ45Yz1lJao&t=72s) No, technology is meant to come alive, to grow and adapt, and really be a thought partner to mankind. So I knew this in this moment, which is why I decided to devote my career to being a software engineer in the AI industry. [laughs]

[1:29](https://www.youtube.com/watch?v=jJ45Yz1lJao&t=89s) And so I dropped the Nintendo, and I started grinding away on LeetCode. [laughs] And soon enough, I landed a job in the self-driving car industry. Now, we can all learn a lot about self-driving cars, but the, the thing I took away was that, you know, you can spend all day tuning the model, changing the architecture, you know, adjusting

[1:52](https://www.youtube.com/watch?v=jJ45Yz1lJao&t=112s) the loss function, all good stuff, but it's never gonna be enough for you to actually ship it to production, right? I can't say, "Oh, my image classification rate went from 98% to 99%, put it on the road."

[2:07](https://www.youtube.com/watch?v=jJ45Yz1lJao&t=127s) Right? We need some way to, you know, contextualize this model and understand if it actually works for our real-world application. You know, does it avoid pedestrians? Does it negotiate traffic scenarios appropriately?

[2:22](https://www.youtube.com/watch?v=jJ45Yz1lJao&t=142s) Does it obey the law? All this stuff we actually need to understand. And how we're gonna do that is with evals. Now, you know, the whole point here is, you know, evals aren't just unit tests for AI.

[2:36](https://www.youtube.com/watch?v=jJ45Yz1lJao&t=156s) They're not just for finding regressions, right? If I didn't have evals, the only way I can get any signal on my changes is by shipping it to prod and then getting signal, you know, in the real world, but that's expensive, it's slow, and ultimately, it's pretty risky.

[2:53](https://www.youtube.com/watch?v=jJ45Yz1lJao&t=173s) So what do evals do is, is kind of like if you invest in good evals, you're kind of building a laboratory that lets you run experiments to your heart's content and do 90% of the product iteration loop before going to prod, and then now you can ship much more quickly, much more confidently.

[3:15](https://www.youtube.com/watch?v=jJ45Yz1lJao&t=195s) Um, now, furthermore, if you actually apply the same metrics from offline to your online production data, you now have data-driven signal about which examples in prod are gonna be most useful for that next iteration loop.

[3:32](https://www.youtube.com/watch?v=jJ45Yz1lJao&t=212s) And so with, with all of this knowledge, I was ... I, my evals journey had completed, and I transformed from this guy to this guy. So, success. [laughs] Now, if this heartfelt childhood story isn't enough to do it for you, you don't have to take my word.

[3:50](https://www.youtube.com/watch?v=jJ45Yz1lJao&t=230s) You can take the words of all of these tech luminaries. We have Kevin Weil, Gary Tan, Mike Krieger, Greg Brockman, all extolling the virtues and the necessities of evals.

[4:05](https://www.youtube.com/watch?v=jJ45Yz1lJao&t=245s) And surely, if they're all saying it, there's gotta be something to it. It can't be a total scam. So there's gotta be some ... There's gotta be something worth checking out here.

[4:18](https://www.youtube.com/watch?v=jJ45Yz1lJao&t=258s) So with all that buzz, I made my way to Braintrust, where our goal is to sort of build the dev platform to, of course, let you do evals, but also do all the things that go along with it.

[4:31](https://www.youtube.com/watch?v=jJ45Yz1lJao&t=271s) So that involves, you know, tweaking prompts and experimenting in the playground. It involves logging data and sort of getting the observability component and kind of connecting all those together in this beautiful data flywheel, so that we can, we can let you build the data flywheel to let your AI dreams come true, 'cause that's really what, what we're

[4:54](https://www.youtube.com/watch?v=jJ45Yz1lJao&t=294s) here for, for. Now, I know this was a dense and content-heavy presentation. [laughs] So I'll try to distill it with one simple message, which is that the key to industry transformation, the key to success is evals.

[5:14](https://www.youtube.com/watch?v=jJ45Yz1lJao&t=314s) Evals. [clapping] Evals. Evals. Evals. Evals. Evals. Evals. Evals. Evals. Evals. Evals. Evals. Evals. Evals.

[5:24](https://www.youtube.com/watch?v=jJ45Yz1lJao&t=324s) Woohoo. [clapping] All right. Thank you. Please join the Evals track, Golden Gate Ballroom B. I'll see you there. [upbeat music]
