AI Engineer World's Fair 2025
Evals Are Not Unit Tests
Read the talk
Evals Are Not Unit Tests
A fruit-counting app shows why successful demos are not enough—and how representative data, shared production logic, simple scoring, and regular evaluation make improvements measurable.
From a talk by Ido Pesok
Before you start: Familiarity with prompts, API calls, and basic software testing is helpful; the code example uses TypeScript.
What would make a fruit-counting app reliable?
How do you know an AI application will work for its users, rather than just during your demo? At the time of this talk, Ido Pesok was an engineer at Vercel working on v0, a platform for generating full-stack web applications. Its recently launched GitHub Sync could push generated code, pull repository changes into a chat, switch branches, and open pull requests. Pesok reported that v0 had crossed 100 million messages. That product context motivates application-level evaluation: model-release benchmarks describe capabilities, but your application still needs evidence about its own users, inputs, and behavior.
Consider Pesok’s illustrative app, Fruit Letter Counter. The premise plays on the familiar question about how many Rs appear in strawberry. ChatGPT supplies a logo, v0 builds the UI and backend, and the generated backend uses the AI SDK’s streamText call. In the story, GPT-4.1 answers three correctly in two consecutive trials. The slide makes the temptation clear: it worked, then worked again, so perhaps it is ready to ship.
The app launches on Vercel with Fluid Compute enabled. Then John reports that the same strawberry question returned two. The earlier successes did not establish dependable behavior. A polished demo can conceal failures that become visible only after deployment, and the problem applies well beyond letter counting: users need the application’s central function to work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A better prompt still needs representative tests
The next move is prompt engineering: perhaps additional instructions or chain of thought will make counting more reliable. Pesok’s story escalates to an elaborate fruit-loving persona prompt. The revised prompt succeeds in ten consecutive ChatGPT trials. After another deployment, however, John asks for the Rs across strawberry, banana, pineapple, mango, kiwi, dragon fruit, apple, and raspberry. The app answers five and fails again. These trial counts are illustrative outcomes, not a controlled benchmark of the model.
The second failure exposes a different weakness in the testing approach. Repeating one familiar question does not cover the range of questions users will invent. A production user can expand the input in an entirely reasonable way and reach behavior the developer never tested.
Conventional tests remain useful. Individual functions can have unit tests; authentication, login, and sign-out can have end-to-end tests. Pesok illustrates the gap as 95% of the application working all the time while the crucial remaining 5% can fail. Those percentages are a rhetorical split, not measured reliability. Correct surrounding software does not establish the quality of the model behavior at its center.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Map the application onto a basketball court
A basketball court gives this uncertainty a useful shape. The basket is a glowing golden circle. Blue dots represent made shots; red dots represent misses. Distance from the basket stands for difficulty. The court also has boundaries: even a successful shot from outside the court does not count toward the game you are playing. For an application, those boundaries define the work it is supposed to do.
Place the strawberry question near the basket and color it blue after the prompt improvement. Place the longer fruit list farther away and color it red. These points become the evaluation dataset: collect the questions users ask, retain them over time, and build a picture of where the application succeeds and fails.
Now consider counting Rs in strawberry, pineapple, dragon fruit, and mango after replacing every vowel with R. That is difficult, but still within this app’s domain. Counting syllables in carrot is outside the boundary Pesok chooses for Fruit Letter Counter. Difficulty and relevance are separate dimensions: a hard case can deserve attention, while an easy success can be irrelevant.
The metaphor maps directly onto the components of an eval. Braintrust calls the system execution a task.
| Eval component | Basketball equivalent | Application meaning |
|---|---|---|
| Data | Shot location | The input being tested |
| Task | Taking the shot | Running the system on that input |
| Score | Make or miss | Checking the resulting output |
Before optimizing the shot, understand the court: what belongs in the application’s domain, which cases are difficult, and what success means.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build coverage from actual usage
Two mistakes can make a growing eval suite misleading. One is collecting out-of-bounds cases that users do not care about. The other is concentrating points in a small part of the court. A large collection of similar, easy questions may reveal little about the application’s broader behavior. Cover the relevant domain, including its difficult regions.
Production evidence helps locate those regions:
- Thumbs-up and thumbs-down feedback: Noisy, but useful for finding where users struggle.
- Random log review: Observability lets you inspect interactions even when users leave no explicit feedback. Pesok suggests reviewing about 100 random samples once a week as a practical routine, not an established optimal sample size.
- Community reports: Forums and X can surface failures users care enough to describe, though social feedback also contains noise.
These sources complement one another: explicit complaints identify pain, while random review helps reveal usage that never becomes a complaint.
There is no shortcut around understanding the data. A useful court has recognizable boundaries, coverage within them, and visible clusters of success and failure. If many users struggle in the bottom-right corner, that region can become next week’s engineering priority. The objective is to turn relevant failures from red to blue, rather than merely increase the number of eval cases.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep inputs stable and share the production logic
Put constants in the data and variables in the task. The user’s strawberry question stays fixed while the implementation changes. A system-prompt experiment should change the task, not require rewriting every dataset entry. The same separation applies when testing preprocessing or retrieval-augmented generation, or RAG.
| Keep in the data | Vary in the task |
|---|---|
| User’s question | System prompt |
| Stable test case | Preprocessing |
| Same input across experiments | Retrieval behavior |
This makes the cases reusable and keeps the experiment’s moving parts visible.
AI SDK middleware provides a place to share prompting and preprocessing logic, including retrieval, between the production completion route and the evaluation task. The linked documentation is for AI SDK 4; it supports this middleware pattern without establishing the exact SDK version used in the talk.
Reuse that logic in both paths so an eval exercises substantially the same behavior that users receive. Basketball practice is useful because it resembles the game. Likewise, a model comparison becomes less informative if the evaluation task omits the retrieval or prompt preparation that production uses. Shared code reduces that mismatch.
Suggest correction
Score the failure you can recognize
Scoring depends on the domain. A fruit counter has a concrete correct count; judging writing can be much harder. Where possible, prefer deterministic pass/fail scoring. Evaluation produces many outputs and logs, so an understandable score makes debugging easier. An elaborate scoring system can also make it harder for colleagues and other teams to understand or reuse the evals.
Start with the evidence you would inspect manually to recognize failure, then write a check for that evidence. For v0, one concern is whether the generated code works. For writing, the relevant signal might be a linguistic property. If code cannot capture the judgment reliably, human review is acceptable: the purpose is to collect trustworthy signal about the court, even when obtaining it takes work.
Pesok also suggests requesting the final answer inside answer tags to make extraction and string matching easier. This is a deliberate exception to keeping evaluation and production identical: the extra formatting may be unwanted in the user-facing response. For the same strawberry case, the following TypeScript example makes that extraction rule concrete. It uses a constructed tagged response, not a recorded model output.
typescript
const testCase = {
input: "How many Rs are in strawberry?",
expected: 3,
};
const formattingInstruction =
"Output your final answer inside <answer>...</answer> tags.";
function scoreAnswer(output: string, expected: number): 0 | 1 {
const match = /^\s*([0-9]+)\s*$/.exec(
output.match(/<answer>(.*?)<\/answer>/s)?.[1] ?? "",
);
return match !== null && Number(match[1]) === expected ? 1 : 0;
}
const evaluationPrompt = `${testCase.input}\n${formattingInstruction}`;
const exampleOutput = "<answer>3</answer>";
console.log({
evaluationPrompt,
score: scoreAnswer(exampleOutput, testCase.expected),
});
The check isolates the answer field instead of searching all response text for a digit that might appear in an explanation. It scores the formatted answer; it does not establish how reliably the model will produce it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Review changes across the whole court
Adding evals to CI makes them part of reviewing a change. Braintrust can run the task across the dataset and produce a report of improvements and regressions. For a pull request that changes the prompt, the review becomes concrete:
- Run the proposed task against the evaluation data.
- Inspect cases that changed from failure to success.
- Inspect cases that previously passed but now fail.
A prompt may fix one region while breaking another. The report exposes both movements; Pesok does not prescribe a universal score threshold that must block deployment.
The same court supports other comparisons. Switching models resembles trying a different player in the same practice environment. Changing retrieval or the system prompt becomes an experiment whose effects can be shown to a colleague. Measurement gives a claim of improvement something firmer than a handful of convincing outputs.
Pesok connects this discipline to better reliability and quality, higher conversion and retention, and less support and operations work. These are anticipated product benefits, not measured business results presented in the talk. The court itself was built with a v0-generated app that let him place made and missed shots—a small tool for making the evaluation landscape visible.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Scheduled runs and repeated trials answer different questions
The closing Q&A asks whether to run the same eval repeatedly. Pesok initially answers in terms of practice over time: even a generally accurate basketball player misses some shots. Pesok says his team runs evals at least daily to locate failures and detect regressions. A regular schedule helps reveal where the system is going wrong.
The questioner then clarifies a different idea: run the same question five times and distinguish four successes from five. That concerns variability for one input, rather than the schedule for checking the system. Pesok begins relating harder questions to shots farther from the basket, but the recording ends during his answer. The exchange leaves the repetition count and success-rate estimation method unresolved.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Versioned examples for wrapping models with reusable preprocessing and using them with streamText.
Further reading
Pesok and colleagues explain v0's grading methods, user-feedback loop and evaluation reports for pull requests.
A contemporaneous account of v0's retrieval, model routing and automated error correction, with internal evaluation results.
Updates since the talk
Current guidance for running evaluations in code or CI and comparing recorded results.
Read the complete timestamped transcript
- 0:00
[upbeat music] My name is Ido.
- 0:16
I'm an engineer at Vercel working on v0.
- 0:20
If you don't know, v0 is a full-stack vibe coding platform. It's the easiest and fastest way to prototype, build on the web, and express new ideas. Uh, here are some examples of cool things people have built and shared on Twitter.
- 0:35
And to catch you up, we recently just launched GitHub Sync, so you can now push generated code to GitHub directly from v0. You can also, uh, automatically pull changes from GitHub into your chat and furthermore switch branches and open PRs to collaborate with your team.
- 0:52
I'm very excited to announce we recently crossed one hundred million messages sent, and we're really excited to keep growing from here.
- 1:00
So my goal of this talk is for it to be an introduction to evals, and specifically at the application layer. You may be used to evals at the model layer, which is what the research labs will cite in model releases.
- 1:13
But this will be a focus on what do evals mean for your users, your apps, and your data. The model's now in the wild, out of the lab, and it needs to work for your use case.
- 1:23
And to do this, I have a story. Uh, it's a story about this app called Fruit Letter Counter. And if the name didn't already give it away, all it is is an app that counts the letters in fruit.
- 1:36
So the vision is we'll make a logo with ChatGPT. Uh, there might be product market fit already because everyone on X is dying to know the number of letters in fruit.
- 1:45
If you didn't get it, it's a joke on the how many Rs are in strawberry prompt. Uh, we'll have v0 make all the UI and back end, and then we can ship.
- 1:55
So we had v0 write the code. It, it used, uh, AI SDK to do the stream text call. And what do you know? It worked first try. GPT-4.1 said three.
- 2:05
And not only did it say three once, I even tested it twice, and it worked both times in a row. So from there, we're good to ship, right? Let's launch on Twitter.
- 2:15
Wanna know how many letters are in a fruit? Just launched fruitlettercounter.io. The dot com and dot AI were taken. Um, and yeah, everything was going great. We launched and deployed on Vercel.
- 2:28
We had Fluid Compute on until we suddenly get this tweet. John said, "I asked how many Rs in strawberry, and it said two." So of course, I just tested it twice.
- 2:39
How is this even possible? Um, but I think you get where I'm going with this, which is that by nature, LLMs can be very unreliable. And this principle scales from a small letter-counting app all the way to the biggest AI apps in the world.
- 2:56
The reason why it's so important to recognize this is because no one is going to use something that doesn't work. It's literally unusable. Um, and this is a significant challenge when you're building AI apps.
- 3:06
So I have a funny meme here, but basically AI apps have this unique property. They're very, like, demo savvy. You'll demo it. It looks super good. You'll show it to your coworkers, and then you ship to prod, and then suddenly hallucinations come and get you.
- 3:20
Um, so we always have this in our, in the back of our head when we're building.
- 3:25
Back to where we were, let's actually not give up, right? We actually wanna solve this for our users, and we wanna make a really good fruit letter-counting app. So you might say, how do we make reliable software that uses LLMs?
- 3:38
Our initial, uh, prompt was a simple question, right? But maybe we can try prompt engineering. Maybe we can add some chain of thought, something else to make it more reliable.
- 3:46
So we spend all night working on this new prompt. Uh, you're an exuberant fruit-loving AI on an epic quest, dot, dot, dot. Uh, and this time we actually tested it ten times in a row on ChatGPT, and it worked every single time, ten times in a row.
- 4:01
It's amazing. So we ship, and everything was going great until John tweeted at me again. And he said, "I asked how many Rs are in strawberry, banana, pineapple, mango, kiwi, dragon fruit, apple, raspberry," and it said five.
- 4:19
So we failed John again. Um, although this example is pretty simple, but this is actually what will happen when you start deploying to production. You'll get users that come up with queries you could have never imagined, and you actually have to start thinking about how do we solve it.
- 4:34
And the interesting thing, if you think about it, is ninety-five percent of our app works one hundred percent of the time. We can have unit tests for every single function, end-to-end tests for the auth, the login, the sign-out.
- 4:45
It will all work. But it's that most crucial five percent that can fail on us. So let's improve it.
- 4:52
Now, to visualize this, I have a diagram for you. Hopefully, you can see the code. Uh, maybe I need to make my screen brighter. Can you see the code?
- 4:59
I don't know. Okay. Um, okay. Well, we'll come back to this. But basically, we're gonna start building evals. And to visualize this, I have a basketball court. So today's day one of the NBA Finals.
- 5:14
I don't know if you care. Um, you don't need to know much about basketball, but just know that someone is trying to throw a ball in the basket, and here the basket is the glowing golden circ-- uh, glowing golden circle.
- 5:26
So blue will represent a shot make, and red will represent a shot miss. And one property to consider is that the farther away your shot is from the basket, the harder it is.
- 5:37
Uh, another property is that the court has boundaries. So this blue dot, although the shot goes in, it's out of your, uh, out of the court, so it doesn't really count in the game.
- 5:46
Let's start plotting our data. So here we have a question, how many Rs in strawberry? This, after our new prompt, will probably work, so we'll label it blue. Um, and we'll put it close to the basket because it's pretty easy.
- 5:59
However, how many Rs are in that big array? We'll label it red, and we'll put it farther away from the basket
- 6:06
Hopefully you can see that. Maybe we can make it a little bit brighter. But this is the data part of our eval. Basically, you're trying to collect, uh, what, what prompts your users are asking, and you wanna just store this over time and keep building it and store it where these points are on your court.
- 6:23
Two more prompts I wanna bring up is, like, what if someone says, "How many Rs are in strawberry, pineapple, dragon fruit, mango?" after we replace all the vowels with Rs, right?
- 6:33
Insane prompt, but still technically in our domain. Uh, so we'll, we'll label it as red all the way down there. Um, but a funny one is like, "How many syllables are in carrot?"
- 6:44
So this, we'll call it out of bounds, right? This, no- none of our users are actually gonna ask. Um, it's not part of our app, so no one is going to care.
- 6:53
Um, I hope you can see the code. But basically, when you're making eval, here's how you can think about it. Your data is the point on the court. Your shot, or in this case, in Braintrust they call it a task, is the way you shoot the ball towards the basket, and your score is basically a check of
- 7:10
did it go in the basket or did it not go in the basket?
- 7:13
To make good evals, you must understand your court. This is the most important step.
- 7:20
And you have to be careful of falling into some traps. First is the out of bounds traps. Don't spend time making evals for your data your users don't care about.
- 7:29
You have enough problems, I promise you, of problem, uh, queries that your users do care about. So be careful not t- try and be productive and, you know, you, you're making a lot of evals, but they're not really applicable to your app.
- 7:41
And another visualization is don't have a concentrated set of points. When you really understand your court, you're gonna understand, you know, where the boundaries are, and you wanna make sure you t- you test across the entire court.
- 7:53
Uh, a lot of people have been talking about this today, but to collect as much data as possible, here are some, uh, things you can do. First is collect thumbs up, thumbs down data.
- 8:02
This can be noisy, but it also can be a really, really good signal as to where your app is struggling. Another thing is if you have observability, which is highly recommended, you can just read through random samples in your log, in your logs.
- 8:14
Um, although users might not be, you know, giving you signal, but if you take like 100 random samples and go through it like once a week, you'll get a really unders- good understanding of what your users are and how your users are using the product.
- 8:26
Uh, c- if you have community forums, these are also great. People will often report issues they're having with the LLM, and also X and Twitter are also great, but can be noisy.
- 8:36
And there really is no shortcut here. You really have to do the work and understand what your court looks like. So here is actually what if you are doing a good job of understanding your court and a good job of building your data set, this is what it should look like.
- 8:50
You should know the boundaries, you should be testing in your boundaries, and you should understand where your system is, has blue and versus where it has red. So here it's really easy to tell, okay, maybe next week we need to prioritize, uh, the team to work on that bottom right corner.
- 9:05
This is something where a lot of users are struggling, and we can really do a good job on flipping the tiles from red to blue.
- 9:15
Another thing you can do, and I hope, I really hope you can see, but you wanna put constants in data, variables in the task. So just like in math or programming, you wanna factor constants so it improves clarity, reuse, and generalizations.
- 9:32
If you have a... Let's say you wanna test your system prompt, right? Keep the constant data that all, that your users are gonna ask. So for example, how many Rs in strawberry, that goes in the data.
- 9:41
That's a constant. It's never gonna change throughout your app. But what you're gonna test is in that task, you're gonna try different system prompts. You might try different preprocessing, different RAG, and that's what you wanna put in your task section.
- 9:52
This way, your app actually scales and you never have to, let's say, when you change your system prompt, redo all your data. And this is a really nice feature of Braintrust.
- 10:01
Um, and if you don't know, AI SDK actually offers a thing called middleware, and it's a really good abstraction to put basically all your logic of preprocessing, so RAG, system prompt you can put in here, et cetera.
- 10:15
And you can now share this between your actual API route that's doing the completion and your evals. So if you think about the court, the basketball court as if we're doing, we're going like basketball practice and we're trying to practice our system acro- across different models, um, you want your practice to be as similar as possible to
- 10:31
the real game. That's what makes a good practice. So you want to share the, pretty much the exact same code between the evals and what you're actually running.
- 10:40
Now, I wanna talk a little bit about scores, which is the last step of the eval. The unfortunate thing is it does vary greatly depending on your domain. So in this case, it's like super simple.
- 10:52
Uh, you're just checking if, you know, the output contains the correct number of letters. But maybe if you're doing writing or tasks like writing, that's very, very difficult. Um,
- 11:03
from principles, you wanna actually lean towards deterministic scoring and pass-fail. This is because when you're doing debugging, uh, you're going to get a ton of input and logs, and you wanna make it as easy as possible for you to actually figure out what's going wrong.
- 11:16
So if you're sh- if you're building, if you're over-engineering your score, it might be very difficult to share with your team and distribute across different teams, uh, your evals because no one will understand how these things are getting scored.
- 11:28
Keep your scores as simple as possible. Um, and a good question to ask yourself is when you're looking at the data, what am I looking for to see if this failed, right?
- 11:37
So with v0, we're looking for if the code didn't work. Um, but maybe for writing you're looking for s- a certain linguistics. Ask yourself that question and write the code that looks for you.
- 11:48
Um, there are some cases where it's so hard to write the code that you may need to do human review, and that's okay. At the end of the day, you wanna build your court and you wanna collect signal.
- 11:57
Even if you need, you must do human, human review to get the correct signal, don't worry. At the... If you do the correct practice, it will pay off in the long run and you'll get better results for your users.
- 12:09
One trick you can do for scoring is don't be scared to, like, add a little bit of extra, um, prompt to your-- to the original prompt. So for example, here we can say, "Output your final answer, uh, in these answer tags."
- 12:22
What this will do is basically make it very easy for you to do string matching, um, and et cetera, whereas in production you don't really want this. But yeah, you can do some little tw- tweaks to your prompt so that scoring is easier.
- 12:36
Another thing we really highly recommend is add evals to your CI. So Braintrust is really nice because you can get these eval reports. Um, so it'll run your ev- your task across all your data, and then it'll give you this, uh, report at the end for the improvements and regressions.
- 12:51
Assume my colleague made a PR that changes a bit of the prompt. We wanna know, like, how did it do across the court, right? Visualize, like, did it change more tiles from red to blue?
- 13:00
Maybe now our prompt fixed one part, but it broke the other part of our app. Um, so this is a really useful report to have when you're doing PRs.
- 13:10
So yeah, going back, this, this is the summary of the talk. You wanna make your evals a, a core of your data, and this, you can treat it like practice.
- 13:20
Your model is basically going to practice. Maybe you wanna switch players, right? When you switch models, you can see how a different player is gonna perform in your practice.
- 13:27
But this gives you such a good understanding of how your system is doing when you change things like maybe your RAG or your system prompt, and you can now go to your colleague and say, "Hey, this actually did help our app," right?
- 13:39
Because improvement without measurement is limited and imprecise, and evals give you the clarity you need to systematically improve your app.
- 13:50
When you do that, you're gonna get better reliability and quality, higher conversion and retention, and you also get to do-- just spend less time on support and ops, right?
- 13:59
Because your evals, your practice environment will take care of that for you. Uh, and if you're wondering about how I built all these court diagrams, I actually just used v0, and it made me some app that I just added these shots, uh, made and missed in, uh, the basket.
- 14:13
So yeah, thank you very much. I hope you learned a little bit about evals. [audience applauding]
- 14:17
Thank you. So we do have some time for some questions. There are two mics, one over here, one over there. Um, we can take two or three of those, please, if anybody's interested in asking.
- 14:29
We have one over there. Yeah, real quick. Um, mic five, please.
- 14:36
Or you can repeat the question as well, if you don't mind. Do you run the same eval over and over again, like basically measuring the same thing?
- 14:42
Yeah. Yeah, you can think of it... It's really like practice. Like maybe you're a basketball player with like, you know, in general will score like ninety percent, but they might miss more shots here or there.
- 14:53
If you run it like-- We do it, like, we run every day at least. Um, and then we get a good sense of like where are we actually, like, failing?
- 15:00
Did we have some regression? Um, so yeah, running it like pre- daily or at least in some, some schedule will give you a good idea.
- 15:06
I was thinking, what if you ran like, you know, the same question through it five times, right? As like-
- 15:11
Yeah
- 15:11
... like what's the percentage? It's maybe a four out of five or, you know, five out of five.
- 15:14
Oh, I see.
- 15:15
Right.
- 15:15
So it's definitely like as you go further away, like the harder questions get like-