Evals Are Not Unit Tests

Talk transcript

1. 0:00

   [upbeat music] My name is Ido.

2. 0:16

   I'm an engineer at Vercel working on v0.

3. 0:20

   If you don't know, v0 is a full-stack vibe coding platform. It's the easiest and fastest way to prototype, build on the web, and express new ideas. Uh, here are some examples of cool things people have built and shared on Twitter.

4. 0:35

   And to catch you up, we recently just launched GitHub Sync, so you can now push generated code to GitHub directly from v0. You can also, uh, automatically pull changes from GitHub into your chat and furthermore switch branches and open PRs to collaborate with your team.

5. 0:52

   I'm very excited to announce we recently crossed one hundred million messages sent, and we're really excited to keep growing from here.

6. 1:00

   So my goal of this talk is for it to be an introduction to evals, and specifically at the application layer. You may be used to evals at the model layer, which is what the research labs will cite in model releases.

7. 1:13

   But this will be a focus on what do evals mean for your users, your apps, and your data. The model's now in the wild, out of the lab, and it needs to work for your use case.

8. 1:23

   And to do this, I have a story. Uh, it's a story about this app called Fruit Letter Counter. And if the name didn't already give it away, all it is is an app that counts the letters in fruit.

9. 1:36

   So the vision is we'll make a logo with ChatGPT. Uh, there might be product market fit already because everyone on X is dying to know the number of letters in fruit.

10. 1:45

    If you didn't get it, it's a joke on the how many Rs are in strawberry prompt. Uh, we'll have v0 make all the UI and back end, and then we can ship.

11. 1:55

    So we had v0 write the code. It, it used, uh, AI SDK to do the stream text call. And what do you know? It worked first try. GPT-4.1 said three.

12. 2:05

    And not only did it say three once, I even tested it twice, and it worked both times in a row. So from there, we're good to ship, right? Let's launch on Twitter.

13. 2:15

    Wanna know how many letters are in a fruit? Just launched fruitlettercounter.io. The dot com and dot AI were taken. Um, and yeah, everything was going great. We launched and deployed on Vercel.

14. 2:28

    We had Fluid Compute on until we suddenly get this tweet. John said, "I asked how many Rs in strawberry, and it said two." So of course, I just tested it twice.

15. 2:39

    How is this even possible? Um, but I think you get where I'm going with this, which is that by nature, LLMs can be very unreliable. And this principle scales from a small letter-counting app all the way to the biggest AI apps in the world.

16. 2:56

    The reason why it's so important to recognize this is because no one is going to use something that doesn't work. It's literally unusable. Um, and this is a significant challenge when you're building AI apps.

17. 3:06

    So I have a funny meme here, but basically AI apps have this unique property. They're very, like, demo savvy. You'll demo it. It looks super good. You'll show it to your coworkers, and then you ship to prod, and then suddenly hallucinations come and get you.

18. 3:20

    Um, so we always have this in our, in the back of our head when we're building.

19. 3:25

    Back to where we were, let's actually not give up, right? We actually wanna solve this for our users, and we wanna make a really good fruit letter-counting app. So you might say, how do we make reliable software that uses LLMs?

20. 3:38

    Our initial, uh, prompt was a simple question, right? But maybe we can try prompt engineering. Maybe we can add some chain of thought, something else to make it more reliable.

21. 3:46

    So we spend all night working on this new prompt. Uh, you're an exuberant fruit-loving AI on an epic quest, dot, dot, dot. Uh, and this time we actually tested it ten times in a row on ChatGPT, and it worked every single time, ten times in a row.

22. 4:01

    It's amazing. So we ship, and everything was going great until John tweeted at me again. And he said, "I asked how many Rs are in strawberry, banana, pineapple, mango, kiwi, dragon fruit, apple, raspberry," and it said five.

23. 4:19

    So we failed John again. Um, although this example is pretty simple, but this is actually what will happen when you start deploying to production. You'll get users that come up with queries you could have never imagined, and you actually have to start thinking about how do we solve it.

24. 4:34

    And the interesting thing, if you think about it, is ninety-five percent of our app works one hundred percent of the time. We can have unit tests for every single function, end-to-end tests for the auth, the login, the sign-out.

25. 4:45

    It will all work. But it's that most crucial five percent that can fail on us. So let's improve it.

26. 4:52

    Now, to visualize this, I have a diagram for you. Hopefully, you can see the code. Uh, maybe I need to make my screen brighter. Can you see the code?

27. 4:59

    I don't know. Okay. Um, okay. Well, we'll come back to this. But basically, we're gonna start building evals. And to visualize this, I have a basketball court. So today's day one of the NBA Finals.

28. 5:14

    I don't know if you care. Um, you don't need to know much about basketball, but just know that someone is trying to throw a ball in the basket, and here the basket is the glowing golden circ-- uh, glowing golden circle.

29. 5:26

    So blue will represent a shot make, and red will represent a shot miss. And one property to consider is that the farther away your shot is from the basket, the harder it is.

30. 5:37

    Uh, another property is that the court has boundaries. So this blue dot, although the shot goes in, it's out of your, uh, out of the court, so it doesn't really count in the game.

31. 5:46

    Let's start plotting our data. So here we have a question, how many Rs in strawberry? This, after our new prompt, will probably work, so we'll label it blue. Um, and we'll put it close to the basket because it's pretty easy.

32. 5:59

    However, how many Rs are in that big array? We'll label it red, and we'll put it farther away from the basket

33. 6:06

    Hopefully you can see that. Maybe we can make it a little bit brighter. But this is the data part of our eval. Basically, you're trying to collect, uh, what, what prompts your users are asking, and you wanna just store this over time and keep building it and store it where these points are on your court.

34. 6:23

    Two more prompts I wanna bring up is, like, what if someone says, "How many Rs are in strawberry, pineapple, dragon fruit, mango?" after we replace all the vowels with Rs, right?

35. 6:33

    Insane prompt, but still technically in our domain. Uh, so we'll, we'll label it as red all the way down there. Um, but a funny one is like, "How many syllables are in carrot?"

36. 6:44

    So this, we'll call it out of bounds, right? This, no- none of our users are actually gonna ask. Um, it's not part of our app, so no one is going to care.

37. 6:53

    Um, I hope you can see the code. But basically, when you're making eval, here's how you can think about it. Your data is the point on the court. Your shot, or in this case, in Braintrust they call it a task, is the way you shoot the ball towards the basket, and your score is basically a check of

38. 7:10

    did it go in the basket or did it not go in the basket?

39. 7:13

    To make good evals, you must understand your court. This is the most important step.

40. 7:20

    And you have to be careful of falling into some traps. First is the out of bounds traps. Don't spend time making evals for your data your users don't care about.

41. 7:29

    You have enough problems, I promise you, of problem, uh, queries that your users do care about. So be careful not t- try and be productive and, you know, you, you're making a lot of evals, but they're not really applicable to your app.

42. 7:41

    And another visualization is don't have a concentrated set of points. When you really understand your court, you're gonna understand, you know, where the boundaries are, and you wanna make sure you t- you test across the entire court.

43. 7:53

    Uh, a lot of people have been talking about this today, but to collect as much data as possible, here are some, uh, things you can do. First is collect thumbs up, thumbs down data.

44. 8:02

    This can be noisy, but it also can be a really, really good signal as to where your app is struggling. Another thing is if you have observability, which is highly recommended, you can just read through random samples in your log, in your logs.

45. 8:14

    Um, although users might not be, you know, giving you signal, but if you take like 100 random samples and go through it like once a week, you'll get a really unders- good understanding of what your users are and how your users are using the product.

46. 8:26

    Uh, c- if you have community forums, these are also great. People will often report issues they're having with the LLM, and also X and Twitter are also great, but can be noisy.

47. 8:36

    And there really is no shortcut here. You really have to do the work and understand what your court looks like. So here is actually what if you are doing a good job of understanding your court and a good job of building your data set, this is what it should look like.

48. 8:50

    You should know the boundaries, you should be testing in your boundaries, and you should understand where your system is, has blue and versus where it has red. So here it's really easy to tell, okay, maybe next week we need to prioritize, uh, the team to work on that bottom right corner.

49. 9:05

    This is something where a lot of users are struggling, and we can really do a good job on flipping the tiles from red to blue.

50. 9:15

    Another thing you can do, and I hope, I really hope you can see, but you wanna put constants in data, variables in the task. So just like in math or programming, you wanna factor constants so it improves clarity, reuse, and generalizations.

51. 9:32

    If you have a... Let's say you wanna test your system prompt, right? Keep the constant data that all, that your users are gonna ask. So for example, how many Rs in strawberry, that goes in the data.

52. 9:41

    That's a constant. It's never gonna change throughout your app. But what you're gonna test is in that task, you're gonna try different system prompts. You might try different preprocessing, different RAG, and that's what you wanna put in your task section.

53. 9:52

    This way, your app actually scales and you never have to, let's say, when you change your system prompt, redo all your data. And this is a really nice feature of Braintrust.

54. 10:01

    Um, and if you don't know, AI SDK actually offers a thing called middleware, and it's a really good abstraction to put basically all your logic of preprocessing, so RAG, system prompt you can put in here, et cetera.

55. 10:15

    And you can now share this between your actual API route that's doing the completion and your evals. So if you think about the court, the basketball court as if we're doing, we're going like basketball practice and we're trying to practice our system acro- across different models, um, you want your practice to be as similar as possible to

56. 10:31

    the real game. That's what makes a good practice. So you want to share the, pretty much the exact same code between the evals and what you're actually running.

57. 10:40

    Now, I wanna talk a little bit about scores, which is the last step of the eval. The unfortunate thing is it does vary greatly depending on your domain. So in this case, it's like super simple.

58. 10:52

    Uh, you're just checking if, you know, the output contains the correct number of letters. But maybe if you're doing writing or tasks like writing, that's very, very difficult. Um,

59. 11:03

    from principles, you wanna actually lean towards deterministic scoring and pass-fail. This is because when you're doing debugging, uh, you're going to get a ton of input and logs, and you wanna make it as easy as possible for you to actually figure out what's going wrong.

60. 11:16

    So if you're sh- if you're building, if you're over-engineering your score, it might be very difficult to share with your team and distribute across different teams, uh, your evals because no one will understand how these things are getting scored.

61. 11:28

    Keep your scores as simple as possible. Um, and a good question to ask yourself is when you're looking at the data, what am I looking for to see if this failed, right?

62. 11:37

    So with v0, we're looking for if the code didn't work. Um, but maybe for writing you're looking for s- a certain linguistics. Ask yourself that question and write the code that looks for you.

63. 11:48

    Um, there are some cases where it's so hard to write the code that you may need to do human review, and that's okay. At the end of the day, you wanna build your court and you wanna collect signal.

64. 11:57

    Even if you need, you must do human, human review to get the correct signal, don't worry. At the... If you do the correct practice, it will pay off in the long run and you'll get better results for your users.

65. 12:09

    One trick you can do for scoring is don't be scared to, like, add a little bit of extra, um, prompt to your-- to the original prompt. So for example, here we can say, "Output your final answer, uh, in these answer tags."

66. 12:22

    What this will do is basically make it very easy for you to do string matching, um, and et cetera, whereas in production you don't really want this. But yeah, you can do some little tw- tweaks to your prompt so that scoring is easier.

67. 12:36

    Another thing we really highly recommend is add evals to your CI. So Braintrust is really nice because you can get these eval reports. Um, so it'll run your ev- your task across all your data, and then it'll give you this, uh, report at the end for the improvements and regressions.

68. 12:51

    Assume my colleague made a PR that changes a bit of the prompt. We wanna know, like, how did it do across the court, right? Visualize, like, did it change more tiles from red to blue?

69. 13:00

    Maybe now our prompt fixed one part, but it broke the other part of our app. Um, so this is a really useful report to have when you're doing PRs.

70. 13:10

    So yeah, going back, this, this is the summary of the talk. You wanna make your evals a, a core of your data, and this, you can treat it like practice.

71. 13:20

    Your model is basically going to practice. Maybe you wanna switch players, right? When you switch models, you can see how a different player is gonna perform in your practice.

72. 13:27

    But this gives you such a good understanding of how your system is doing when you change things like maybe your RAG or your system prompt, and you can now go to your colleague and say, "Hey, this actually did help our app," right?

73. 13:39

    Because improvement without measurement is limited and imprecise, and evals give you the clarity you need to systematically improve your app.

74. 13:50

    When you do that, you're gonna get better reliability and quality, higher conversion and retention, and you also get to do-- just spend less time on support and ops, right?

75. 13:59

    Because your evals, your practice environment will take care of that for you. Uh, and if you're wondering about how I built all these court diagrams, I actually just used v0, and it made me some app that I just added these shots, uh, made and missed in, uh, the basket.

76. 14:13

    So yeah, thank you very much. I hope you learned a little bit about evals. [audience applauding]

77. 14:17

    Thank you. So we do have some time for some questions. There are two mics, one over here, one over there. Um, we can take two or three of those, please, if anybody's interested in asking.

78. 14:29

    We have one over there. Yeah, real quick. Um, mic five, please.

79. 14:36

    Or you can repeat the question as well, if you don't mind. Do you run the same eval over and over again, like basically measuring the same thing?

80. 14:42

    Yeah. Yeah, you can think of it... It's really like practice. Like maybe you're a basketball player with like, you know, in general will score like ninety percent, but they might miss more shots here or there.

81. 14:53

    If you run it like-- We do it, like, we run every day at least. Um, and then we get a good sense of like where are we actually, like, failing?

82. 15:00

    Did we have some regression? Um, so yeah, running it like pre- daily or at least in some, some schedule will give you a good idea.

83. 15:06

    I was thinking, what if you ran like, you know, the same question through it five times, right? As like-

84. 15:11

    Yeah

85. 15:11

    ... like what's the percentage? It's maybe a four out of five or, you know, five out of five.

86. 15:14

    Oh, I see.

87. 15:15

    Right.

88. 15:15

    So it's definitely like as you go further away, like the harder questions get like-