7 Habits of Highly Effective Generative AI Evaluations

Talk transcript

1. 0:05

   Hello. Welcome to my talk on the seven habits of highly effective generative AI evaluations.

2. 0:13

   I'm Justin Muller. I'm a principal applied AI architect at AWS. I have degrees in physics, math, law, and business. I've been working in natural language processing for about fifteen years on and off, and generative AI for the last four years.

3. 0:28

   Uh, inside of AWS, my team is a small specialist team that helps customers scale GenAI workloads. So I've had the opportunity to work on many, many workloads across all different industries, all different sizes of customers.

4. 0:40

   Some small workloads, some of the largest in North America. Um, I'll also mention that because of that experience, I've seen a lot of workloads fail, and I've seen a lot of workloads succeed.

5. 0:51

   And so one of the things my team does now is that we've gathered up a lot of the best practices we've seen across the successful workloads, as well as some of the common failure points we've seen on the failed workloads and share them as best practices in talks like this one.

6. 1:06

   All right, so let's get going. The first question is, uh, what is the biggest challenge in scaling generative AI? You may, you may be able to guess the solution here, as this is what the talk's all about.

7. 1:18

   Um, but I, I would argue that it's evaluations.

8. 1:23

   I- in, in my experience, having been called in to help many different customers scale their GenAI workloads, there's lots of concerns that come up. And when I ask this question with a live audience, there's things like cost, hallucinations, accuracy, capacity, uh, all these different types of, um, concerns come up.

9. 1:42

   But the number one thing that I see across all workloads is a lack of evaluations. And in particular, I call it the missing piece to scaling GenAI because a lot of times when a customer has, uh, successfully built a, a POC and they have a, uh, a workload that they like or chatbot that seems cool, and they're

10. 1:59

    figuring out how to scale it, um, this is the piece that I come in, and most often it's missing. And when we add evaluations, it unlocks the ability to scale, and we're gonna look at how that happens.

11. 2:13

    But it's by far the most common way to unlock scale.

12. 2:17

    All right, I'll give you a quick customer example. Um, in, uh, July 2024, I was called in to-- as a, as like an escalation on a workload where a customer was doing document processing, and they'd been working on this document processing workload for about, I think six months, twelve months, maybe.

13. 2:34

    Um, and they had six or eight engineers working on it. So it was a pretty big project at that point. And the, uh, the, uh, the VP of technology that was in charge of the project called me in.

14. 2:45

    He asked everyone else to leave the room, and he was like: "Look, this project, we spent a bunch of money on it. Accuracy is at twenty-two percent.

15. 2:53

    We're thinking about just cutting the project. We're not getting the results we need. So I need you to tell me if we can save this thing." And, uh, essentially, I spent a couple of weeks doing discovery and understanding what was going on in the workload.

16. 3:06

    And the number one thing that I realized was they had zero evaluations, right? They had this big process, and end to end, they had a single number coming out the end that said, well, it's twenty-two percent accurate.

17. 3:16

    So what I did is design a evaluations framework, and the results were almost funny because once the evaluation framework was in place and you could see exactly where the problems were, fixing them were trivial, right?

18. 3:28

    Fixing the problems wasn't really the challenge. Increasing accuracy wasn't really the challenge. It was knowing where the problems were and what was causing them. So over the course of the next six months, they built the evaluation framework.

19. 3:39

    They fixed, uh, many of the issues that came up. They were able to achieve ninety-two accurate-- percent accuracy by January, uh, when, uh, which ninety percent was their threshold for launching to, to production at scale.

20. 3:52

    And they launched and became the single largest document processing workload, uh, on A-AWS in North America at the time.

21. 4:01

    All right, so evaluations are really, really important. What are they? How do we actually build these things? The-- this is the place I like to start with evaluations because m-many times, especially if you're coming from a traditional AI/ML back- uh, background, you look at evaluations as a way to measure quality

22. 4:21

    because you're, you're familiar with things like, well, what's your F1 score? What's your precision and recall? And you get a score, and it just tells you how well you're doing.

23. 4:30

    And while, uh, generative AI evaluations do produce a score, if anything, it's a, it's a far third of the top reasons why we do evaluations with GenAI workloads.

24. 4:41

    The main goal with any evaluation framework should be to discover problems. Similar to the workload example that I just shared, if your evaluation framework tells you where the problems are and perhaps even suggests solutions because it includes an element of generative AI reasoning, now we can improve our workloads.

25. 4:57

    And of course, yes, we need to measure quality. That's very important. But I, I start this way because when you think about designing an evaluation framework, if you come in with the mindset that it's gonna find errors, you design it in a very different way than if you come in and, and with the mindset that it's just

26. 5:14

    gonna measure how well you're doing. All right? So let's double-click on what that looks like.

27. 5:21

    Um, and, and before we do that, though, I do wanna, I would-- I do wanna add a second aside on why I think evaluations are so por- important. Um,

28. 5:31

    my team within AWS is a very small specialist team, and so we have to have very, very strict filters on which projects we say yes to and which ones we support.

29. 5:40

    And my experience has been evaluations are the number one filter that separates a science project from a successful project. When I talk to a team and the team says, "Can you teach me about multi-agent collaboration?"

30. 5:55

    And I say, "Great. Let's sit down and spend two hours putting together a gold standard set for our evaluations so that your multi-agent collaboration framework project is successful." If the team comes back and says, "Oh gosh Two hours on eval sounds boring.

31. 6:10

    Can you just give me the toys to play with? I, I know right away that's a science project, and it's not gonna go anywhere. And, and science projects are important.

32. 6:18

    They're a fun way to learn. They're just not what goes to scale. On the other hand, for my projects that have been wildly successful, where customers have had a hundred x return on investment or maybe cut their costs by ten thousand percent or something like that, those customers, when I say, "Hey, can we spend two hours, uh,

33. 6:34

    building an eval framework?" They say, "Why don't we spend, spend four?" Right? Evals are so important, and we recognize that, and this project is so important that we're gonna invest the time.

34. 6:44

    So I, as an aside, I'll just mention it as my number one filter for deciding which projects are gonna be successful and which are worth spending time on.

35. 6:52

    All right. So evals in the generative AI world come with baggage, right? And especially if you have a traditional AI/ML background, it can be a little scary, right? You, you look at evals and it's free text coming out, and you're like, "Oh my gosh, it's not gonna be a specific number.

36. 7:07

    I can't mathematically calculate an exact number every time. Let's panic."

37. 7:12

    Right? And it turns out that the human race has been grading and evaluating, uh, free text workloads for, uh, hundreds of years, if not thousands, right? Um, can you imagine if my professor in English assigns me an essay and says, uh, "Write me a three-page essay on the, on the emotional meaning of sunshine."

38. 7:37

    And I, I spend an, I spend my hours and I write my essay and I turn it in. The professor says, "Well, I can't calculate an exact answer to this.

39. 7:44

    I can't calculate your exact score, so I give up. I'm just not gonna score it." No, of course not. The professor's gonna give it a score. And we can evaluate generative AI in the same way that we've been evaluating free text for, for decades and generations, right?

40. 7:59

    But the key understanding is if you have a bad professor, they just give you a score. Maybe you've had this experience like me, you've turned in an essay and got an F, right?

41. 8:07

    If you just get a score, you're like, "Well, that, that stinks, but what do I do with it?" Right? With generative AI, we can go a bit deeper. We can do what the good professors do, which is point out what you did wrong, point out where you can improve, right?

42. 8:21

    This is why evaluations in the term, in the, in the context of grade school are very, very valuable because you can improve. This is the same for generative AI.

43. 8:32

    All right. Let me give you another example, a-another, um, unique complexity to the evaluation of generative AI. This is a two by four, and I gave myself the task of drilling a one-inch hole into the two by four all the way through it.

44. 8:50

    And when I talk, uh, to a live audience, I ask for a thumbs up, thumbs down, how did I do on this task? Right? Very similar to the types of feedback we often have for generative AI, uh, workloads.

45. 9:01

    And often I get mostly thumbs up. There's always that one guy that gives me a thumbs down. Um,

46. 9:07

    and what you're done, what you've done is evaluated my output. But if the methodology I used to, to create this hole looks like this,

47. 9:17

    right? Suddenly, maybe I didn't do such a good job. Maybe how I got there matters. And the reason is, even though I managed to make a good hole this time, if this is my setup, if this is the methodology I used to, to achieve it, or in the context of generative AI, if this is the reasoning I

48. 9:34

    took to create the hole, then maybe I need to, I need to rethink my system. So let me give you another example in the context of a real customer use case.

49. 9:43

    This is a meteorology company that was creating, um, summaries of local weather based on sensor data. So I've summarized what their prompt looked, just so it's easy to read here.

50. 9:54

    But essentially, they have a prompt that gives sensor data like it's raining and it's forty degrees and it's windy, and it's asking for a summary. And if the summary says res-- uh, today it's sunny and bright outside, all of a sudden it's like, hmm, something has gone wrong, right?

51. 10:11

    If we evaluate this, the score is zero. You know, this prompt gets an F, but we don't know why. Right? But what if we ask the model to explain its reasoning?

52. 10:19

    And its reasoning is, it's important to mental health to be happy, so I decided not to talk about the rain. Now that we've looked behind the scenes and we've seen kind of what the model, how the model got there, we suddenly have a lot more insight into how to fix the problem, right?

53. 10:32

    And you can imagine this in the context and why this is such a big problem in the context of evals. Let's look at the, uh, at the oppid-- opposite example.

54. 10:40

    If we give it data and it says it's sunny and then the response comes out and sunny, well, we'll say response is a hundr-- uh, ten out of ten.

55. 10:48

    Awesome. I've built a, a brilliant prompt and I can go to scale. And this is a danger because what happens is if you didn't eval the reasoning and this was the reasoning,

56. 10:59

    all of a sudden you're like, "Oh, okay, maybe my prompt isn't working right, because even though I got the right answer, I drilled the hole correctly in this one case," right?

57. 11:07

    The reasoning is, is, uh, not getting me where I need to be.

58. 11:12

    Okay. The last, um, the last step before I get into my t- my seven habits of generative AI, uh, evaluations is what I call, uh, prompt decomposition. And although this, uh, this technique is not specific to evaluations, it's often done in the context of evaluations.

59. 11:31

    Because when it comes to GenAI, you only can attach an eval to one prompt. It's difficult to evaluate the first half of a prompt versus the second half of a, of a prompt, right?

60. 11:44

    I think of it like a multimeter for my electrical engineers out there. You've got your two sensors. You put one sensor at the beginning of prompt, one sensor at the end of the prompt, and you're sensing what went on in the whole prompt.

61. 11:54

    Many times, if the prompt is very large and it does a lot of complicated things, it's difficult to understand where the error is because you just kind of get like, well, there's something going wrong in this big pile of code or big pile of prompting.

62. 12:07

    And so, um, this is, uh, that same weather company. This is an example from them where they were having to write a summary of today's weather. Inside their prompt, they did a lot of different instructions based on what the sensor data was.

63. 12:20

    And one of the things they noticed is that for a portion of their prompt, they actually had it say, okay, well, if the wind speed is like less than five, then it's not very windy, but if it's more than five, then it's windy.

64. 12:31

    And what was happening is in a, in the, at the POC stage, that worked great. When they tried to scale out, maybe two to three percent of the time, Claude was coming back and saying, "Oh, uh, the wind speed is seven.

65. 12:44

    Seven is less than five, so it's not windy." Right? And just completely messing up the math.

66. 12:50

    So what we did here is a series of prompt decompositions, and if you're interested in it, I've got a blog here you can check out. Um,

67. 12:58

    the basic idea is take the big prompt and break it into a se-- a chaining series of prompts. From an eval standpoint, that means that you can attach evals to each section of the prompt.

68. 13:11

    And what that allows you to do is decide, okay, well, this section's working great, this section isn't, so now I know where to focus my efforts. The other thing it does is allows you to decide, is generative AI even the right tool for that section of the prompt, right?

69. 13:25

    So in the case of this weather prompt, um, doing a s- a, a mathematical comparison, is seven larger than five? You don't need GenAI for that. Python is perfect at that.

70. 13:35

    It gets it accurate, perfectly accurate. And so in this series of chaining steps, one step was a Python mathematical comparison. When they made this change, the accuracy went to one hundred percent,

71. 13:46

    and they could tell the accuracy of each step. All right.

72. 13:51

    I will briefly mention, uh, as well in the context of that, uh, prompt decomposition, it's a very, very common pattern, and so I wanted to talk about its impact on evaluations.

73. 14:01

    One of the most common patterns we see is semantic routing. Some kind of query comes in or an input to a workload, and the first step is, well, what kind of task is this?

74. 14:11

    If it's a, it's an easy task, go to a small model. If it's a hard task, go to a large model. And by doing this, um, what I'm doing is using the right model for the job, not de- depending on the prompt itself, but depending on the input from the user and, and the complicatedness, the complication level

75. 14:30

    of the task from the user. And the interesting piece here is, again, attaching evals to each step. And so that means for, for an eval and a semantic router, often the input that you're evaluating is a query and the output is just the number one or the number two in this case.

76. 14:47

    And what you'll see when you run the evals is often even breaking this down significantly increases your accuracy because you're removing what I call dead space or dead tokens, right?

77. 14:57

    If this was all one big prompt, what you'd see is your instructions, your instructions for your easy stuff, your instructions for your hard stuff, right? And if an easy query comes in, it only needed those easy instructions.

78. 15:10

    Those hard instructions are basically dead space. They weren't necessary, but what they are is extra cost and an opportunity for the model to get confused. So often by breaking it down this way, we see, we see the evaluation showing the accuracy popping up because you're only sending the necessary information for the task.

79. 15:28

    All right, so let's jum- jump into the good stuff. I di- I didn't mean to keep you waiting, but here we are. These are the seven most common trends I see across all generative AI, uh, workloads that have successfully scaled.

80. 15:42

    I will say I have never seen a workload go to scale without evaluations, and most of them also include these, uh, seven habits. The first one is fast.

81. 15:51

    It is the case that I have seen evaluations built with the mindset that getting results within a week is okay.

82. 16:00

    What happens is I, I make a prompt, I push it to some kind of test site, I email a team to go test it, they test it whenever they feel like, they email me back whenever they feel like, and a week later, I get to make one more change to my prompt.

83. 16:12

    And you can imagine a team that's working on that mindset, that they get to have like four changes or eight changes a month versus a team that's managed to build an evaluation framework that operates in seconds, and they can make hundreds of changes every day and hundreds of tests every day, right?

84. 16:27

    The pace of innovation and the pace of increasing your accuracy is gonna be much, much faster, uh, to the point where I don't think I've ever seen a workload scale unless it's gone through many, many iterations.

85. 16:37

    And so sometimes just knowing that at the outset is helpful for teams, um,

86. 16:41

    rather than somehow you're gonna craft this beautiful, perfect prompt your first try and never change it again. Uh, I will say, and this is what we're gonna see a f- example of in, in just a minute, um, often, uh, my, my rule of thumb target is thirty seconds to run your evaluation framework.

87. 16:57

    And you do that by using generative AI as a judge to evaluate. Or if the, if the output's numeric, like from a semantic router where it's one, two, three, you're using Python to say, "Is this...

88. 17:08

    Did it output a one? Yes or no? Is one equal one? Yes. Okay, great. Good job." Um, the reason I say thirty seconds is often what it means is you spend ten seconds in parallel sending a hundred test cases for generation.

89. 17:21

    I've created a test. I've created a prompt template I wanna test. I run it across my hundred test cases and get results, ten seconds. Then I take those results, and in a hundred more parallel calls, I judge them.

90. 17:34

    So I take my results, compare them against my gold standard, and I, and I get a hundred judges going for ten seconds. And in the final ten seconds, I take my output from my hundred judges, and I summarize it.

91. 17:46

    Often that means breaking it down by categories and by what's right and what's wrong. And so what I see in my output is not a hundred generated out responses that now I have to go read through and figure junk out.

92. 17:57

    What I see in that last ten seconds is a summary where it says, "Hey, you know what? For all of your questions, all of your queries that had to do with taxes, when you got it right, here's what the trends were.

93. 18:06

    When you got it wrong, here's what the trends were." And again, it's going from that pers- from that mindset of how do I figure out what's going wrong and where the errors are and how to fix

94. 18:18

    Quantifiable, uh, is my second one. All, all effective frameworks also produce numbers, right? And I-- sometimes, depending on the teams, I get a lot of pushback on this because you're right that they're not always the exact same number every time you run the test, right?

95. 18:33

    If I turn in an essay, maybe I turned it in on Thursday and I get a score, or I turn it in on Friday. Friday, my professor has a few cups of, uh, uh, glasses of wine before grading, and so my score is a little better, right?

96. 18:45

    This happens in real life. It happens with GenAI evals. The way that we deal with it is the same in real life as it is with GenAI evals, which is we, we make it numerous, right?

97. 18:56

    Even if one-- if there's a little bit of jitter in the score, in the quantifiable score that we ask for, uh, if we have enough test cases and we average across those test cases, that jitter goes out.

98. 19:06

    Just like in grade school, hopefully you didn't turn in one assignment ever. Uh, you turned in a bunch of assignments and your sc-- your final score was the average of all of those.

99. 19:15

    The other reason why numerous is so important is that we wanna cover all of our-- we wanna be diverse and cover all of our, uh, test cases, uh, right?

100. 19:25

     So, um, oftentimes when I'm running this as an exercise with customers, even the, the, the exercise of building a hundred test cases is very valuable for the team to figure out what the scope of the project is.

101. 19:37

     Because inevitably someone will say, "Hey, when someone asks about taxes, we can, we can direct them to this site where they can get their W-2s." And then someone else will say, "No, no, no, if they ask about taxes, we can't respond at all because we're not allowed to," right?

102. 19:49

     And the team is debating the product design, nothing to do with GenAI, but it's a good exercise to understand what is the scope of what you wanna answer and, and make sure you have questions outside of what you wanna answer so that you can, you can measure that the, that the model is, is, uh, redirecting those that

103. 20:05

     are outside of scope. The next habit is explainable. This is that insight where it's like, don't just look at the outputs, look at how you got there, right? Look at how the model is reasoning.

104. 20:16

     And in particularly, uh, I said reasoning for generation and scoring, look at how your judge is reasoning as well for, for-- in, in some cases where you're asking the judge to do a lot of reasoning, because just like you're doing prompt engineering for your, for your, for your prompt for your users, you also need to eng-engineer the,

105. 20:33

     the judge prompt and make sure the judge is, is scoring correctly. Um, I'll s- I'll show a small example of that in just a minute, but the general idea is just like a, a professor grading a paper, right?

106. 20:44

     It's not necessarily that you just say, "Hey, give me a score," right? But the professor has a rubric. The professor says, "Okay, well, you need to be three pages long.

107. 20:53

     If it's not three pages long, 100% off. It needed to have five different sources cited. You didn't do that. If you only cited three, well, that's, uh, 20% off.

108. 21:01

     Uh, you needed a professional tone. Every time your tone slips out of professional, uh, I take 3% off your score." Right? And so you can see that the rules start building up, and you give really clear instructions on how the, how the, the output should be judged.

109. 21:16

     And asking your judge to explain its reasoning is often a good way to, uh, do prompt engineering for the judge.

110. 21:23

     The next is segmented. This is why I showed those slides on prompt decomposition. In practice, almost all scaled workloads are multiple steps, right? There's very, very few workloads I've ever seen that are a single prompt, right?

111. 21:35

     And what that means is you need to evaluate each step individually. And this actually is, is, uh, powerful for a number of reasons, not the least of which is that it's very rare for a workload to be a single model.

112. 21:47

     It's much more appropriate for each step. Because they're evaluated individually, we evaluate which model is appropriate. Many times as semantic router, you wanna use something like Nova Micro that's gonna give you an instant response.

113. 21:58

     It's very simple and easy. It's gonna give you one, two, three. Um, and so you can, you can, with an evaluation framework, prove to yourself the smallest model that's appropriate to use on each step.

114. 22:09

     Diverse is what I was mentioning earlier. You wanna cover all of your use cases. So sometimes I'll start by listing out what are the use cases in scope, and then make sure we have questions applying to all of them.

115. 22:18

     I will say the hundred is a rule of thumb. Sometimes if they're use cases that are like edge cases we don't really care about, I'll put in like three or four examples for that.

116. 22:25

     But the core use cases we want a lot of examples for. And then the last one I'll mention is traditional. Um, and I put this in there because there is a tendency to, uh, throw the baby out with the bathwater and say, "Hey, this is GenAI.

117. 22:38

     We're all gonna use GenAI. We're gonna GenAI everything." You know, the output of your My Semantic Router is one. I'll use GenAI to see if that one is really the number one, right?

118. 22:47

     And, uh, the answer is no. There are a lot of traditional techniques that are very, very powerful. Um, obviously, if the output is numeric, just, just use a nu-numeric evaluation.

119. 22:57

     Uh, for things like RAG Architecture is one of the most common. Um, there's many, many types of, uh, database accuracy evaluations, retrieval and precision and F1 scores, and all of those kinds of things for question answering.

120. 23:09

     Um, simple things like measuring cost and latency are still traditional tooling. So, uh, I just wanna-- I, I put that there to highlight the fact that traditional tooling is still very, very powerful and very important, uh, in the context of GenAI evaluations.

121. 23:23

     All right. I'm gonna give a quick example of what this looks like, uh, and, uh, just, uh, visually. We start with that gold standard set. Again, this is where I think the most important, uh, use of your time is, is, is building this.

122. 23:38

     Your whole system is gonna be pointed towards and designed towards this gold standard set. So if your gold standard set has a bunch of errors, what you've done is designed a whole system that does-- that creates errors, right?

123. 23:49

     Um, so, so invest. It's always worth the time investing in building a good gold standard set.

124. 23:57

     I will, uh, as a brief aside, mention this is a, a terrible place to use GenAI. If you use GenAI to create your gold standard set, if the GenAI set creates errors, you've built, you built a system that generates the same errors that the GenAI system has, uh, and it can lead to a lot of problems.

125. 24:14

     It can be helpful to generate a, what I w- I would call a silver standard set, which is the GenAI system taking a guess at a gold standard, but it still needs to be reviewed by a human to confirm its accuracy.

126. 24:25

     All right. We take an input from that gold standard set, one of those. We put it into our prompt template, put it into our LLM in order to generate an output.

127. 24:35

     We include the answer and the reasoning. Then we take the, uh, the, the matching answer from that gold standard input, uh, and we compare it with that generated output into a judge prompt, and the judge generates a number and the reasoning behind that number.

128. 24:54

     Um, and then we pull the category. Often I'll, I'll include the category in the gold standard set so that, um, we can then, as that final step, uh, break things down by category and, um, and generate a summary for the right and wrong answers, uh, for each category.

129. 25:17

     All right. That's it. Thank you so much. I hope this was helpful. Uh, please feel free to reach out with questions and comments, and enjoy your day.