AI Engineer World's Fair 2025
7 Habits of Highly Effective Generative AI Evaluations
Read the talk
Seven Habits of Effective Generative AI Evaluations
A convincing prototype can still hide failures. Effective evaluations expose where a workflow breaks, guide model selection, and turn each prompt change into actionable feedback.
From a talk by Justin Muller
Before you start: Familiarity with prompt templates, language-model calls, and basic Python will help; no prior evaluation-framework experience is required.
What keeps a promising prototype from scaling?
What is the biggest obstacle between a generative AI prototype people like and a system they can depend on? Justin Muller approaches that question from a small AWS specialist team helping customers scale workloads across industries and company sizes. Introducing himself as a principal applied AI architect, he describes a background spanning physics, mathematics, law, and business, with roughly fifteen years in natural language processing and four in generative AI. His team's practices come from seeing both successful projects and failures. Cost, hallucinations, accuracy, and capacity all matter, but the recurring missing piece he encounters is evaluations: teams have built something appealing without establishing how to diagnose its failures.
In July 2024, a document-processing customer called Muller into a project that was close to cancellation. He recalls roughly six to twelve months of work by six to eight engineers. The technology executive responsible for it reported 22% accuracy and wanted to know whether the project could be saved. Discovery exposed the central problem: the team had one end-to-end number, but no diagnostic evaluation framework showing where the process failed.
Once the framework made individual failures visible, Muller says the fixes were comparatively straightforward. The difficult part had been finding the problems and their causes. Muller reports that, over the following six months, the document-processing customer reached 92% accuracy by January, exceeding its 90% production launch threshold. The talk does not define the accuracy measure or test set. He describes the resulting deployment as AWS's largest document-processing workload in North America at that time.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Design for diagnosis, not just a score
Traditional machine-learning evaluation makes scores such as F1, precision, and recall familiar starting points. Generative AI evaluations also need to measure quality, but the first design goal should be discovering problems. An evaluator that identifies a failure—and perhaps uses generated analysis to suggest a correction—gives the team something to change. That goal changes the framework's design: instead of asking only how well the system performs, ask what evidence will reveal where it goes wrong.
Willingness to do that work also becomes a project-selection signal. Muller's small team cannot support every request. When a team asks for multi-agent collaboration, he proposes spending two hours assembling a gold-standard evaluation set first. Some teams want to skip that and start experimenting with the tools; others offer four hours because getting the project right matters. Exploration is useful for learning, but Muller distinguishes it from the commitment needed to scale. His stories of unusually successful customers emphasize the same behavior: they invest in evaluations rather than treating them as a distraction.
Free text does not make evaluation impossible. Consider a professor assigning a three-page essay on the emotional meaning of sunshine. There is no single mathematically exact answer, yet the professor can still assess it. The difference between useful and useless assessment is feedback: an unexplained F tells the student little, while comments identifying mistakes and opportunities to improve make the next draft better. A generative AI evaluator should behave like that second professor.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A correct answer can hide a faulty method
Muller illustrates the limitation of output-only feedback with a one-inch hole drilled through a two-by-four. Asked to judge the finished hole, audiences mostly give a thumbs-up. He then reveals the method used to make it, changing the assessment: producing an acceptable hole once does not establish that the process is dependable. Thumbs-up feedback on a generated answer has the same blind spot.
A meteorology customer's task makes the problem concrete. Its prompt asks for a local weather summary from sensor data: rain, a temperature of 40 degrees, and wind. A response describing sunny, bright weather is clearly wrong. A failing score identifies the mismatch, but an explanation can reveal a more specific problem. In Muller's example, the model says it omitted rain because happiness is important to mental health. That stated rationale points toward a conflict between reporting the observations and trying to make the reader feel good. Treat such explanations as diagnostic evidence, not a guaranteed account of the model's internal causal process.
Now change the input to sunny weather. The same preference for cheerful answers can produce an apparently perfect response while leaving the underlying defect intact.
| Sensor input | Generated summary | Output-only assessment | What the explanation exposes |
|---|---|---|---|
| Rainy | Sunny | Wrong | Happiness displaces sensor evidence |
| Sunny | Sunny | Correct | The same defective priority remains |
The second case is more dangerous precisely because the answer looks right. Inspecting the explanation gives the team a reason to test whether the prompt follows the evidence when the weather is less pleasant.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Break the prompt where you need to measure
A large prompt creates a measurement problem. Muller compares evaluating a model call to placing a multimeter's probes at its input and output: you can observe the behavior across the whole operation, but that does not isolate the tasks inside it. If one prompt interprets sensor data, applies thresholds, chooses wording, and writes a summary, an incorrect answer leaves several possible causes. Prompt decomposition creates intermediate outputs that can be evaluated separately.
The weather customer's prompt included a wind-speed rule: below five meant not very windy; above five meant windy. It worked in the prototype, but Muller reports that Claude incorrectly treated seven as less than five about 2–3% of the time as the weather workload scaled. The Claude version, test size, and wind-speed units are unspecified. The example shows why a plausible prototype result is insufficient evidence for a deterministic comparison embedded in a language task.
The team split the large prompt into a chain of smaller operations. Each boundary became an evaluation point, allowing them to identify which parts worked and where to focus. Decomposition also raised a more useful question than how to improve every prompt: does this step need a generative model at all? Comparing seven with five belongs in Python. Muller also points readers toward his writing on decomposition; his announcement of the Prompt Decomposition article provides a related starting point.
The comparison can be isolated directly. This implementation leaves equality unresolved because the described rule specifies only values below and above five:
python
def classify_wind(speed: float) -> str:
if speed < 5:
return "not very windy"
if speed > 5:
return "windy"
raise ValueError("Define the wind classification at exactly 5")
wind_speed = 7
wind_description = classify_wind(wind_speed)
assert wind_description == "windy"
The downstream generation step can now receive wind_description instead of being responsible for calculating it. Muller reports 100% accuracy after replacing the mathematical comparison with Python in this weather example. He does not specify the measurement scope or sample size; the useful architectural result is that the team could inspect the accuracy of each step independently.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Route tasks without carrying unused instructions
Semantic routing applies the same decomposition pattern to incoming requests. First classify the task, then send an easy request to a small model and a hard request to a larger one. The routing decision depends on the actual input's difficulty. The router itself has a compact evaluation contract: a query goes in, and an identifier such as 1 or 2 comes out. Its correctness can be assessed separately from the quality of the eventual answer.
This separation can also remove irrelevant prompt content. A single large prompt may contain instructions for both easy and hard tasks, even when a particular request needs only one branch. Muller calls those unused instructions dead space or dead tokens: they add cost and create opportunities for confusion. The slide makes the branches concrete with Nova Micro routing to Nova Lite for an easy task or Nova Premier for a hard one, then shades the hard-task branch as unused for the easy task. Muller attributes observed accuracy improvements in this pattern to sending only the instructions needed for the selected task.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Habit 1: Make feedback fast enough to use
The seven habits describe patterns Muller sees in workloads that successfully scale. He says he has never seen one scale without evaluations. The first habit, fast, determines how much learning a team can do. In a slow workflow, an engineer changes a prompt, publishes it to a test site, emails reviewers, and waits for replies. A week can pass before the next change. An evaluation framework that returns results in seconds lets the team test repeatedly while the problem is still fresh, rather than hoping to write the perfect prompt on the first attempt.
Muller's rule-of-thumb target is a 30-second evaluation loop: roughly ten seconds each for generation, judging, and summarization across about 100 test cases. It is a design target, not a measured runtime guarantee. The proposed sequence is:
- Generate: Apply the candidate prompt template to the test inputs, sending approximately 100 generation requests in parallel.
- Judge: Compare each result with its gold-standard answer, again using parallel calls where a model judge is appropriate. For a numeric route identifier, use an ordinary equality check instead.
- Summarize: Aggregate the judgments into an explanation of what worked and what failed.
Parallelism makes the first two phases suitable for a short feedback loop; the last phase makes the output usable.
The developer should not receive a pile of 100 responses to read manually. Group the results by category and separate correct from incorrect cases. For tax questions, for example, the summary should explain the patterns shared by successful answers and the patterns behind failures. That returns the framework to its primary job: finding the errors and helping the team decide what to change.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Habits 2 and 3: Quantifiable and numerous
Quantifiable means the framework produces numbers even when judgment is not perfectly repeatable. Muller returns to the professor analogy: an essay might receive a slightly different grade on Thursday than on Friday after the professor has had some wine. Variation does not make grading useless. The companion habit, numerous, is to evaluate enough cases that the aggregate is less dependent on any one judgment. Averaging can reduce random fluctuation; it does not remove systematic grading errors.
A larger test set also helps define the product. In Muller's customer exercises, building around 100 cases regularly exposes disagreements that prompt engineering cannot resolve. One teammate expects the assistant to direct tax questions to a site containing W-2 forms. Another believes the assistant must not answer tax questions at all. The team must settle that policy before a judge can score the behavior consistently. Include questions outside the intended scope, too, so the evaluation measures whether the system redirects them appropriately.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Habit 4: Make generation and judging explainable
Explainable applies to both sides of the evaluation. Inspect the generator's explanation, but also ask the judge to explain its score. The judge prompt is itself an engineered component: it can misunderstand the task, apply the wrong criteria, or penalize the wrong behavior. Its explanations help reveal those defects, especially when scoring requires substantial judgment.
A rubric gives the evaluator specific rules instead of an open-ended request to assign a score. Muller's deliberately strict essay example makes the rules visible:
| Criterion | Illustrative penalty |
|---|---|
| Three-page requirement unmet | 100% off |
| Three sources cited when five are required | 20% off |
| Each lapse in professional tone | 3% off |
The particular penalties are less important than making the criteria explicit. A judge's explanation should show which rules it applied, giving the team evidence for improving the judge prompt as well as the generation prompt.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Habits 5 and 6: Segmented and diverse
Segmented evaluations attach to individual workflow steps. Muller describes scaled systems as usually involving multiple steps and often multiple models. Evaluating each step separately makes model selection an empirical decision: determine the smallest model that satisfies that step's requirements. A simple semantic router returning 1, 2, or 3 is his example of a task suited to a small, fast model such as Nova Micro. The Amazon Nova launch announcement describes Micro's low-latency positioning; the workflow's own evaluations must establish whether it is adequate for the routing task.
Diverse means covering the intended range of behavior, rather than collecting many variations of one easy case. Start by listing the in-scope use cases, then assign test questions to each. Muller's approximately 100 cases are a heuristic. He might allocate three or four examples to a low-priority edge case and many more to a core use case. The distribution should reflect what the product needs to do reliably.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Habit 7: Keep traditional checks
Traditional evaluation tools remain useful inside generative AI systems. There is no reason to ask another model whether a router's output of 1 is the expected number 1. Choose the measurement that directly fits the operation:
- Routing and numeric outputs: Use deterministic comparisons against expected values.
- Retrieval and question answering: Retain appropriate retrieval, database-accuracy, precision, and F1 evaluations.
- Operational behavior: Measure cost and latency with conventional instrumentation.
A model judge is one tool in the framework, not a replacement for every existing test.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From trustworthy references to useful feedback
The complete workflow begins with the gold-standard set. It deserves substantial attention because the rest of the system will be optimized against it. If the reference answers contain errors, improving the evaluation score can mean teaching the system to reproduce those errors.
Muller warns against accepting model-generated answers as gold. A model can help propose a silver-standard set, but a human must review those candidates and confirm their accuracy before they become trusted references. Generating more examples does not substitute for deciding whether their expected answers are right.
With reviewed references in place, the evaluation follows a concrete data flow:
- Select a reference input. Insert it into the prompt template and send it to the generation model.
- Collect the generated output. Retain the answer and its accompanying explanation.
- Judge against the matching reference. Supply the generated output and gold-standard answer to the judge prompt. Request a numerical score and an explanation of that score.
- Group the judgments. Carry a category label from the reference set into the results, then summarize the correct and incorrect answers within each category.
The final diagram shows why these fields matter. Its example includes the category Sunny Day and a judge explanation distinguishing correct weather content from an incorrect tone. That is the useful endpoint of the framework: a score connected to a specific defect, organized so the team can decide what to fix next.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Historical introduction to Amazon Nova, including the text-only Nova Micro model used as the talk's example of a small semantic router.
Further reading
Anthropic experiments examining when generated reasoning fails to disclose information that influenced an answer.
Updates since the talk
Muller's later discussion of reference-answer evaluation and rubric-based judging. The public introduction defines human-reviewed gold data; the full article requires membership.
Read the complete timestamped transcript
- 0:05
Hello. Welcome to my talk on the seven habits of highly effective generative AI evaluations.
- 0:13
I'm Justin Muller. I'm a principal applied AI architect at AWS. I have degrees in physics, math, law, and business. I've been working in natural language processing for about fifteen years on and off, and generative AI for the last four years.
- 0:28
Uh, inside of AWS, my team is a small specialist team that helps customers scale GenAI workloads. So I've had the opportunity to work on many, many workloads across all different industries, all different sizes of customers.
- 0:40
Some small workloads, some of the largest in North America. Um, I'll also mention that because of that experience, I've seen a lot of workloads fail, and I've seen a lot of workloads succeed.
- 0:51
And so one of the things my team does now is that we've gathered up a lot of the best practices we've seen across the successful workloads, as well as some of the common failure points we've seen on the failed workloads and share them as best practices in talks like this one.
- 1:06
All right, so let's get going. The first question is, uh, what is the biggest challenge in scaling generative AI? You may, you may be able to guess the solution here, as this is what the talk's all about.
- 1:18
Um, but I, I would argue that it's evaluations.
- 1:23
I- in, in my experience, having been called in to help many different customers scale their GenAI workloads, there's lots of concerns that come up. And when I ask this question with a live audience, there's things like cost, hallucinations, accuracy, capacity, uh, all these different types of, um, concerns come up.
- 1:42
But the number one thing that I see across all workloads is a lack of evaluations. And in particular, I call it the missing piece to scaling GenAI because a lot of times when a customer has, uh, successfully built a, a POC and they have a, uh, a workload that they like or chatbot that seems cool, and they're
- 1:59
figuring out how to scale it, um, this is the piece that I come in, and most often it's missing. And when we add evaluations, it unlocks the ability to scale, and we're gonna look at how that happens.
- 2:13
But it's by far the most common way to unlock scale.
- 2:17
All right, I'll give you a quick customer example. Um, in, uh, July 2024, I was called in to-- as a, as like an escalation on a workload where a customer was doing document processing, and they'd been working on this document processing workload for about, I think six months, twelve months, maybe.
- 2:34
Um, and they had six or eight engineers working on it. So it was a pretty big project at that point. And the, uh, the, uh, the VP of technology that was in charge of the project called me in.
- 2:45
He asked everyone else to leave the room, and he was like: "Look, this project, we spent a bunch of money on it. Accuracy is at twenty-two percent.
- 2:53
We're thinking about just cutting the project. We're not getting the results we need. So I need you to tell me if we can save this thing." And, uh, essentially, I spent a couple of weeks doing discovery and understanding what was going on in the workload.
- 3:06
And the number one thing that I realized was they had zero evaluations, right? They had this big process, and end to end, they had a single number coming out the end that said, well, it's twenty-two percent accurate.
- 3:16
So what I did is design a evaluations framework, and the results were almost funny because once the evaluation framework was in place and you could see exactly where the problems were, fixing them were trivial, right?
- 3:28
Fixing the problems wasn't really the challenge. Increasing accuracy wasn't really the challenge. It was knowing where the problems were and what was causing them. So over the course of the next six months, they built the evaluation framework.
- 3:39
They fixed, uh, many of the issues that came up. They were able to achieve ninety-two accurate-- percent accuracy by January, uh, when, uh, which ninety percent was their threshold for launching to, to production at scale.
- 3:52
And they launched and became the single largest document processing workload, uh, on A-AWS in North America at the time.
- 4:01
All right, so evaluations are really, really important. What are they? How do we actually build these things? The-- this is the place I like to start with evaluations because m-many times, especially if you're coming from a traditional AI/ML back- uh, background, you look at evaluations as a way to measure quality
- 4:21
because you're, you're familiar with things like, well, what's your F1 score? What's your precision and recall? And you get a score, and it just tells you how well you're doing.
- 4:30
And while, uh, generative AI evaluations do produce a score, if anything, it's a, it's a far third of the top reasons why we do evaluations with GenAI workloads.
- 4:41
The main goal with any evaluation framework should be to discover problems. Similar to the workload example that I just shared, if your evaluation framework tells you where the problems are and perhaps even suggests solutions because it includes an element of generative AI reasoning, now we can improve our workloads.
- 4:57
And of course, yes, we need to measure quality. That's very important. But I, I start this way because when you think about designing an evaluation framework, if you come in with the mindset that it's gonna find errors, you design it in a very different way than if you come in and, and with the mindset that it's just
- 5:14
gonna measure how well you're doing. All right? So let's double-click on what that looks like.
- 5:21
Um, and, and before we do that, though, I do wanna, I would-- I do wanna add a second aside on why I think evaluations are so por- important. Um,
- 5:31
my team within AWS is a very small specialist team, and so we have to have very, very strict filters on which projects we say yes to and which ones we support.
- 5:40
And my experience has been evaluations are the number one filter that separates a science project from a successful project. When I talk to a team and the team says, "Can you teach me about multi-agent collaboration?"
- 5:55
And I say, "Great. Let's sit down and spend two hours putting together a gold standard set for our evaluations so that your multi-agent collaboration framework project is successful." If the team comes back and says, "Oh gosh Two hours on eval sounds boring.
- 6:10
Can you just give me the toys to play with? I, I know right away that's a science project, and it's not gonna go anywhere. And, and science projects are important.
- 6:18
They're a fun way to learn. They're just not what goes to scale. On the other hand, for my projects that have been wildly successful, where customers have had a hundred x return on investment or maybe cut their costs by ten thousand percent or something like that, those customers, when I say, "Hey, can we spend two hours, uh,
- 6:34
building an eval framework?" They say, "Why don't we spend, spend four?" Right? Evals are so important, and we recognize that, and this project is so important that we're gonna invest the time.
- 6:44
So I, as an aside, I'll just mention it as my number one filter for deciding which projects are gonna be successful and which are worth spending time on.
- 6:52
All right. So evals in the generative AI world come with baggage, right? And especially if you have a traditional AI/ML background, it can be a little scary, right? You, you look at evals and it's free text coming out, and you're like, "Oh my gosh, it's not gonna be a specific number.
- 7:07
I can't mathematically calculate an exact number every time. Let's panic."
- 7:12
Right? And it turns out that the human race has been grading and evaluating, uh, free text workloads for, uh, hundreds of years, if not thousands, right? Um, can you imagine if my professor in English assigns me an essay and says, uh, "Write me a three-page essay on the, on the emotional meaning of sunshine."
- 7:37
And I, I spend an, I spend my hours and I write my essay and I turn it in. The professor says, "Well, I can't calculate an exact answer to this.
- 7:44
I can't calculate your exact score, so I give up. I'm just not gonna score it." No, of course not. The professor's gonna give it a score. And we can evaluate generative AI in the same way that we've been evaluating free text for, for decades and generations, right?
- 7:59
But the key understanding is if you have a bad professor, they just give you a score. Maybe you've had this experience like me, you've turned in an essay and got an F, right?
- 8:07
If you just get a score, you're like, "Well, that, that stinks, but what do I do with it?" Right? With generative AI, we can go a bit deeper. We can do what the good professors do, which is point out what you did wrong, point out where you can improve, right?
- 8:21
This is why evaluations in the term, in the, in the context of grade school are very, very valuable because you can improve. This is the same for generative AI.
- 8:32
All right. Let me give you another example, a-another, um, unique complexity to the evaluation of generative AI. This is a two by four, and I gave myself the task of drilling a one-inch hole into the two by four all the way through it.
- 8:50
And when I talk, uh, to a live audience, I ask for a thumbs up, thumbs down, how did I do on this task? Right? Very similar to the types of feedback we often have for generative AI, uh, workloads.
- 9:01
And often I get mostly thumbs up. There's always that one guy that gives me a thumbs down. Um,
- 9:07
and what you're done, what you've done is evaluated my output. But if the methodology I used to, to create this hole looks like this,
- 9:17
right? Suddenly, maybe I didn't do such a good job. Maybe how I got there matters. And the reason is, even though I managed to make a good hole this time, if this is my setup, if this is the methodology I used to, to achieve it, or in the context of generative AI, if this is the reasoning I
- 9:34
took to create the hole, then maybe I need to, I need to rethink my system. So let me give you another example in the context of a real customer use case.
- 9:43
This is a meteorology company that was creating, um, summaries of local weather based on sensor data. So I've summarized what their prompt looked, just so it's easy to read here.
- 9:54
But essentially, they have a prompt that gives sensor data like it's raining and it's forty degrees and it's windy, and it's asking for a summary. And if the summary says res-- uh, today it's sunny and bright outside, all of a sudden it's like, hmm, something has gone wrong, right?
- 10:11
If we evaluate this, the score is zero. You know, this prompt gets an F, but we don't know why. Right? But what if we ask the model to explain its reasoning?
- 10:19
And its reasoning is, it's important to mental health to be happy, so I decided not to talk about the rain. Now that we've looked behind the scenes and we've seen kind of what the model, how the model got there, we suddenly have a lot more insight into how to fix the problem, right?
- 10:32
And you can imagine this in the context and why this is such a big problem in the context of evals. Let's look at the, uh, at the oppid-- opposite example.
- 10:40
If we give it data and it says it's sunny and then the response comes out and sunny, well, we'll say response is a hundr-- uh, ten out of ten.
- 10:48
Awesome. I've built a, a brilliant prompt and I can go to scale. And this is a danger because what happens is if you didn't eval the reasoning and this was the reasoning,
- 10:59
all of a sudden you're like, "Oh, okay, maybe my prompt isn't working right, because even though I got the right answer, I drilled the hole correctly in this one case," right?
- 11:07
The reasoning is, is, uh, not getting me where I need to be.
- 11:12
Okay. The last, um, the last step before I get into my t- my seven habits of generative AI, uh, evaluations is what I call, uh, prompt decomposition. And although this, uh, this technique is not specific to evaluations, it's often done in the context of evaluations.
- 11:31
Because when it comes to GenAI, you only can attach an eval to one prompt. It's difficult to evaluate the first half of a prompt versus the second half of a, of a prompt, right?
- 11:44
I think of it like a multimeter for my electrical engineers out there. You've got your two sensors. You put one sensor at the beginning of prompt, one sensor at the end of the prompt, and you're sensing what went on in the whole prompt.
- 11:54
Many times, if the prompt is very large and it does a lot of complicated things, it's difficult to understand where the error is because you just kind of get like, well, there's something going wrong in this big pile of code or big pile of prompting.
- 12:07
And so, um, this is, uh, that same weather company. This is an example from them where they were having to write a summary of today's weather. Inside their prompt, they did a lot of different instructions based on what the sensor data was.
- 12:20
And one of the things they noticed is that for a portion of their prompt, they actually had it say, okay, well, if the wind speed is like less than five, then it's not very windy, but if it's more than five, then it's windy.
- 12:31
And what was happening is in a, in the, at the POC stage, that worked great. When they tried to scale out, maybe two to three percent of the time, Claude was coming back and saying, "Oh, uh, the wind speed is seven.
- 12:44
Seven is less than five, so it's not windy." Right? And just completely messing up the math.
- 12:50
So what we did here is a series of prompt decompositions, and if you're interested in it, I've got a blog here you can check out. Um,
- 12:58
the basic idea is take the big prompt and break it into a se-- a chaining series of prompts. From an eval standpoint, that means that you can attach evals to each section of the prompt.
- 13:11
And what that allows you to do is decide, okay, well, this section's working great, this section isn't, so now I know where to focus my efforts. The other thing it does is allows you to decide, is generative AI even the right tool for that section of the prompt, right?
- 13:25
So in the case of this weather prompt, um, doing a s- a, a mathematical comparison, is seven larger than five? You don't need GenAI for that. Python is perfect at that.
- 13:35
It gets it accurate, perfectly accurate. And so in this series of chaining steps, one step was a Python mathematical comparison. When they made this change, the accuracy went to one hundred percent,
- 13:46
and they could tell the accuracy of each step. All right.
- 13:51
I will briefly mention, uh, as well in the context of that, uh, prompt decomposition, it's a very, very common pattern, and so I wanted to talk about its impact on evaluations.
- 14:01
One of the most common patterns we see is semantic routing. Some kind of query comes in or an input to a workload, and the first step is, well, what kind of task is this?
- 14:11
If it's a, it's an easy task, go to a small model. If it's a hard task, go to a large model. And by doing this, um, what I'm doing is using the right model for the job, not de- depending on the prompt itself, but depending on the input from the user and, and the complicatedness, the complication level
- 14:30
of the task from the user. And the interesting piece here is, again, attaching evals to each step. And so that means for, for an eval and a semantic router, often the input that you're evaluating is a query and the output is just the number one or the number two in this case.
- 14:47
And what you'll see when you run the evals is often even breaking this down significantly increases your accuracy because you're removing what I call dead space or dead tokens, right?
- 14:57
If this was all one big prompt, what you'd see is your instructions, your instructions for your easy stuff, your instructions for your hard stuff, right? And if an easy query comes in, it only needed those easy instructions.
- 15:10
Those hard instructions are basically dead space. They weren't necessary, but what they are is extra cost and an opportunity for the model to get confused. So often by breaking it down this way, we see, we see the evaluation showing the accuracy popping up because you're only sending the necessary information for the task.
- 15:28
All right, so let's jum- jump into the good stuff. I di- I didn't mean to keep you waiting, but here we are. These are the seven most common trends I see across all generative AI, uh, workloads that have successfully scaled.
- 15:42
I will say I have never seen a workload go to scale without evaluations, and most of them also include these, uh, seven habits. The first one is fast.
- 15:51
It is the case that I have seen evaluations built with the mindset that getting results within a week is okay.
- 16:00
What happens is I, I make a prompt, I push it to some kind of test site, I email a team to go test it, they test it whenever they feel like, they email me back whenever they feel like, and a week later, I get to make one more change to my prompt.
- 16:12
And you can imagine a team that's working on that mindset, that they get to have like four changes or eight changes a month versus a team that's managed to build an evaluation framework that operates in seconds, and they can make hundreds of changes every day and hundreds of tests every day, right?
- 16:27
The pace of innovation and the pace of increasing your accuracy is gonna be much, much faster, uh, to the point where I don't think I've ever seen a workload scale unless it's gone through many, many iterations.
- 16:37
And so sometimes just knowing that at the outset is helpful for teams, um,
- 16:41
rather than somehow you're gonna craft this beautiful, perfect prompt your first try and never change it again. Uh, I will say, and this is what we're gonna see a f- example of in, in just a minute, um, often, uh, my, my rule of thumb target is thirty seconds to run your evaluation framework.
- 16:57
And you do that by using generative AI as a judge to evaluate. Or if the, if the output's numeric, like from a semantic router where it's one, two, three, you're using Python to say, "Is this...
- 17:08
Did it output a one? Yes or no? Is one equal one? Yes. Okay, great. Good job." Um, the reason I say thirty seconds is often what it means is you spend ten seconds in parallel sending a hundred test cases for generation.
- 17:21
I've created a test. I've created a prompt template I wanna test. I run it across my hundred test cases and get results, ten seconds. Then I take those results, and in a hundred more parallel calls, I judge them.
- 17:34
So I take my results, compare them against my gold standard, and I, and I get a hundred judges going for ten seconds. And in the final ten seconds, I take my output from my hundred judges, and I summarize it.
- 17:46
Often that means breaking it down by categories and by what's right and what's wrong. And so what I see in my output is not a hundred generated out responses that now I have to go read through and figure junk out.
- 17:57
What I see in that last ten seconds is a summary where it says, "Hey, you know what? For all of your questions, all of your queries that had to do with taxes, when you got it right, here's what the trends were.
- 18:06
When you got it wrong, here's what the trends were." And again, it's going from that pers- from that mindset of how do I figure out what's going wrong and where the errors are and how to fix
- 18:18
Quantifiable, uh, is my second one. All, all effective frameworks also produce numbers, right? And I-- sometimes, depending on the teams, I get a lot of pushback on this because you're right that they're not always the exact same number every time you run the test, right?
- 18:33
If I turn in an essay, maybe I turned it in on Thursday and I get a score, or I turn it in on Friday. Friday, my professor has a few cups of, uh, uh, glasses of wine before grading, and so my score is a little better, right?
- 18:45
This happens in real life. It happens with GenAI evals. The way that we deal with it is the same in real life as it is with GenAI evals, which is we, we make it numerous, right?
- 18:56
Even if one-- if there's a little bit of jitter in the score, in the quantifiable score that we ask for, uh, if we have enough test cases and we average across those test cases, that jitter goes out.
- 19:06
Just like in grade school, hopefully you didn't turn in one assignment ever. Uh, you turned in a bunch of assignments and your sc-- your final score was the average of all of those.
- 19:15
The other reason why numerous is so important is that we wanna cover all of our-- we wanna be diverse and cover all of our, uh, test cases, uh, right?
- 19:25
So, um, oftentimes when I'm running this as an exercise with customers, even the, the, the exercise of building a hundred test cases is very valuable for the team to figure out what the scope of the project is.
- 19:37
Because inevitably someone will say, "Hey, when someone asks about taxes, we can, we can direct them to this site where they can get their W-2s." And then someone else will say, "No, no, no, if they ask about taxes, we can't respond at all because we're not allowed to," right?
- 19:49
And the team is debating the product design, nothing to do with GenAI, but it's a good exercise to understand what is the scope of what you wanna answer and, and make sure you have questions outside of what you wanna answer so that you can, you can measure that the, that the model is, is, uh, redirecting those that
- 20:05
are outside of scope. The next habit is explainable. This is that insight where it's like, don't just look at the outputs, look at how you got there, right? Look at how the model is reasoning.
- 20:16
And in particularly, uh, I said reasoning for generation and scoring, look at how your judge is reasoning as well for, for-- in, in some cases where you're asking the judge to do a lot of reasoning, because just like you're doing prompt engineering for your, for your, for your prompt for your users, you also need to eng-engineer the,
- 20:33
the judge prompt and make sure the judge is, is scoring correctly. Um, I'll s- I'll show a small example of that in just a minute, but the general idea is just like a, a professor grading a paper, right?
- 20:44
It's not necessarily that you just say, "Hey, give me a score," right? But the professor has a rubric. The professor says, "Okay, well, you need to be three pages long.
- 20:53
If it's not three pages long, 100% off. It needed to have five different sources cited. You didn't do that. If you only cited three, well, that's, uh, 20% off.
- 21:01
Uh, you needed a professional tone. Every time your tone slips out of professional, uh, I take 3% off your score." Right? And so you can see that the rules start building up, and you give really clear instructions on how the, how the, the output should be judged.
- 21:16
And asking your judge to explain its reasoning is often a good way to, uh, do prompt engineering for the judge.
- 21:23
The next is segmented. This is why I showed those slides on prompt decomposition. In practice, almost all scaled workloads are multiple steps, right? There's very, very few workloads I've ever seen that are a single prompt, right?
- 21:35
And what that means is you need to evaluate each step individually. And this actually is, is, uh, powerful for a number of reasons, not the least of which is that it's very rare for a workload to be a single model.
- 21:47
It's much more appropriate for each step. Because they're evaluated individually, we evaluate which model is appropriate. Many times as semantic router, you wanna use something like Nova Micro that's gonna give you an instant response.
- 21:58
It's very simple and easy. It's gonna give you one, two, three. Um, and so you can, you can, with an evaluation framework, prove to yourself the smallest model that's appropriate to use on each step.
- 22:09
Diverse is what I was mentioning earlier. You wanna cover all of your use cases. So sometimes I'll start by listing out what are the use cases in scope, and then make sure we have questions applying to all of them.
- 22:18
I will say the hundred is a rule of thumb. Sometimes if they're use cases that are like edge cases we don't really care about, I'll put in like three or four examples for that.
- 22:25
But the core use cases we want a lot of examples for. And then the last one I'll mention is traditional. Um, and I put this in there because there is a tendency to, uh, throw the baby out with the bathwater and say, "Hey, this is GenAI.
- 22:38
We're all gonna use GenAI. We're gonna GenAI everything." You know, the output of your My Semantic Router is one. I'll use GenAI to see if that one is really the number one, right?
- 22:47
And, uh, the answer is no. There are a lot of traditional techniques that are very, very powerful. Um, obviously, if the output is numeric, just, just use a nu-numeric evaluation.
- 22:57
Uh, for things like RAG Architecture is one of the most common. Um, there's many, many types of, uh, database accuracy evaluations, retrieval and precision and F1 scores, and all of those kinds of things for question answering.
- 23:09
Um, simple things like measuring cost and latency are still traditional tooling. So, uh, I just wanna-- I, I put that there to highlight the fact that traditional tooling is still very, very powerful and very important, uh, in the context of GenAI evaluations.
- 23:23
All right. I'm gonna give a quick example of what this looks like, uh, and, uh, just, uh, visually. We start with that gold standard set. Again, this is where I think the most important, uh, use of your time is, is, is building this.
- 23:38
Your whole system is gonna be pointed towards and designed towards this gold standard set. So if your gold standard set has a bunch of errors, what you've done is designed a whole system that does-- that creates errors, right?
- 23:49
Um, so, so invest. It's always worth the time investing in building a good gold standard set.
- 23:57
I will, uh, as a brief aside, mention this is a, a terrible place to use GenAI. If you use GenAI to create your gold standard set, if the GenAI set creates errors, you've built, you built a system that generates the same errors that the GenAI system has, uh, and it can lead to a lot of problems.
- 24:14
It can be helpful to generate a, what I w- I would call a silver standard set, which is the GenAI system taking a guess at a gold standard, but it still needs to be reviewed by a human to confirm its accuracy.
- 24:25
All right. We take an input from that gold standard set, one of those. We put it into our prompt template, put it into our LLM in order to generate an output.
- 24:35
We include the answer and the reasoning. Then we take the, uh, the, the matching answer from that gold standard input, uh, and we compare it with that generated output into a judge prompt, and the judge generates a number and the reasoning behind that number.
- 24:54
Um, and then we pull the category. Often I'll, I'll include the category in the gold standard set so that, um, we can then, as that final step, uh, break things down by category and, um, and generate a summary for the right and wrong answers, uh, for each category.
- 25:17
All right. That's it. Thank you so much. I hope this was helpful. Uh, please feel free to reach out with questions and comments, and enjoy your day.