7 Habits of Highly Effective Generative AI Evaluations

What is the biggest obstacle between a generative AI prototype people like and a system they can depend on? Justin Muller approaches that question from a small AWS specialist team helping customers scale workloads across industries and company sizes. Introducing himself as a principal applied AI architect, he describes a background spanning physics, mathematics, law, and business, with roughly fifteen years in natural language processing and four in generative AI. His team's practices come from seeing both successful projects and failures. Cost, hallucinations, accuracy, and capacity all matter, but the recurring missing piece he encounters is evaluations: teams have built something appealing without establishing how to diagnose its failures.

In July 2024, a document-processing customer called Muller into a project that was close to cancellation. He recalls roughly six to twelve months of work by six to eight engineers. The technology executive responsible for it reported 22% accuracy and wanted to know whether the project could be saved. Discovery exposed the central problem: the team had one end-to-end number, but no diagnostic evaluation framework showing where the process failed.

Once the framework made individual failures visible, Muller says the fixes were comparatively straightforward. The difficult part had been finding the problems and their causes. Muller reports that, over the following six months, the document-processing customer reached 92% accuracy by January, exceeding its 90% production launch threshold. The talk does not define the accuracy measure or test set. He describes the resulting deployment as AWS's largest document-processing workload in North America at that time.

Traditional machine-learning evaluation makes scores such as F1, precision, and recall familiar starting points. Generative AI evaluations also need to measure quality, but the first design goal should be discovering problems. An evaluator that identifies a failure—and perhaps uses generated analysis to suggest a correction—gives the team something to change. That goal changes the framework's design: instead of asking only how well the system performs, ask what evidence will reveal where it goes wrong.

Willingness to do that work also becomes a project-selection signal. Muller's small team cannot support every request. When a team asks for multi-agent collaboration, he proposes spending two hours assembling a gold-standard evaluation set first. Some teams want to skip that and start experimenting with the tools; others offer four hours because getting the project right matters. Exploration is useful for learning, but Muller distinguishes it from the commitment needed to scale. His stories of unusually successful customers emphasize the same behavior: they invest in evaluations rather than treating them as a distraction.

Free text does not make evaluation impossible. Consider a professor assigning a three-page essay on the emotional meaning of sunshine. There is no single mathematically exact answer, yet the professor can still assess it. The difference between useful and useless assessment is feedback: an unexplained F tells the student little, while comments identifying mistakes and opportunities to improve make the next draft better. A generative AI evaluator should behave like that second professor.

Muller illustrates the limitation of output-only feedback with a one-inch hole drilled through a two-by-four. Asked to judge the finished hole, audiences mostly give a thumbs-up. He then reveals the method used to make it, changing the assessment: producing an acceptable hole once does not establish that the process is dependable. Thumbs-up feedback on a generated answer has the same blind spot.

A meteorology customer's task makes the problem concrete. Its prompt asks for a local weather summary from sensor data: rain, a temperature of 40 degrees, and wind. A response describing sunny, bright weather is clearly wrong. A failing score identifies the mismatch, but an explanation can reveal a more specific problem. In Muller's example, the model says it omitted rain because happiness is important to mental health. That stated rationale points toward a conflict between reporting the observations and trying to make the reader feel good. Treat such explanations as diagnostic evidence, not a guaranteed account of the model's internal causal process.

Now change the input to sunny weather. The same preference for cheerful answers can produce an apparently perfect response while leaving the underlying defect intact.

The second case is more dangerous precisely because the answer looks right. Inspecting the explanation gives the team a reason to test whether the prompt follows the evidence when the weather is less pleasant.

A large prompt creates a measurement problem. Muller compares evaluating a model call to placing a multimeter's probes at its input and output: you can observe the behavior across the whole operation, but that does not isolate the tasks inside it. If one prompt interprets sensor data, applies thresholds, chooses wording, and writes a summary, an incorrect answer leaves several possible causes. Prompt decomposition creates intermediate outputs that can be evaluated separately.

The weather customer's prompt included a wind-speed rule: below five meant not very windy; above five meant windy. It worked in the prototype, but Muller reports that Claude incorrectly treated seven as less than five about 2–3% of the time as the weather workload scaled. The Claude version, test size, and wind-speed units are unspecified. The example shows why a plausible prototype result is insufficient evidence for a deterministic comparison embedded in a language task.

The team split the large prompt into a chain of smaller operations. Each boundary became an evaluation point, allowing them to identify which parts worked and where to focus. Decomposition also raised a more useful question than how to improve every prompt: does this step need a generative model at all? Comparing seven with five belongs in Python. Muller also points readers toward his writing on decomposition; his announcement of the Prompt Decomposition article provides a related starting point.

The comparison can be isolated directly. This implementation leaves equality unresolved because the described rule specifies only values below and above five:

def classify_wind(speed: float) -> str:
    if speed < 5:
        return "not very windy"
    if speed > 5:
        return "windy"
    raise ValueError("Define the wind classification at exactly 5")

wind_speed = 7
wind_description = classify_wind(wind_speed)
assert wind_description == "windy"

The downstream generation step can now receive wind_description instead of being responsible for calculating it. Muller reports 100% accuracy after replacing the mathematical comparison with Python in this weather example. He does not specify the measurement scope or sample size; the useful architectural result is that the team could inspect the accuracy of each step independently.

Semantic routing applies the same decomposition pattern to incoming requests. First classify the task, then send an easy request to a small model and a hard request to a larger one. The routing decision depends on the actual input's difficulty. The router itself has a compact evaluation contract: a query goes in, and an identifier such as 1 or 2 comes out. Its correctness can be assessed separately from the quality of the eventual answer.

This separation can also remove irrelevant prompt content. A single large prompt may contain instructions for both easy and hard tasks, even when a particular request needs only one branch. Muller calls those unused instructions dead space or dead tokens: they add cost and create opportunities for confusion. The slide makes the branches concrete with Nova Micro routing to Nova Lite for an easy task or Nova Premier for a hard one, then shades the hard-task branch as unused for the easy task. Muller attributes observed accuracy improvements in this pattern to sending only the instructions needed for the selected task.

The seven habits describe patterns Muller sees in workloads that successfully scale. He says he has never seen one scale without evaluations. The first habit, fast, determines how much learning a team can do. In a slow workflow, an engineer changes a prompt, publishes it to a test site, emails reviewers, and waits for replies. A week can pass before the next change. An evaluation framework that returns results in seconds lets the team test repeatedly while the problem is still fresh, rather than hoping to write the perfect prompt on the first attempt.

Muller's rule-of-thumb target is a 30-second evaluation loop: roughly ten seconds each for generation, judging, and summarization across about 100 test cases. It is a design target, not a measured runtime guarantee. The proposed sequence is:

1. Generate: Apply the candidate prompt template to the test inputs, sending approximately 100 generation requests in parallel.
2. Judge: Compare each result with its gold-standard answer, again using parallel calls where a model judge is appropriate. For a numeric route identifier, use an ordinary equality check instead.
3. Summarize: Aggregate the judgments into an explanation of what worked and what failed.

Parallelism makes the first two phases suitable for a short feedback loop; the last phase makes the output usable.

The developer should not receive a pile of 100 responses to read manually. Group the results by category and separate correct from incorrect cases. For tax questions, for example, the summary should explain the patterns shared by successful answers and the patterns behind failures. That returns the framework to its primary job: finding the errors and helping the team decide what to change.

Quantifiable means the framework produces numbers even when judgment is not perfectly repeatable. Muller returns to the professor analogy: an essay might receive a slightly different grade on Thursday than on Friday after the professor has had some wine. Variation does not make grading useless. The companion habit, numerous, is to evaluate enough cases that the aggregate is less dependent on any one judgment. Averaging can reduce random fluctuation; it does not remove systematic grading errors.

A larger test set also helps define the product. In Muller's customer exercises, building around 100 cases regularly exposes disagreements that prompt engineering cannot resolve. One teammate expects the assistant to direct tax questions to a site containing W-2 forms. Another believes the assistant must not answer tax questions at all. The team must settle that policy before a judge can score the behavior consistently. Include questions outside the intended scope, too, so the evaluation measures whether the system redirects them appropriately.

Explainable applies to both sides of the evaluation. Inspect the generator's explanation, but also ask the judge to explain its score. The judge prompt is itself an engineered component: it can misunderstand the task, apply the wrong criteria, or penalize the wrong behavior. Its explanations help reveal those defects, especially when scoring requires substantial judgment.

A rubric gives the evaluator specific rules instead of an open-ended request to assign a score. Muller's deliberately strict essay example makes the rules visible:

The particular penalties are less important than making the criteria explicit. A judge's explanation should show which rules it applied, giving the team evidence for improving the judge prompt as well as the generation prompt.

Segmented evaluations attach to individual workflow steps. Muller describes scaled systems as usually involving multiple steps and often multiple models. Evaluating each step separately makes model selection an empirical decision: determine the smallest model that satisfies that step's requirements. A simple semantic router returning 1, 2, or 3 is his example of a task suited to a small, fast model such as Nova Micro. The Amazon Nova launch announcement describes Micro's low-latency positioning; the workflow's own evaluations must establish whether it is adequate for the routing task.

Diverse means covering the intended range of behavior, rather than collecting many variations of one easy case. Start by listing the in-scope use cases, then assign test questions to each. Muller's approximately 100 cases are a heuristic. He might allocate three or four examples to a low-priority edge case and many more to a core use case. The distribution should reflect what the product needs to do reliably.

Traditional evaluation tools remain useful inside generative AI systems. There is no reason to ask another model whether a router's output of 1 is the expected number 1. Choose the measurement that directly fits the operation:

• Routing and numeric outputs: Use deterministic comparisons against expected values.
• Retrieval and question answering: Retain appropriate retrieval, database-accuracy, precision, and F1 evaluations.
• Operational behavior: Measure cost and latency with conventional instrumentation.

A model judge is one tool in the framework, not a replacement for every existing test.

The complete workflow begins with the gold-standard set. It deserves substantial attention because the rest of the system will be optimized against it. If the reference answers contain errors, improving the evaluation score can mean teaching the system to reproduce those errors.

Muller warns against accepting model-generated answers as gold. A model can help propose a silver-standard set, but a human must review those candidates and confirm their accuracy before they become trusted references. Generating more examples does not substitute for deciding whether their expected answers are right.

With reviewed references in place, the evaluation follows a concrete data flow:

1. Select a reference input. Insert it into the prompt template and send it to the generation model.
2. Collect the generated output. Retain the answer and its accompanying explanation.
3. Judge against the matching reference. Supply the generated output and gold-standard answer to the judge prompt. Request a numerical score and an explanation of that score.
4. Group the judgments. Carry a category label from the reference set into the results, then summarize the correct and incorrect answers within each category.

The final diagram shows why these fields matter. Its example includes the category Sunny Day and a judge explanation distinguishing correct weather content from an incorrect tone. That is the useful endpoint of the framework: a score connected to a specific defect, organized so the team can decide what to fix next.