Your Evals Are Meaningless (And Here’s How to Fix Them)

You have automated tests, you run them before production, and the results look good. Does that establish that your AI application works for its users? Mohak Sharma recalls having the same confidence while building the ML systems platform at Templafy—before seeing recurring evaluation failures across other teams.

Sharma describes co-founding HoneyHive in late 2022 after leading Templafy’s data and ML platform team. Working with hundreds of teams, from two-person startups to Fortune 100 enterprises, exposed similar problems across multi-agent systems, text-to-SQL, and retrieval-augmented generation. These were not confined to one application architecture.

The practical question is whether evaluation helps build a system that delivers value in production. Catching bugs and measuring accuracy matter, but neither guarantees that an application will produce a return on investment beyond a convincing demo. An evaluation must measure the quality the business and its users actually need.

Evaluations serve a purpose familiar from unit and integration testing: check quality before shipping changes. But saying that tests exist leaves three questions unanswered: what is being evaluated, what cases represent its workload, and how will its outputs be judged? These correspond to the agent, dataset, and evaluators.

The agent is simply the target under evaluation. It might be an end-to-end agent, a function inside one, or the retrieval pipeline alone. Choosing that boundary matters because a customer service chatbot and a legal-contract Q&A system have different responsibilities, even if both generate text.

Consider financial document Q&A. A correct answer may still be insufficient if the system must comply with regulations, explain its reasoning, or account for nuances in financial accounting standards. Define those requirements before choosing a score; otherwise, a convenient metric can become a substitute for the actual quality contract.

The dataset determines what the evaluation can reveal. Fifty queries handwritten by developers may be a useful beginning, but their existence does not establish coverage. Include both the inputs the system will receive and the ideal outputs that describe successful behavior. Cover ordinary requests and difficult edge cases, with domain experts supplying the business context needed to define what the agent owes its users.

Evaluators then measure whether the outputs meet that contract. The available approaches have different strengths:

Sharma also dismisses ROUGE-L as unsuitable here, without developing a separate argument about when that metric works. The appeal of LLM evaluation is its promise to combine nuanced judgment with automated speed and scale.

Together, the three components make the evaluation easier to interrogate: which system behaviors matter, how representative is the dataset, and do the evaluators measure what you think they measure? None is fixed. As the agent improves, the dataset may need harder cases; as the quality criteria become more sophisticated, the evaluator may need to change too.

The appeal is strong enough that Sharma reports seeing teams move their entire evaluation stack to LLM judges. Lower cost and faster turnaround let teams evaluate more frequently. He describes work that took roughly eight to ten hours with human evaluators completing in under an hour with LLM evaluators, then gives a more concrete workload comparison.

These are Sharma’s reported estimates, not a controlled benchmark or a current pricing guide:

The sequential execution condition matters: the timing example does not assume parallel evaluation. Sharma characterizes the cost improvement as tenfold, but the broad ranges do not establish a fixed savings ratio, and he does not specify the model, token lengths, or matched quality conditions.

Sharma reports over 80% agreement with human judgments in the evaluations he describes. He compares this with the imperfect agreement between human evaluators: humans are not a unanimous reference panel either. That comparison motivates using an LLM judge, but it does not establish its accuracy on a new application.

He points to research under the names NLG-Eval and SPADE, and describes OpenAI and Anthropic as pursuing model-based evaluation for alignment. The research references need to be read narrowly: SPADE concerns synthesizing data-quality assertions from prompt histories, rather than establishing a general human–judge correlation result. The practical promise of an LLM judge still leaves the application-specific question unanswered: what, exactly, is it rewarding?

The first failure is a mismatch between the evaluator’s criteria and the application’s needs. Built-in criteria in frameworks such as Ragas, promptfoo, and LangChain need to work across many applications. That generality does not make their defaults a complete definition of quality for yours. The issue is accepting generic criteria unchanged, not an inherent inability to customize them.

Sharma’s example is an unnamed startup building LLM-based recommendations for e-commerce websites. Its evaluator checked context relevance on the retrieval side and answer relevance on the generation side. Testing looked reassuring. After launch, however, users complained: the relevance checks emphasized keywords without adequately considering what a product description meant or whether the product met the user’s request. The scores missed the relevance failures that mattered in production.

A separate source of instability appeared when an underlying judge model changed without a stable version being used. The same test case could then receive inconsistent judgments. Sharma uses criteria drift broadly for the evaluator’s notion of good diverging from the user’s notion of good; both an unsuitable rubric and changes in judge behavior can undermine that alignment.

EvalGen, introduced by Shreya Shankar and collaborators, highlights a more specific process: people refine their criteria as they observe outputs. That is distinct from changing the underlying judge model. Either way, evaluation criteria cannot simply be assumed correct forever. Sharma connects the repair to measuring agreement with human judgments, balancing true and false positives, and tracking F1.

The second failure is dataset drift, which Sharma describes as a lack of test coverage. A team can spend weeks writing clear queries with obvious right and wrong answers, then discover during beta that users submit vague, context-dependent requests. The carefully constructed test bank no longer represents the workload.

The gap takes several forms:

• Broader topics: users ask about subjects beyond those covered by the test cases.
• External information: requests may depend on real-world search data, such as SERP API results.
• Combined questions: users join multiple requests in ways the developers did not anticipate.

The evaluator can continue scoring its existing cases successfully while the application fails on these missing inputs. A capable judge cannot compensate for an unrepresentative dataset.

Sharma compares this to training for a marathon entirely on a treadmill. Improvement on the treadmill does not establish readiness for the course: incline and surface traction are absent. Likewise, better scores on clean, familiar cases can conceal the conditions the application will actually encounter.

The repair is to develop evaluators and datasets iteratively, much as you develop the LLM application itself. A one-time setup cannot keep both aligned as requirements, inputs, and model behavior change.

1. Align the judge with domain experts. Have experts regularly grade outputs and critique the evaluator’s decisions. Ask what it missed and what it overemphasized. Turn those critiques into few-shot examples, inspect the underlying evaluator prompt, and revise it until its judgments reach a satisfactory level of agreement.
2. Align the dataset with real usage. Start logging production behavior. When a query underperforms, bring it into the test suite, either manually or through automation. The test bank should change as production exposes missing cases.
3. Measure alignment at every iteration. Use F1 for binary judgments or correlation coefficients for Likert ratings. Compare the evaluator with human judgments to determine whether a revision improves alignment or introduces a regression.

This adds work to evaluator development, but it is the work that makes the resulting scores interpretable. Sharma argues that it is cheaper than dealing with the consequences of evaluations that fail to measure meaningful quality.

The most useful initial intervention, in Sharma’s experience, is to customize the evaluator prompt. Tune its criteria to the application and business context, then include examples of expert critiques. Choose the rating scale deliberately: a binary judgment and a Likert rating ask the evaluator to make different kinds of decisions. Sharma recommends binary ratings rather than treating a more granular scale as automatically more informative.

Bring domain experts into this process early. A spreadsheet with around 20 examples can provide an initial sense of whether their judgments and the evaluator’s judgments line up. This is a starting review, not a claim that such a small sample establishes comprehensive reliability. The disagreements tell you what to change next in the evaluator prompt.

Operationally, begin by reading production logs. Sharma credits Hamel with this emphasis: every underperforming interaction is an opportunity to improve the test bank. Those failures are valuable because they expose the exact situations the application needs to handle better. Add the cases and their ground-truth labels continuously, so the dataset carries both the observed request and a judgment of the desired behavior.

As that bank expands, test new evaluator-prompt versions against it. Make the criteria more specific to the use case rather than treating the original prompt as permanent. Give domain experts an evaluation console where they can edit prompts and inspect the resulting critiques and judgments; build one internally if necessary. Their role should extend beyond supplying labels once at the beginning.

A simple dashboard can make evaluator progress visible. Track alignment with human judgments across evaluator-template revisions: F1 for binary decisions, or correlation for Likert ratings. The dashboard answers a different question from an application quality score. It asks whether the instrument used to judge the application is becoming better aligned with the people who understand the task.

Treat evaluator-prompt changes as systematically as application-prompt changes. The goal is continuous improvement rather than perfection: inspect failures, revise criteria, and measure the effect. Evaluation remains useful only while it stays connected to real-world usage. Build that feedback loop into development instead of treating the test suite as a finished artifact that can be set aside after launch.