Judge the Judge: Building LLM Evaluators That Actually Work with GEPA — Mahmoud Mabrouk, Agenta AI

An agent is in production, and the team adds a hallucination judge to monitor reliability. The dashboard looks healthy. Customers complain anyway, and the traces confirm that the agent is failing. Under the hood, the evaluator’s prompt amounts to: given an LLM output, decide whether it is a hallucination, and make no mistakes. The instruction supplies neither the missing facts nor the criteria needed to recognize a failure.

An evaluator needs a standard of correctness that matches the application. Mahmoud Mabrouk’s workshop builds toward that standard by calibrating an LLM judge against annotations, using GEPA to optimize its prompt. Here, calibration means making the judge’s decisions agree with the reference judgments; it is not a claim about calibrated probability estimates.

Offline development follows a familiar loop: change the prompt or harness, run evaluations, retain useful changes, and repeat. Human review can provide a strong signal, but reviewing the entire test set at every iteration is slow. An automated judge speeds up the loop only if its decisions track the judgments that matter. Otherwise, the team moves faster without knowing whether the application improved.

Online evaluation extends the same requirement to production. A judge aligned with business goals can help reveal regressions, improvements and changes in how users interact with an agent. Those observations can feed a data flywheel: inspect traces, turn edge cases into new evaluations, optimize the harness, and observe again. If both the harness and its evaluators can improve from new annotations, more of that cycle becomes amenable to automation.

Mabrouk introduces himself as Agenta’s co-founder and CEO, with a machine-learning background that includes computational biology and protein structure prediction. Agenta brings together observability, prompt management and evaluation; his current work includes sampling and automatic optimization. The practical exercise is an airline customer-support judge, taken through metric design, data curation, annotation, optimization and validation. The workshop repository accompanies the notebook walkthrough.

The exercise uses airline conversations from the τ-bench benchmark family. The agent can manage reservations and retrieve flight and user information, but those tools operate under detailed rules about what it may change and when. A successful tool call is therefore insufficient evidence of a successful interaction: the agent may cancel a reservation without first establishing that it qualifies for cancellation.

Mabrouk describes an initial collection of 599 traces, approximately 62% compliant and 38% non-compliant, generated across multiple models and trials. Assertion-based results are converted into trace-level verdicts with explanations. In the cancellation example, the useful annotation identifies both the outcome—non-compliant—and the missing prerequisite: checking the airline’s cancellation rules before approving the request. The policy is complex, and Mabrouk describes the collection as imperfect demonstration data.

The workflow has four steps: design the metrics, annotate the data, optimize the judge, and validate the result. The first step requires application knowledge. A generic hallucination score does not capture every way an airline agent can fail; a subject-matter expert needs to examine actual conversations and identify what went wrong. Mabrouk points to Hamel Husain’s error-analysis workflow: begin with open-ended comments on trajectories, then group recurring failures into categories.

The Agenta demonstration produces four categories:

Information delivery is distinct from performing the operation: an agent might make a change and fail to tell the customer. Each category becomes a separate judge, keeping the decision narrower than an all-purpose success score.

Use a binary verdict with a reason before attempting a graded score. Even agreement on compliant versus non-compliant is difficult to learn. A one-to-five scale adds distinctions that human annotators may struggle to apply consistently, making the target harder to define as well as harder to optimize.

In an annotation queue, the reviewer receives a named criterion such as policy adherence and supplies a verdict plus supporting reasoning. The reason is essential training information. A non-compliant label alone tells the optimizer that its answer was wrong; explaining that the agent approved cancellation before checking eligibility tells it which rule is missing. Some narrow tool failures may be evident from the trace itself, but reconstructing a complex business policy from unexplained binary labels is much harder.

Metric design and annotation receive less screen time than optimization, but Mabrouk identifies them as the hardest parts. The data must cover the relevant cases, and its annotations must contain enough information to learn a useful decision rule. This exercise has a small, unevenly distributed collection and a complex target policy. Its explanations are AI-generated from assertions and the original data, rather than collected from human reviewers. The companion extraction code identifies τ²-bench simulation inputs and policy-related assertion tags: the resulting task is derived policy-adherence classification, distinct from the original benchmark’s agent-success score. The recommended human-calibration workflow and the demonstration’s generated reference labels should therefore remain separate.

GEPA resembles a genetic search over prompts. Start with a seed, generate revised candidates, evaluate them, filter the candidate pool, and repeat. Understanding these operations matters because practical optimization requires tuning their behavior. The seed in this workshop asks whether a customer-service agent violated policy and begins by assuming compliance.

New candidates come from two operations:

• Mutation: Run a candidate judge on a trajectory, examine its input, output and evaluation feedback, and use an LLM’s reflection to propose a better prompt.
• Merge: Combine useful instructions from existing candidates—for example, guidelines learned by one prompt with complementary guidelines learned by another.

The first iteration has only the seed to mutate. As the pool grows, merging becomes possible. Proposed candidates are tested on mini-batches rather than the entire evaluation collection, and candidates that improve on their starting point can enter the pool.

Selection then asks more than which prompt has the highest average score. Mabrouk explains Pareto-aware selection through per-trajectory performance: preserve candidates with complementary strengths so that a prompt that solves a difficult case is not discarded merely because another prompt has a better overall average. The pool can collectively cover more cases than any single member. Merging attempts to consolidate those strengths into one rubric; it does not guarantee that consolidation will succeed. Mutation, evaluation and merging continue from the filtered pool until the compute budget is exhausted.

Mabrouk names DSPy as a prominent implementation of prompt optimization, then uses the GEPA authors’ library through optimize_anything. Its candidate is the configuration being optimized: a prompt string, a dictionary containing a prompt and temperature, or potentially a chain of prompts. The interface can therefore search over more than the wording of a single instruction.

The evaluator executes the system configured by that candidate. For this task, it runs the judge, compares its verdict with the reference annotation, and supplies diagnostic information for reflection. A scalar score tells GEPA which candidate did better; the trajectory, verdict, reasoning and annotation help explain how to improve it. Configuration controls the search budget and other behavior, while an objective supplies context for refinement.

The feedback boundary can be expressed in Python without tying it to a particular GEPA release. In this example, run_judge receives the candidate rubric and trajectory; the reference label and explanation are added afterward for the optimizer’s reflection. The function returns both an agreement score and the evidence needed to revise the rubric.

from collections.abc import Callable
from typing import Literal, TypedDict

Verdict = Literal["compliant", "non-compliant"]

class Example(TypedDict):
    trajectory: str
    verdict: Verdict
    annotation: str

class Judgment(TypedDict):
    verdict: Verdict
    reasoning: str


def evaluate_candidate(
    candidate: str,
    example: Example,
    run_judge: Callable[[str, str], Judgment],
) -> tuple[float, dict[str, object]]:
    judgment = run_judge(candidate, example["trajectory"])
    score = float(judgment["verdict"] == example["verdict"])
    feedback = {
        "trajectory": example["trajectory"],
        "judge_verdict": judgment["verdict"],
        "judge_reasoning": judgment["reasoning"],
        "reference_verdict": example["verdict"],
        "reference_reasoning": example["annotation"],
    }
    return score, feedback

The recording uses its workshop wrapper around optimize_anything; current GEPA documentation confirms the same candidate/evaluator/diagnostic-feedback pattern, but is not a pinned version of that historical interface.

The notebook uses dotenv, LiteLLM and GEPA, with some helper functions extracted into separate files. Optimization has already run before the recording, so the walkthrough inspects its outputs rather than waiting for another run. Mabrouk’s runtime estimates range from roughly an hour to a couple of hours across the walkthrough.

After loading the processed annotations, the notebook separates optimization data from final validation by benchmark task. The notebook shows 480 training traces and 112 validation traces. The saved optimization summary confirms those split sizes; together they total 592, leaving the earlier stated collection size of 599 unexplained. Repeated runs of the same task and model create redundancy within the data, but Mabrouk says that redundancy does not cross the training/validation boundary. This separation matters because another run of a familiar task would be a weaker test of generalization than an unseen task.

The annotation examples make the learning target concrete. A compliant trajectory correctly identifies a basic-economy reservation; a non-compliant one fails to recognize that a customer has regular membership. Those details carry the policy knowledge. Without an explanation of why each case is correct or incorrect, the optimizer would have to infer both the hidden business rules and how to apply them from verdicts alone.

The initial rubric is deliberately simple: evaluate whether the agent violated policy, assume compliance, and change the verdict only when a specific reason supports non-compliance. This is an engineered starting point. Earlier prompts that simply asked the model to judge policy violations encouraged it to substitute its own assumptions for rules it had never been given. Mabrouk found those nearly random starting judgments difficult to improve.

Mabrouk reports 61% validation accuracy for the seed judge, with very low recall for non-compliant cases. Its strong tendency toward compliance is intentional at this stage: the seed has little policy knowledge, so it should not manufacture reasons to fail a trace. Inspecting false-compliant cases then reveals the rules that the rubric needs to acquire. Seed sensitivity was consequential for this complex policy task, rather than proof that every judge should use the same default.

The next intervention changes the prompt that proposes better judge prompts. GEPA’s default reflection template did not produce the improvements Mabrouk expected, so he supplied a custom template with airline-domain context and an explicit instruction to learn policy criteria. Reflection sees the judge’s verdict alongside the reference annotation, then uses that discrepancy to decide what the rubric lacks.

The allowed edits are concrete: add rules, restructure existing rules, and reword instructions for clarity. For the cancellation failure, the useful change is a criterion about verifying eligibility before approval, rather than a general instruction to be more careful. The template directs the refiner toward reusable policy rules. Mabrouk reports that this domain prior helped, despite only limited iteration on the reflection template itself.

The experiment’s optimization wrapper constructs configuration from keyword arguments and calls optimize_anything. Its evaluator runs the candidate judge and supplies the trajectory and annotation as feedback. The resulting rubric acquires criteria for flight cancellations and refunds, flight modifications, and communication. The prompt has moved from a default assumption about compliance toward explicit instructions grounded in the annotated cases.

For the recorded experiment, Mabrouk reports validation accuracy rising from 61% to 74%. The rounded endpoints imply a gain of 13 percentage points. Non-compliant precision and recall also improve from near zero, indicating that the optimized rubric has learned to flag violations rather than simply passing almost everything.

Mabrouk also reports a compliance-related figure falling from 98% to 64%, but describes it inconsistently as bias, prediction frequency and recall. It should not be treated as a precisely defined metric. The companion repository currently reports different accuracy results, 65.2% to 69.6%; those checked-in artifacts have not been tied to the exact recorded run, so they do not replace the recording’s figures.

Training accuracy reportedly improves by nine percentage points. More strikingly, Mabrouk reports 100% training-task coverage across the Pareto candidate set: for every task, at least one generated candidate gets it right. That is a property of the collection, not the accuracy of the selected judge.

The experiment achieves the second property without fully achieving the third. GEPA struggles to merge all the useful information into one prompt, leaving the final judge substantially short of the strong agreement Mabrouk wanted. Imperfect data and limited tuning constrain the result, and reaching it required several unsuccessful experiments.

The retrospective begins with model capacity. Using GPT-4o for both judging and refinement failed in Mabrouk’s trials on this complex airline-policy logic. He also tried models described as Mini and Nano, along with Gemini and DeepSeek. His best reported combination used Gemini for reflection and Grok for judging; he also says a model he calls GPT-4 Mini worked fairly well in both roles. The companion Grok/Gemini experiment specifies Grok 4.1 Fast and Gemini 3 Flash Preview, clarifying that saved configuration without resolving all the model names in the spoken comparison.

The useful debugging unit is a small iteration, not a full search budget:

1. Run a short optimization and inspect the reflection reasoning.
2. Read the generated candidates and check which ones actually improve.
3. Stop at a failed example and identify what the refinement prompt misunderstood.
4. Add the missing context or prior, then test whether the next candidate fixes the training case.

Mabrouk stopped after the first iteration and used Claude Code to edit the refinement prompt. His immediate goal was to make the process fit the training examples before scaling it. That separates an optimizer that cannot learn the available signal from one that can learn but may not generalize. The strong collective frontier coverage then directs attention toward candidate merging as an additional unresolved problem.

Mabrouk tested two seed families. One omitted the agent’s original policy; the other copied that policy directly into the judge prompt. The policy-free seed performed better in his experiments. His hypothesis is that the copied policy constrained the search around a difficult local optimum, while a simpler seed left more room to explore alternative rubric structures.

The better-performing seed was not learning without policy information. Its annotations still explained why trajectories complied or failed, carrying much of the policy into the reflection loop. The distinction is where the knowledge enters: as a large starting instruction, or incrementally through evidence about specific decisions. That makes this a finding about initialization and feedback in these experiments, rather than a reason to withhold necessary business rules from evaluators generally.

Mabrouk estimates that his small experiments cost about $200–$300 in model tokens. Long trajectories make every evaluation input expensive, and capable judge and reflection models add to that cost. He stopped an experiment with GPT-4 because of expense; even the model he calls GPT-4 mini incurred meaningful spending. Moving to cheaper Nano or other smaller models did not produce adequate results in those trials.

The economic motivation for a larger refiner and a smaller judge remains compelling when the judge will evaluate many production traces. Refinement is an upfront search expense; judging becomes a recurring inference expense. Spending more to discover a rubric that a less expensive model can apply reliably may pay off over a large volume of online evaluations. The capability requirement still has to be demonstrated on the application’s actual cases.

Before increasing sampling, instrument the optimization and inspect its traces, generated prompts and training behavior. Mabrouk used Agenta for that inspection and estimates roughly 200–300 iterations per experiment. Batch size and other search parameters also needed tuning. The practical stopping point for the first small run is evidence that reflection can turn a diagnosed failure into a useful candidate; only then does a larger budget have a clear job to do.