Your LLM Judge Is a Confident Liar: Building Better Verifiers — Browserbase

Early Stagehand evaluations used deterministic environments: controlled sites and checkpoints that could establish whether an agent had completed a task. Miguel González Fernández, Browserbase’s agent-platform tech lead, describes how that approach became harder to maintain as models and the agent harness improved. More capable agents needed longer tasks, and the evaluation infrastructure required constant patching, generated static sites, and additional checkpoints. Corby Rosset, a Microsoft Research researcher, joined the effort to build better verifiers.

A live website makes those checkpoints fragile. Several paths can lead to a correct result; a product can disappear; a blockage can prevent an otherwise sound action. The desired outcome may be clear while the environment no longer supplies the fixed conditions needed for a deterministic test. An LLM judge offers a way to interpret these varied trajectories without hand-building every possible path.

But human experts reviewing the judges bundled with Online-Mind2Web and WebVoyager found confidently wrong decisions. That creates a problem beyond misleading benchmark scores. If the same judge supplies reinforcement-learning rewards or guides harness optimization, its mistakes become targets for the optimization loop. A system can improve at earning approval while failing to improve at the user’s task.

Microsoft’s Fara-7B browser agent made the discrepancy concrete. On the same benchmark, the official WebVoyager judge, GPT-4o, assigned a 74% success rate. The team’s Universal Verifier assigned 38%. The agent had not changed; the decision about which attempts counted as successful had changed. The lower estimate came from a verifier whose agreement with human labels the team separately evaluated.

Several weaknesses explain why judging a trajectory is harder than reading an agent’s confident final answer:

• Missing rubrics. Without explicit success criteria, the judge has no clear structure for assigning credit.
• Poor evidence selection. A judge may omit relevant screenshots or ingest every screenshot until the context becomes crowded or overflows.
• Incomplete trajectory review. Omitting the final answer or action history removes information needed to compare the agent’s claims with what it actually did.

The Universal Verifier addresses these together. A stronger judging model still needs a well-defined task and the right evidence.

Consider the example request to book the cheapest flight from Seattle to Boston. The verifier first generates a rubric containing the criteria that define success. It then examines the agent’s trajectory and ranks screenshots separately for each criterion. The top K screenshots become the evidence used to decide whether that criterion was satisfied and whether the agent’s account contradicts the visible environment.

This makes evidence selection part of verification. A screenshot’s relevance depends on the question being checked. Instead of asking one model to remember an entire visual history and deliver a single verdict, the system narrows the evidence around individual obligations, while also considering the final answer and action history.

The output preserves two different judgments. A scored rubric supplies a process score, allowing partial credit for steps performed correctly. An outcome Boolean, accompanied by an explanation, states whether the result meets what a reasonable user would expect. How does the trajectory become those two judgments? The diagram shows the per-criterion evidence path and the separate outputs. Screenshots support specific criteria before those judgments become an assessment of the whole task.

Rubric generation can introduce its own errors. In the Jakarta example, the request is to find a cheap hotel for specified dates, use its address to locate the closest coffee shop, and return that shop’s name and address. A generated rubric added a requirement to report the total hotel-stay price. Price matters when choosing a cheap hotel, but reporting the total was not requested. Penalizing its omission artificially lowers the score.

Dependent steps require another distinction. One task asks for the net worth of the member of NSYNC or Backstreet Boys with the longest last name. The agent selects Timberlake, whose surname has ten letters, instead of Kirkpatrick. That selection is wrong. But if it then accurately finds Timberlake’s net worth, the lookup step has worked for the person it selected.

The observable change is in credit assignment: fail the name-selection criterion and preserve credit for an accurate downstream lookup. Otherwise, one upstream mistake produces several penalties, obscuring which capability needs improvement. This partial process credit does not make the requested answer correct; the outcome judgment remains a separate question.

Evidence checking also catches small factual substitutions that a fluent answer can hide. In an image-captioning research task, the agent reported a 6.2% improvement in CIDEr score while the underlying paper’s abstract stated 2.8%. The error requires comparing the returned claim with the source, rather than merely recognizing a plausible metric and a plausible number. Such discrepancies can also escape human reviewers.

The Amazon plushie example explains why process and outcome should remain separate. The agent searches for the requested toy correctly and discovers that it is out of stock. The environment has blocked the purchase. Under the team’s grading scheme, that attempt receives full process marks for its best effort, while the outcome remains unsuccessful because the toy was not bought.

Different continuations call for different judgments:

• Unavailable product. Credit the correct work while recording that the purchase goal was unmet.
• Successful alternative. In the scenario described, finding a similar alternative can receive both process and outcome success.
• Controllable error or hallucination. Penalize a mistake attributable to the agent, including unsupported claims about what happened.

The verifier follows an explicit failure-mode schema for these cases. An external blockage and an agent mistake can both end without the requested purchase, yet they supply very different signals for improving the agent.

To evaluate the verifier, human experts first reviewed the full trajectory and judged the evidence and action results. Only afterward did they see the Universal Verifier’s judgment and indicate agreement or disagreement. This ordering preserved an initial human assessment while allowing the verifier’s explanation to expose something the reviewer had missed. The resulting labels were released through CUAVerifierBench.

On the evaluated labels, González Fernández reports that false positives fell from almost half to zero. That is a result on the tested set, rather than a guarantee that future false positives cannot occur. The team also measured agreement using Cohen’s kappa, reporting 0.58 and describing verifier–human agreement as comparable to agreement between the two human annotators per task. Human labels supplied a reference, but the verifier sometimes identified mistakes humans had overlooked.

A second test asked whether the verifier could improve training data. The team held dataset size constant at 3,000 or 9,000 trajectories and changed which verifier filtered those trajectories for supervised fine-tuning, or SFT. At 3,000 examples, selection using the Universal Verifier’s process score produced a better model than selection using the baseline verifier; the advantage also held at the larger scale.

Holding the number of examples fixed makes selection quality the useful comparison. The causal path is straightforward: a better verifier chooses better demonstrations, and SFT learns from those demonstrations. This experiment tests whether verification helps the downstream agent, extending the evidence beyond agreement with human judgments.

Building the Universal Verifier took González Fernández and Rosset about three weeks and roughly thirty experiments, with prompt and code changes evaluated against human labels. Could an automated research loop rebuild it after their code and prompts were removed? In the reported run using Opus 4.6, the loop ran the same number of experiments in about one day, but reached only about 70% of the human-built verifier’s agreement. That figure is relative to the verifier’s agreement level, rather than a claim of 70% task accuracy. Rosset had not yet tested the proposed follow-up with another model, so the speed and agreement comparison applies to this run.

A second automated run received the findings from the three weeks of human experimentation. That primed run exceeded the level the team had previously reached. The comparison separates two contributions: automation compresses the time needed to run experiments, while human discoveries can give the search a better starting point. In this experiment, combining them worked better than asking automation to rediscover everything.

The released work includes a preprint, golden labels in the Microsoft Fara repository, and experiment code. González Fernández also describes a new benchmark used to find remaining capability gaps as established web benchmarks become saturated. His concern is that familiar test tasks may offer little fresh signal when they resemble the training distribution; the harder benchmark serves as a working instrument for finding where agents still fail.

The first audience question extends verification beyond the browser. Microsoft was working on a version for enterprise-style desktop tasks, with a further benchmark planned. The proposed verifier would inspect terminal information, telemetry, and logs alongside visual evidence. Those additional channels expand what the system can check; the underlying approach still connects task criteria to evidence of what happened.

The final substantive question concerns overfitting: repeated prompt improvements can make a verifier agree with the labels used during development without making it a better judge on new trajectories. The team’s answer was to separate development labels from evaluation labels. Of approximately 150 trajectories, Rosset labeled about 50 and used them during the roughly thirty experiments. The other 100 were held out, with two paid human annotators labeling each trajectory.

The reported CUAVerifierBench results came from those held-out trajectories, on which the team did not iterate. Keeping at least two-thirds of the labels out of development gives the final measurement a different job from the optimization score. Rosset’s practical recommendation is to train human reviewers to perform the verification task themselves, then test whether the system agrees with them. Building a judge requires both a standard to learn from and examples that remain unseen until it is time to measure.