The Benchmarks Game: Why It's Rigged and How You Can (Really) Win

What happens when the score used to judge an AI system also determines who wins investment, contracts and developer attention? Improving the score becomes a commercial objective—and that objective can diverge from improving the system. Darius Emrani, introducing himself as Scorecard’s CEO, approaches this problem from his experience evaluating autonomous systems at Waymo and Uber ATG, rockets at SpaceX, and his claimed Waymo evaluation patents. His current work spans legal technology, health technology and finance. The practical question is how to recognize distorted comparisons and build evaluations that remain useful despite those incentives.

A benchmark has three components: the model, the test set and the metric. It bundles individual evaluations into a common comparison. Standardizing the questions and scoring makes different models comparable, much as the SAT gives different test takers the same questions and scoring rules. Without that shared contract, two numbers that look comparable may describe different experiments.

Those numbers influence much more than research discussions. Emrani invokes Simon Willison’s warning about investment decisions resting on benchmark scores, then traces the consequences through enterprise contracts, developer mindshare and the attention generated by figures such as Andrej Karpathy. A claim to leadership by OpenAI or Anthropic can shape which systems an entire ecosystem tries next.

Sonar’s acquisition of AutoCodeRover makes the stakes concrete. Emrani connects the acquisition to AutoCodeRover’s strong SWE-bench results; the acquisition itself is established, while that causal explanation remains his interpretation. When a single score can help define a market leader, there is a strong incentive to find favorable ways to produce and present it.

The first failure mode is comparing one model’s best configuration with another model’s standard configuration. Emrani uses the Grok 3 launch as his example, alleging that xAI’s presentation gave its own system the favorable setup. His analogy is a sports car with a nitrous booster racing cars without one: the result may describe the race accurately while concealing the difference that made it possible.

Specifically, he says the presentation omitted OpenAI o3 at consensus-64. The omitted competitor configuration remains an allegation in this account, rather than an established reconstruction of the chart. Consensus-64 runs the model 64 times and selects a consensus answer. That changes the inference procedure as well as its expense: the score is no longer the result of asking the model once. It does not establish an exact multiplier for total system cost.

The configuration is part of the result. A performance claim needs the inference settings and resource requirements that produced it; selective omission can otherwise turn a configuration advantage into an apparent model advantage.

The second failure mode concerns who can see the questions. FrontierMath was presented as a protected benchmark of advanced mathematics, designed to resist gaming. Emrani describes OpenAI’s funding and access as a challenge to that premise. His description of access to the entire dataset needs a qualification: Epoch’s clarification describes a holdout whose problem statements would be available to OpenAI, but whose solutions would be withheld.

Emrani also acknowledges a reported verbal agreement not to train on the data and employees’ characterization of it as a strongly held-out evaluation set. Access is not evidence of training contamination. His concern is the asymmetry: a funder can inspect questions, evaluate internally and announce a score before independent verification. That arrangement can weaken confidence even when everyone follows the agreement.

Emrani cites an announced o3 score of approximately 25% on the then-current FrontierMath benchmark. The historical announcement is the relevant context, not a current score or an independent reproduction. The lesson is about governance: financial relationships and privileged access need scrutiny even when there is no demonstrated misconduct.

The third failure mode sits inside the metric itself. Emrani introduces Llama 4 Maverick and describes private model variants tested in LLM Arena before the public release. The reported count is 27 private Meta variants before the Llama 4 release, not necessarily 27 versions of Maverick. Testing many candidates and publicizing the favorable result creates a selection problem; it does not establish that every candidate was trained to be inaccurate.

His distinctive example is a request for a riddle whose answer is 3.145. In his account, a private variant produced a long, flattering, emoji-filled response that made no sense, yet won against Claude’s correct answer. The reported outcome illustrates a mismatch between preference and correctness: a voter can enjoy an answer without checking whether it satisfies the task. A system optimized for that vote can learn to prioritize the attractive presentation.

The Arena style-control analysis makes the distinction more precise. Its controls address answer length and specific Markdown features, including headings, bold text and lists—not a comprehensive removal of personality. In that analysis, GPT-4o-mini and Grok-2-mini fell below most frontier models, while Claude 3.5 Sonnet tied for first in the Hard Prompt subset. Those conditions matter: this is not a claim that Sonnet became the universal winner after all style effects disappeared.

Emrani compares choosing by charm to choosing a surgeon by bedside manner rather than surgical skill. Presentation can matter to the experience, but it is not interchangeable with competence. His SAT analogy has the same structure: in the cited SAT essay study, word count explained 39% of essay-score variance. That concerns essay scores, not overall SAT performance, and the nonexperimental relationship does not show that adding words causes a better score.

These mechanisms converge on Goodhart’s law: “When a measure becomes a target, it ceases to be a good measure.” A benchmark begins as a proxy for a capability. Once commercial rewards attach to the proxy, optimization can shift toward favorable configurations, privileged testing or appealing presentation. The number may improve without an equivalent improvement in the capability the reader cares about.

Emrani reinforces the concern with remarks attributed to people close to evaluation work: Karpathy’s uncertainty about which metrics to consult, SWE-bench creator John Yang’s observation that benchmark designers made the benchmarks up, and CMU’s Maarten Sap’s criticism of the yardsticks. These comments express distrust in the available measures; they do not establish that every benchmark is useless. They make the next question urgent: what conditions would make a score worth trusting?

Reform starts with the same three components that make a benchmark comparable. For model comparisons, Emrani proposes matching computational budgets and constraints, avoiding configuration cherry-picking, and exposing cost–performance trade-offs. He points to the ARC Prize chart as an example of showing score alongside cost per task. A high score then becomes a point on a trade-off curve, rather than a claim stripped of its price.

For test sets, his proposal combines open data, methodology and code with financial independence from the companies being evaluated. He also calls for regular question rotation to reduce overfitting. Transparency makes the procedure inspectable; refreshing the questions helps keep a published test from becoming a fixed target indefinitely.

For metrics, he calls for style controls and disclosure of all attempts. These address different distortions: style controls reduce the influence of presentation on the measured result, while publishing attempts exposes selection of the best run. Together, the reforms make it easier to distinguish capability from the conditions used to obtain a favorable number.

There are partial steps in this direction. Arena’s style-controlled rankings address some presentation effects. Domain efforts such as LegalBench and MedQA narrow the task being measured; Emrani also names FinTech. He presents AgentEval and BetterBench as broader efforts to examine evaluation itself. These are examples of progress, not a claim that every named project implements every proposed reform.

A product team does not have to wait for public benchmarks to improve. It can change the decision it is trying to make: from identifying the public leaderboard winner to identifying the system that works for its own users. Application-specific evaluation is a model-selection process grounded in the workload.

1. Gather real data. Start with actual production queries. Emrani’s comparison of five real queries with a hundred academic questions is a heuristic about relevance, not a measured ratio or a sufficient sample-size rule.
2. Choose the metrics. Decide how quality, cost and latency matter for this application. A chatbot and a medical diagnosis system need different criteria.
3. Test candidate models on that data. Emrani suggests trying five leading candidates rather than selecting directly from a leaderboard. Generic strength does not guarantee success on a specialized workload: GPT-4 failing on legal documents is his illustrative possibility, not a reported experiment.
4. Make evaluation repeatable. Use a consistent process, whether built internally or supported by a platform such as Scorecard.
5. Keep iterating. Repeat the evaluation as models improve and application requirements change.

The operational loop begins with an issue, continues through an improvement, and evaluates that improvement before deployment. Evaluation feedback may send the change back for another revision. Deployment happens only after the candidate meets the quality bar. This makes the evaluation an input to a release decision rather than a score collected after the decision has already been made.

A small TypeScript gate can express that policy without tying it to a model provider. Here, a failed required check keeps the candidate pending; a passing set makes it eligible for deployment but does not deploy it.

type QualityCheck = {
  name: string;
  passed: boolean;
};

type ReleaseDecision = {
  status: "pending" | "eligible";
  reasons: string[];
};

function assessRelease(checks: QualityCheck[]): ReleaseDecision {
  if (checks.length === 0) {
    return { status: "pending", reasons: ["No evaluation results"] };
  }

  const reasons = checks
    .filter(check => !check.passed)
    .map(check => check.name);

  return {
    status: reasons.length === 0 ? "eligible" : "pending",
    reasons,
  };
}

const decision = assessRelease([
  { name: "Answers the legal question correctly", passed: true },
  { name: "Supports claims with document evidence", passed: false },
]);
// { status: "pending", reasons: ["Supports claims with document evidence"] }

The team still has to define and measure those checks. The gate makes their consequence explicit: an unresolved failure prevents release eligibility.

After deployment, monitoring feeds the next round of issue identification. Emrani describes this pre-deployment evaluation and post-deployment monitoring cycle as a way to reduce production firefighting, without supplying a measured reliability improvement. It requires more work than checking a leaderboard because it keeps testing whether the system serves the application as the system and its users change.

Market value, acquisitions and developer attention will continue to reward public scores. A product’s evaluation process can nevertheless reward something more specific: improvements that matter to its users. Emrani closes with “All benchmarks are wrong, but some are useful.” The useful benchmark is the one whose questions, scoring and operating conditions support the decision you actually need to make.