AI Engineer World's Fair 2026
When Will The Benchmaxxing Plague End?
Read the talk
When Benchmark Scores Stop Measuring Better Models
A leaderboard can reward progress that users never feel. Nick Heiner traces how benchmark design, verification shortcuts, and lab incentives create that gap—and what better evaluation requires.
From a talk by Nick Heiner
The release chart meets actual use
A new model arrives with a big announcement and impressive benchmark charts. Sometimes the charts themselves stretch the comparison. Then people use the model, discover that it does not meet their expectations, and accuse the lab of benchmaxxing: training toward benchmark success in ways that diverge from what users actually care about. The score improved; the experience did not.
That gap raises two different questions: why do benchmarks fail to reflect useful capability, and is the failure unavoidable? Nick Heiner’s answer is that incentives and poor methodology explain much of the problem. Neither makes trustworthy evaluation impossible. The task is to understand where the measurement loses contact with the work it is supposed to represent.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Why weak benchmarks remain influential
Heiner describes millions of dollars wagered on LM Arena outcomes in prediction markets, even while industry figures publicly discuss gaming its rankings. He invokes Gwern’s criticism that an easily gamed benchmark may do more harm than good, then Karpathy’s observation that his preferred models did not match Arena’s ordering. Karpathy’s concern, as presented here, is that teams may be making better Arena models rather than better models overall—with nested lists, bullet points, and emojis standing in for useful improvement.
The persistence of those rankings has a straightforward mechanism. AI is available to an enormous audience, and that audience needs help choosing models. Assessing benchmark methodology requires expertise and time; recognizing a popular leaderboard does not. Popularity becomes a substitute for validity, reinforcing incumbency and marketing. Heiner includes himself in the problem: without inspecting a benchmark in detail, he does not consider himself equipped to judge it. Understanding the failure modes is therefore a prerequisite to interpreting the scores.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Good tasks are expensive to create and replace
Consider an agentic coding benchmark with tasks created by software engineers. Heiner’s hypothetical estimate puts construction at $15 million for 1,000 tasks requiring 60 hours each, with engineers costing $500,000 per year. That arithmetic implies 2,000 working hours per engineer-year: $250 per hour, or $15,000 per task. Under the same estimate, replacing one-third of the tasks each year costs another $5 million annually. Replacement matters because improving models progressively exhaust the useful difficulty of existing tasks.
A budget like that pushes benchmark creators toward shortcuts. The price slide pairs public training data with contamination, AI assistance with circularity, small samples with noise, and cheap labor with missing expertise. Heiner focuses on the expertise problem: a benchmark intended to measure progress beyond current models needs knowledge and judgment from outside those models. Heavy reliance on AI-generated tasks risks reproducing the frontier’s existing limits, while cheaper labor can remove the expertise needed to recognize a meaningful challenge.
Surge’s stated response is to pay for strong workers and prioritize quality over minimum cost. Heiner argues that this is especially necessary given the capabilities of models in 2026: creating useful tests increasingly requires people who can do more than the systems being tested.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Public tests can become remembered answers
Contamination does not require a deliberate decision to train on the test set. With large volumes of incoming training data, keeping public benchmark questions and answers out of a model requires active effort. Labs must resist that inflow; once test material is widely available online, some memorization becomes a predictable risk.
For SWE-bench Verified, Heiner reports that giving Opus the beginning of a benchmark prompt causes it to reproduce the remainder verbatim, and that it can reproduce answers as well. Surge’s investigation, he says, compared memorization of benchmark material with memorization of other content from the same repositories. That comparison matters because it asks whether benchmark items are unusually familiar, rather than merely showing that a coding model has encountered a public repository.
Heiner also says that the Opus 4.8 model card reports a SWE-bench score without disclosing the contamination his team found. The specific experiment and disclosure allegation are his claims; the broader consequence is that consumers cannot properly contextualize a score without knowing whether successful answers reflect fresh problem solving or remembered test material.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The verifier can reward the wrong thing
Reward hacking occurs when a model satisfies the letter of a scoring rule while evading the intended task. Heiner suggests treating reward design as an adversarial process against an agent seeking the easiest possible success. His analogy is gradient descent as water flowing downhill: it finds the path of least resistance. A verifier has to make that path correspond to doing the work, rather than exploiting a loophole.
A verifier can also make the opposite mistake: rejecting a useful answer because it tests an unnecessarily narrow representation. In Heiner’s AutomationBench example, an agent makes tool calls in an enterprise environment, but a phone-number check accepts only one hard-coded string. Several formats represent the same number, and the prompt does not specify which format the verifier requires.
In Heiner’s phone-number task example, Haiku and Fable both score 20%, although he says Fable gets the task right 80% of the time and loses credit for choosing different formats. Haiku’s score reflects mistakes; Fable’s reflects a mismatch between acceptable behavior and the verifier. Those percentages concern his particular example, not an aggregate AutomationBench result. A test that gives those behaviors the same score has erased the capability difference it was meant to measure.
Benchmarks are supposed to act as a lighthouse for approaching capabilities, including systems that could change entire industries. A hidden string-format requirement cannot carry that responsibility. The check must distinguish failure at the task from harmless variation in how a successful result is represented.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Instruction following needs product judgment
A benchmark expresses a view of what an AI system should do. Choosing tasks and success criteria is therefore a product decision as well as a measurement decision. Heiner describes benchmarks as aspirational artifacts: they encode desired behavior and values. His criticism of IFEval begins with that choice of target.
Despite its frequent appearance in model cards, Heiner characterizes IFEval’s prompts as arbitrary requests combined with additional constraints. Avoiding all commas or using the letter T at most once is easy to specify mechanically, but its relevance depends on a further assumption: improvement on such constraints must generalize to instructions users actually give. A check being reproducible does not establish that connection.
Some examples, he says, fail before the model even responds because their requirements conflict:
| Requested behavior | Conflicting requirement |
|---|---|
| Repeat the response verbatim | Translate it into Hindi |
| Include exactly one bullet point | Include a few bullet points |
A model cannot satisfy both sides of either pair as presented. The resulting failure score cannot cleanly identify an instruction-following weakness.
Other problems appear inside the checking logic. Heiner points to sentence splitting that differs from human interpretation, then a story-writing prompt whose verifier never checks whether the answer is a story. It checks only whether ASCII I appears more than once. That leaves the main requested behavior outside the measurement.
The displayed response D exploits that gap by substituting Cyrillic І for ASCII I, earning full credit under the check Heiner describes. The following Python example isolates the same character-count mechanism with a short teaching string:
python
ASCII_I = "\u0049"
CYRILLIC_I = "\u0406"
def passes_character_limit(text: str) -> bool:
return text.count(ASCII_I) <= 1
original = "I arrived. I waited."
substituted = original.replace(ASCII_I, CYRILLIC_I)
assert not passes_character_limit(original)
assert passes_character_limit(substituted)
The substitution changes the code points without doing anything to improve the writing. More fundamentally, this predicate cannot establish that a story was written. Closing the Unicode loophole alone would leave that missing requirement unresolved.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Quality control is part of measurement
Even a sensible benchmark design requires operational discipline. Heiner describes APEX as a retrieval task in which an agent receives files and answers questions about them. In the examples he criticizes, file contents disagree with the expected answers in the rubric. An agent can therefore follow the supplied evidence and receive a negative score for doing so.
He also reports obvious placeholder values and nonexistent dates or places. These suggest synthetic construction, though they do not establish how the data was produced. Their evaluation cost is more direct: recognizably artificial inputs can tell the model that it is being tested, creating evaluation awareness, while also moving the task away from the distribution of real working data. Bad inputs and inconsistent rubrics are benchmark-creation failures; labs can then compound them through how they optimize and report results.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Optimizing past what people prefer
The reason to evaluate models is ultimately to understand how well they serve people. Having humans inspect and rate responses provides a direct signal, but it is expensive. Benchmarks often try to distill that judgment into a cheaper, more scalable proxy. Their usefulness depends on how faithfully the proxy preserves the human judgment it replaces.
A rising benchmark score can coexist with flat or declining human preference. At first, optimizing the proxy may improve both. Later, benchmark performance can keep increasing after human evaluation stops improving; pushed further, it can improve while human ratings fall. Marketing pressure, organizational politics, and internal incentives can make continuing that optimization attractive despite the divergence.
Heiner illustrates the mismatch with a simple prompt: “What time is it?” The displayed answer expands it into an elaborate response with bold text, emoji, bullet points, and multiple options. He describes it as an Arena success that a human evaluator would reject. The example makes the failure concrete: presentation features associated with winning a comparison can overwhelm the small, direct task the user actually gave.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Anonymous voting and selective disclosure
Heiner then describes stories he has heard about hiring crowdsourced voters to favor a model on LM Arena, alleging insufficient voter filtering. This is a hearsay account, not a demonstrated incident. The proposed mechanism would defeat anonymity through an output watermark: the model emits a recognizable signal, and paid voters use it to identify which response to select. Hiding the model’s name would not prevent coordinated voting if the response itself reveals its identity.
Evaluation conditions create another route to misleading comparisons. Scores collected under different or unrepresentative conditions become difficult to interpret when those conditions are not disclosed. Heiner connects this problem to the Arena research in The Leaderboard Illusion: repeated private testing and selective disclosure can let a lab optimize against the evaluation while outsiders see only the selected result.
Heiner cites a chart reporting that Meta tested 27 private model variants before the Llama 4 release without disclosing that testing. The number concerns private variants, not publicly released models or a quantified amount of score inflation. The interpretive problem is selection: an apparently comparable leaderboard entry may represent a substantially different testing and optimization process from another entry.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build the benchmark around deployment
Raising the standard starts with people who understand the work. Human experts should shape the tasks, the definition of success, the input files, and the available tools. Their involvement cannot be confined to answering questions after the benchmark’s structure has already been decided.
For a medical benchmark, doctors who can answer clinical questions supply necessary expertise, but not all the expertise required to evaluate hospital deployment. Whether an agent is ready to work in a hospital also depends on the business setting, regulatory environment, and legal requirements. Those conditions change which tasks are appropriate to test and what successful behavior means. A benchmark of medical knowledge and a benchmark of deployment readiness therefore need different task design.
The environment must then support that design:
- Faithful inputs: Prefer data drawn from real work and created by actual people. Synthetic inputs are possible, but Heiner emphasizes how difficult it is to make them reliably realistic.
- Working tools: Tools should perform their advertised functions. Unless handling broken tools is the capability under test, tool bugs add noise rather than useful difficulty.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Align the prompt, verifier, and apparent ceiling
Prompt–verifier alignment must work in both directions. Every requested behavior needs appropriate verification, and every enforced requirement needs to correspond to the task the model was actually given. Missing coverage allows shortcuts such as passing a character check without writing a story. Extra requirements punish legitimate behavior, as in the unspecified phone-number format. Both introduce noise into the score.
Thorough quality control is needed across the benchmark, together with a private holdout set to resist contamination. Otherwise, an apparent capability ceiling may actually be a defect ceiling. In Heiner’s illustrative saturation scenario, a lab stops around 80% because the remaining 20% of tasks are broken. That is different from exhausting the model’s capacity to improve on valid tasks.
The difficulty is that a lab may not know which tasks are broken until it has solved the others. During the climb, defective tasks still contribute rewards and penalties. If those errors systematically favor some behaviors over others, they can distort relative model rankings rather than simply making every score a little less precise. Saturation can thus expose a measurement problem that was affecting comparisons all along.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Paying for the judgment the task requires
Surge’s Hemingway Bench applies this reasoning to writing. Heiner argues that writing is too rich, nuanced, and human to assess adequately through mechanical checks. He also rejects relying on LLM judges for this purpose because, in his view, they lack the writing taste needed to evaluate progress beyond their own capabilities. It is the earlier frontier problem again: a system’s existing judgment may not be sufficient to measure improvement beyond it.
Heiner describes a workforce of thousands of professional writers spanning technical writing, poetry, journalism, and editing. Professional writers make blind model comparisons, which are used to produce the leaderboard. The workforce description is his account of Surge’s approach; it should not be confused with a count of individual judges participating in any particular benchmark release.
This does not eliminate the cost problem introduced at the beginning. Professional judgment is expensive, and Heiner explicitly accepts that expense as the consequence of prioritizing evaluation quality. For writing, Surge’s response is to retain the human judgment that a cheaper proxy would otherwise attempt to replace.
Benchmaxxing exploits the gap between a benchmark and human preference. Closing that gap requires responsibility from both sides: benchmark makers must build measurements worth optimizing, and the people reporting results must hold those measurements and their conditions to a higher standard. Heiner includes himself among the benchmark creators who owe the industry that standard.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Surge explains its professional judges, writing prompts, blind comparisons, and examples of evaluation disagreements.
Code and instructions for evaluating agents on simulated business workflows, with task assertions and result visualization.
Research on private model testing, selective score disclosure, and unequal access to Arena evaluation data.
The original paper explaining IFEval's approach to automatically checking verifiable instructions.
Google Research's implementation and benchmark data for instruction-following evaluation.
The benchmark's original human-screening methodology, evaluation infrastructure, and acknowledged limitations.
Read the complete timestamped transcript
- 0:00
[upbeat music] Let's get started. When will the benchmaxxing plague end?
- 0:19
In the tech industry, we love a hype cycle, and in AI, we really love a hype cycle. And the way we do that is when a model comes out, there's a big announcement, there's a lot of benchmarks cited.
- 0:32
Sometimes, to keep things interesting, we do a little chart crime, and then people actually go and use it. And if the expectations aren't met by the reality, then we have allegations of benchmaxxing.
- 0:46
Benchmaxxing, of course, being when labs are training too hard on benchmarks in a way that deviates from what people actually care about. So the existence of that term indicates that we have a sense that benchmarks don't always equal reality.
- 1:00
And so in this talk, we're gonna figure out why does benchmaxxing happen? Why are traditional benchmarks not always accurate reflections of real-world value? Is this intrinsic to all benchmarks, and will we ever know which models are best?
- 1:15
And the answers are incentives, poor methodologies, no, and yes. All right, that was my talk. Thank you so much for coming. Um, actually, it looks like I have a few extra minutes, so let's, let's move on.
- 1:27
I have a few extra slides we'll, we'll go through.
- 1:31
So we have a sense that benchmarks don't equal reality, but the industry is dominated by a lot of popular but very bad benchmarks. So there's millions of dollars on prediction markets being wagered on LM Arena outcomes, even as we have industry leaders openly bragging about gaming LM Arena.
- 1:52
And you have thought leaders like Wern saying it can be easily gamed. It's past time for the LM Arena people to sit down and think about whether they're doing more harm than good.
- 2:02
Andrej Karpathy had a similar observation when he noticed that the models that he thought were best were not lining up with what LM Arena was ranking. And he said, "Unfortunately, the teams are not getting better models overall, but better LM Arena models, whatever that is.
- 2:17
Possibly something with a lot of nested lists, bullet points, and emojis." So why does this happen, that sort of industry insiders are telling us that this benchmark is not useful, but it still gets a lot of play?
- 2:30
The problem is that AI is aimed at everyone in the world, is, is something everyone in the world can use, and so everyone needs some tool to figure out which models are best, and benchmarks are what we have for that.
- 2:43
But if you can't-- If you don't have the ability to assess if a benchmark is good, what you do have is the ability to assess what's popular. And this creates this avalanche, this feedback effect, where the conversation is very much driven by incumbency and marketing, and less by real-world value.
- 2:59
And even myself, right? Like, unless I actually look at a benchmark in a fair amount of detail, I don't have an opinion on it. So it's a very challenging problem.
- 3:08
So what are the things that benchmarks do that lead to these problems?
- 3:14
There are a handful of key anti-patterns that we're gonna go through.
- 3:19
The first is price. Let's say you want to make an agentic coding benchmark, which these days is a very popular thing to want to do, and you want a thousand tasks in your benchmark.
- 3:28
Each task takes sixty hours to make. Each software engineer in your workforce costs half a million a year. That's fifteen million dollars to make your benchmark. And if you think that over time, about a third of those tasks are gonna get washed away every year due to models getting better, that's five million dollars to replace them.
- 3:48
So that puts you out of budget for most projects. So then people turn to a variety of workarounds that have their own problems, one of which is trying to use a lot of AI assistance, which ultimately does not really work.
- 4:02
Like, you can't push the frontier forward from within the frontier. You need to inject that external human expertise, and it needs to be good expertise. If you try to use cheap labor, you're gonna get what you paid for, and the whole result is not gonna be that useful.
- 4:19
At Surge, one of our differentiators has long been that we are not trying to minimize cost. We are trying to maximize quality. And part of that means paying a lot of money for good workers.
- 4:31
We've always believed that, but especially in twenty twenty-six, models are just beyond the point where you can make do with anything less than the best workers.
- 4:41
Contamination is often thought of as when labs are explicitly training on the test set, and that does happen sometimes. But really, contamination is the default outcome unless you are very, very good.
- 4:53
So labs put a lot of effort into holding back this flood of data that's gonna contaminate their models.
- 5:00
But inevitably, if you have public questions and answers on the internet, that's gonna get memorized to some extent. So Sweetbench Verified, here's an example prompt. You can give Opus the first part of the prompt, and it will verbatim spit out the rest.
- 5:16
It does that with the answers as well.
- 5:20
And we actually did an investigation where we compared looking at the repos that Sweetbench Verified was built out of,
- 5:27
how much has Opus memorized the Sweetbench Verified contents versus the rest of the repo? And we found very clear evidence that Opus had memorized a lot of Sweetbench. In the most recent model card, Opus 4.8 talks about its Sweetbench score.
- 5:41
It does not disclose this contamination. We as an industry aren't really in the habit of doing those disclosures. And so what that means is that as benchmark consumers, we're just missing that information.
- 5:54
Reward hacking is also a big problem. Reward hacking is basically when a model finds a lazy and creative way to meet the letter of the law, but not the spirit.
- 6:04
You need to think about designing your rewards as a adversarial process against this maximally lazy agent. Gradient descent is basically like water flowing downhill, looking for the path of least resistance, and so your verifiers need to be robust to that.
- 6:23
Another key challenge is simply just not having the ambition to make a sophisticated enough benchmark. Automation Bench tests that agents are able to make tool calls in an enterprise environment.
- 6:35
The problem is that a lot of the verifiers are these hard-coded string matches. And so you'll see it for things like phone numbers, where there are many different acceptable phone number formats, but this verifier just picks one, and the prompt doesn't tell you which one it is.
- 6:51
So the result of this is that Haiku and Fable both score twenty percent on this task. Haiku scores twenty percent because it makes a bunch of mistakes, and Fable scores twenty percent because it gets it right eighty percent of the time, but then just happens to pick different formats.
- 7:05
So if the benchmark task is not differentiating between Haiku and Fable, it's not a useful task.
- 7:11
And more broadly, in twenty twenty-six, many of us in this room are looking towards AI that's about to remake entire industries, and benchmarks are ideally our lighthouse on the horizon to let us know when that's coming, and a simple hard-coded string match is just not going to do it to measure that sort of impact.
- 7:32
Another important aspect of a good benchmark is taste. Perhaps it used to be the case that benchmarks were these dry academic, you know, question and answer sets, but nowadays, a benchmark is an artifact expressing what-- It's an, it's an aspirational artifact.
- 7:48
It's an expression of values of what you want your AI to do and how you want it to behave. And so you need to have some product sense in this process, some sort of a sense of what you want the AI to do, and that sense is unfortunately missing from IFEval.
- 8:02
IFEval has been cited on many model cards, and the way it was constructed was taking a bunch of arbitrary prompts that no user has ever asked in earnest and mashing them up with a bunch of other prompts to create a prompt set.
- 8:16
The problem is that because no user actually has asked, "Do not use any commas in your response," or, "Use the letter T at most once," you have to believe-- For this to be useful, you have to believe that there's a generalization from this to actual things that users are gonna ask.
- 8:33
IFEval just happens also to have a bunch of prompts that are fully unsolvable due to having contradictory instructions. So this one starts by saying, "Repeat this response verbatim," and it ends by saying, "Translate this into Hindi."
- 8:45
Obviously, you can't do both of those at once.
- 8:49
Here's one that says, "Write a riddle that includes exactly one bullet point. Make sure to inc-include a few bullet points." Again, this is just fully impossible.
- 8:59
It uses a sentence splitter that does not align with how humans would actually split the sentences,
- 9:06
and a lot of the prompts are not fully verified. So this one says, "Write a story." There's nothing in the verifier that checks that a story was written. It just checks that the ASCII character I is not used more than once, which means that all of these responses get a full score, including response D.
- 9:24
The way it gets a full score is by reward hacking and using the Cyrillic I character instead of the ASCII I character. IFEval is totally fine with that.
- 9:35
Another challenge is operational ability. Making a big benchmark requires a lot of QC work, and plenty of organizations just don't make that investment. APEX is a RAG benchmark where the agent is given files and then asked questions about them.
- 9:52
And in some instances, what's in the file and then what's expected in the rubric don't line up. So an agent that does the thing that it's seeing in the ground truth is gonna get a negative score.
- 10:05
And a lot of the data in APEX is seemingly synthetically generated because it's full of obvious placeholder values or dates or places that don't exist.
- 10:17
And so as a result, the model is more likely to develop eval awareness, where it realizes that it's being tested, which undermines the entire exercise. It also just takes you out of distribution from actual real-world data to something that is obviously fake.
- 10:34
So that's an overview of some of the key anti-patterns that happen during benchmark creation. But benchmaxxing is a two-way process, and there are all sorts of fun things that labs can do to benchmark, and that's what we're gonna talk about next.
- 10:51
So the, the core value that we're all trying to get towards is human eval, right? AI exists to serve humans, and so just having humans look at the responses and make ratings, like that's what we care about.
- 11:03
The problem is that human eval is very expensive, and so a lot of what benchmarks are doing is trying to get around that, and you are trying to distill human preference into something more scalable, and you're hoping you do that distillation in a way that's still sufficiently faithful to what human eval wants.
- 11:18
But what this means is that inevitably there is a point where you can keep hill climbing on a benchmark and the human eval stays flat.
- 11:27
And you can actually take it even further if you want, where you keep hill climbing on a benchmark even as the human eval goes down. But if for whatever reason you think this is necessary for marketing or we have sort of organizational politics or incentives that are demanding this, that's how it can end up happening.
- 11:44
In this instance, the prompt is, "What time is it?" And the response is absolutely deranged. No human eval is ever going to choose this. But LM Arena puts it at the top of the leaderboard.
- 11:55
So again, you have this divergence, and if you're trying to benchmax, you just cannot care about that. Another thing you can do that I've heard stories of is you can actually hire a crowdsourced army to vote for you on LM Arena, since LM Arena basically does no filtering of their workforce.
- 12:11
And you might say, "Well, we anonym-- You know, LM Arena anonymizes, so how are they gonna know who to vote for?" That's actually quite simple. You have your model include a watermark that tells the crowd who to vote for.
- 12:25
There's also all sorts of things you can do with running your evals in conditions that are, like, not fully representative of the apples to apples comparison you're trying to make, and then not always being super transparent about those conditions in such a way that undermines the validity that the community is trying to interpret because they don't have
- 12:44
that contextualizing information. This was a paper, um, again, about LM Arena and talking about how some of the dynamics of how it's run lead to models overfitting on LM Arena.
- 12:55
Um, in this instance, the specific chart we're seeing is that Meta tested twenty-seven models without disclosing that it was doing so, um, which, you know, distorts the results.
- 13:09
So how are we gonna end benchmaxxing? We need to hold the benchmark industry and the labs to a higher standard.
- 13:18
The first thing we need to do when making a good benchmark is start with great human experts, and those experts inform everything that is downstream from what types of tasks are we going to have the agent do, how is success measured,
- 13:34
what are the input files that agents are given, what are the tools that they're given. But we also do need that product sense. So imagine you're making a medical benchmark.
- 13:42
It's not enough to have doctors who can answer specific medical questions because if you're trying to test how ready are we for agents to be deployed into hospitals, you also need someone with the business sense to know what's the regulatory environment, what's the legal requirements, because that is gonna impact what types of tasks you're trying to have
- 13:59
the AI solve. You need high-fidelity input data, which is best done by going out and getting it from the real world, having actual people create this data. Synthetic approaches are possible, but it is very, very hard to do it reliably.
- 14:15
The tools need to actually work. A lot of benchmarks have tools that are buggy in various ways, and unless you're intentionally making a benchmark about buggy tools, this just introduces noise.
- 14:26
You need verifiers that are fully aligned with the prompts, and this is a two-way alignment. So the verifiers need to be verifying everything the prompt asks for, and everything the prompt asks for needs to be covered by the verifiers.
- 14:40
And if you get either side of those two misaligned, then it's gonna be unfair to models and you're introducing random noise.
- 14:46
You need to thoroughly QC everything, and you need to have a private holdout set so you don't get contaminated.
- 14:54
And if you do all this right, then you'll avoid what often happens with benchmarks, which is when labs get to, like, eighty percent and say, "Okay, this is saturated."
- 15:03
And I used to think that saturation was just them saying, again, "We don't think training on this further is going to increase real-world value." And it often does mean that, but it can mean that because the lab is saying, "We realize twenty percent of these tasks are broken."
- 15:18
But the problem is that as you're hill climbing, you don't know what twenty percent are broken until you solve all the others. And so as a result, you have a lot of noise, and if that twenty percent of broken tasks is randomly, but in a biased way, assigning the rewards, it's gonna really distort the model relative ranking
- 15:38
you're trying to get. So at Surge, we created a benchmark called Hemingway Bench to measure writing. There have been a number of writing benchmarks that each use various mechanical means to try to assess writing quality, but we believe that writing is just too rich and deep and nuanced and frankly human of an activity to measure with mechanical
- 16:00
benchmarks. And LLM as a judge doesn't really work either because LLMs don't have good taste in writing. Again, this is sort of the you can't expand the frontier from within the frontier situation.
- 16:10
So what we've done is we've just created a workforce of thousands of professional writers in various domains, technical writers, poets, journalists, editors, and we just have them do blind model comparisons, and then we create this leaderboard.
- 16:26
And it is quite expensive, right? Human eval is very expensive. Getting the time of these professionals is quite e-expensive. But again, our goal is to maximize quality, not to minimize costs.
- 16:38
So in conclusion, benchmaxxing is the exploitation of benchmark misalignments between human preference, but we can do better, and we can hold the industry to a higher standard, both the people making the benchmarks, like myself, and the people who are reporting on the benchmarks.
- 16:55
And if you'd like to be a part of that, of course, obligatory pitch, at Surge, we're hiring for basically all aspects of that. Uh, and if you'd like more spicy takes from me, uh, please follow my Substack.
- 17:04
Thank you very much. [audience applauding] [upbeat music]