AI Engineer World's Fair 2026
Benchmarks: The Good, the Bad, and the Ugly
Read the talk
What a Coding Benchmark Actually Measures
A prompt nobody would write exposes a larger problem: coding benchmarks earn trust only when their instructions, tests, and execution environments measure useful engineering work.
From a talk by Ali Khial
Before you start: Familiarity with software tests and coding agents is helpful; no machine-learning background is required.
Would an engineer write this prompt?
Would an engineer ever write the instructions used to evaluate a coding agent? Ali Khial approaches that question as a software engineer. Introducing himself as director of AI/ML at G2i, he jokes about having no ML experience and more than fifty abandoned side projects. His planned classification of good, bad, and ugly benchmarks became an investigation into how benchmarks work. The opening exhibit is one task prompt spread across three screenshots. He shows it to three of his best engineers and asks whether they would write anything like it. All three say no.
That mismatch raises a more basic question: what is a benchmark measuring? Before answering it, Khial encounters a wall of terminology—graders, long horizon, verifiers—and works with his team to reduce it to a small set of components.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The machinery behind a score
The benchmark pipeline is straightforward:
- A prompt or instruction defines the task.
- A model or agent receives it and produces a solution.
- Verifiers and rubrics check and grade that solution.
- A harness controls the environment around the process, limiting external factors.
- The run produces trajectories, scores, and metadata that can support model rankings.
A trajectory records the path through the task; a score compresses the outcome. Both depend on what the agent was asked to do, how its work was judged, and what its environment allowed.
A useful ranking depends on all three: good instructions, appropriate grading, and a controlled environment. If those components work, the resulting measurements should be informative. Khial’s investigation follows the places where that expectation breaks down.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Instructions can give away the work
Khial reports finding an average of 481 words per instruction in a quick investigation of SWE-Bench Pro. He characterizes that as roughly two pages per task. The dataset version, included fields, and counting method are unspecified, but the practical concern is clear: an unusually elaborate specification may test a different interaction from the one engineers expect to have with an agent.
His Go regular-expression example makes the concern more concrete. The instruction points directly to a test file, giving the model a route to discover what the evaluator expects and implement against it. A second excerpt supplies a complete implementation interface, sharply constraining the model’s design choices. Khial calls this a leaky prompt: part of the work of discovering requirements and choosing an implementation has already been done for the agent.
A task can avoid that problem and still raise a question about practical value. The SWE-Marathon example asks for a C compiler written in Rust. Khial credits the prompt with being well formed and abstract enough to leave the model room to work. His objection is its economic relevance: he does not see building that compiler as a useful proxy for the work he wants to delegate. That is a judgment about task selection, separate from whether the task is difficult or clearly specified.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When passing and correctness diverge
Weak verification creates errors in both directions. Citing a comparison by DeepSWE, Khial reports 8.5% false positives and an apparent 24% false negatives for SWE-Bench Pro. His spoken correction makes the second figure uncertain; Datacurve’s research listing instead summarizes the figures as 8% and 25%, without enough methodology to reconcile them or establish their denominators. The distinction between the two failure modes is more secure than the exact rates.
| Failure mode | What the verifier does | Consequence |
|---|---|---|
| False positive | Accepts an incorrect implementation | Gives credit for work that is not correct |
| False negative | Rejects a correct implementation | Penalizes a valid solution |
Khial then examines how a false negative can arise. One test expects a particular variable to exist, even though the instruction never requires that name. An agent could satisfy the requested behavior and still fail because it chose a different internal representation.
The displayed excerpts highlight re and AllowUnexported. Khial also describes a test that checks unexported functions. His objection is that these checks bind the evaluation to internal implementation choices. He says his own projects would reject such tests in pull-request review. The engineering question is whether a test enforces a stated requirement or quietly adds one.
For a small Go regex example, a behavioral test can leave the compiled expression’s variable name unconstrained:
go
package matcher
import (
"regexp"
"testing"
)
func Match(pattern, value string) (bool, error) {
compiled, err := regexp.Compile(pattern)
if err != nil {
return false, err
}
return compiled.MatchString(value), nil
}
func TestMatch(t *testing.T) {
for _, tc := range []struct {
value string
want bool
}{
{"aaa", true},
{"bbb", false},
} {
got, err := Match("^a+$", tc.value)
if err != nil || got != tc.want {
t.Fatalf("Match(%q): got %v, %v; want %v",
tc.value, got, err, tc.want)
}
}
if _, err := Match("[", "aaa"); err == nil {
t.Fatal("expected an error for an invalid pattern")
}
}
Here the example contract covers matching, nonmatching, and invalid patterns. Renaming compiled does not affect the result. This illustrates the distinction in Khial’s critique: evaluate the required behavior without making an unstated internal name part of correctness.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An agent can solve the evaluator’s problem instead
Reward hacking exploits a gap between the intended task and the route to a favorable score. Instead of developing a patch, an agent might search .git directories or the internet for traces of an existing solution. The environment has then supplied a shortcut around the capability the benchmark intended to measure.
Khial describes the displayed graphs as showing increasing reward hacking across newer model versions. The graphs’ exact rates and the distinction between attempted and successful exploitation are not established here. His interpretation is that smarter agents are desirable, but benchmark safeguards are failing to keep pace with their ability to find alternative routes to a reward.
These quality problems become a trust problem. In Khial’s experience over the preceding six months, he had not met an engineer who selected a model on leaderboards alone. Engineers looked at the rankings, then ran their own experiments. G2i’s team had spent the preceding two months developing principles intended to make benchmark tasks more useful to those engineers.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Express intent, then test behavior
The first principle is human instructions: authored by humans and reviewed by humans. Instructions should express desired behaviors, objectives, and hard constraints. They should leave implementation decisions open unless those decisions are themselves requirements. Trying to make every task completely self-contained by adding more detail can erase the judgment the agent was supposed to exercise.
The second principle is holistic graders. Cover a broad behavioral surface without prescribing the internal solution, then add precision where failure is especially consequential:
- Security and business logic: use unit, integration, and end-to-end tests to check correctness at multiple levels.
- Other software behavior: choose coverage that provides useful confidence without insisting on every possible internal detail.
Khial rejects pursuing 100% coverage everywhere as inefficient. The goal is to direct testing effort toward meaningful failure modes, while allowing different valid implementations to pass.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make success useful and explain what it means
The third principle, production-grade tasks, asks whether success would change an engineer’s willingness to delegate real work. Constructing a task that makes an LLM fail proves that the model has a limit. It does not establish the task’s economic value. A useful task supports a more practical inference: success on this problem gives an engineer reason to trust the agent with a related problem.
The fourth principle is contamination-free by design: create novel tasks and maintain private holdout sets. Khial contrasts that approach with tasks drawn from GitHub and other public repositories, where prior exposure or discoverable solutions can weaken the evaluation. His characterization of existing benchmarks as entirely public is too broad—SWE-Bench Pro also includes inaccessible held-out problems and a commercial set from proprietary repositories. The useful design requirement is to protect the evaluation from prior exposure, rather than assume a familiar repository task remains unseen.
The fifth principle is information above leaderboards. A benchmark should help someone make a decision, not merely announce a winner. Khial describes this as putting the X-axis back on the first page: make the context behind a score visible. Runs already contain trajectories and other useful data, but readers often have to dig for it or repeat experiments themselves. Surfacing that information is part of the benchmark’s job.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Look under the hood
Khial replaces a planned lofty ending with an invitation to software engineers: inspect benchmarks, understand their machinery, and contribute. His closing invitation to join the Discord rests on a practical point. Engineers already review requirements, question brittle tests, and judge whether a piece of work has value. Those same skills are needed inside benchmark design.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The benchmark's original paper describes its task construction, repository coverage, and public, held-out, and commercial evaluation sets.
Datacurve's engineering benchmark presents original tasks and a live leaderboard with cost, output-token, and agent-step views.
A project-scale coding benchmark covering compiler construction, application clones, and other extended tasks, with multiple verification methods.
Further reading
OpenAI's July 2026 audit examines overly strict tests, missing requirements, weak coverage, and misleading prompts in SWE-Bench Pro.
Read the complete timestamped transcript
- 0:00
[outro jingle] Hello, everyone. Um, this is the last talk of this session, so hopefully it's gonna be short.
- 0:17
I know that you guys had to go through a long day, so try to keep it short and light for you all. Um, I'm gonna present myself. Um, I'm Ali.
- 0:26
I'm the director of AI/ML at G2i. Um, I have zero experience in ML, so I don't know why they put the ML in my title. I'm a software engineer, uh, at heart, and to prove that I have more than fifty abandoned side projects in my machine.
- 0:40
So, uh, you can know. So, uh, I'm gonna make a disclaimer. The, the title of the, the presentation is a little bit misleading. Uh, as I was working on it, I realized that it would be better if I presented my journey, uh, into benchmarks and what I learned instead of trying to find a dichotomy of the, the
- 1:00
bad, the ugly and, and the good. So, um, let's start with, um... I wanna grab your attention, and I invite you to look at this.
- 1:10
These beautiful three screenshots are a single prompt on one of the benchmark tasks.
- 1:16
As I was looking at it, I was like, "How can an engineer write a task like this?" So I said, "Nah, it's impossible. No one writes prompts like these ever."
- 1:26
But I wanted to double-check with my engineers, so I took three of our best engineers, I showed them the prompt, and I said, "Would you ever write a prompt like this?"
- 1:34
And the answer was no. And they're right. They shouldn't. And so
- 1:43
at that point, um, I was like, "What is a-- what are benchmarks anyway?" Uh, I needed to take a step back. I needed to look more. I need to understand.
- 1:52
And so as I was researching, I faced a wall of keywords. Um, graders, long horizon, verifiers, bench- benchmarking, and a lot of jargon. So I was like, either this is too complicated or, um, there's a lot of jargon and a lot of, um, words to, to, to work through here.
- 2:16
So, um, I worked through it, worked with my team. I have a lot of good researchers in the team, and we, uh, kinda like nailed-- like simplified to the most basics.
- 2:27
Um, and so the way I see it is that it starts as a prompt or an instruction. That prompt is fed to models and agents.
- 2:40
Agents provide solutions. Those solutions are verified, uh, and graded through verifiers and rubrics. All of that is wrapped in a harness that's, that's preventing it from, um, from the external factors.
- 2:56
And if it all goes good, uh, we have, um,
- 3:00
trajectories, scores, and, um, metadata that we can use, uh, to, to, to verif-- to
- 3:09
basically, uh, rank, uh, models. And so the equation is simple. If prompts and instructions are great, and verifiers are-- and rubrics are doing their job while the harness is preventing, uh, or creating an environment that is good for a benchmark, we should have amazing results.
- 3:31
Um, but that's not the reality. So what, what, what went wrong?
- 3:37
So the first thing is, when looking deeper in benchmarks, uh, most of the instructions are unrealistic. Um, I did a quick research on SWE-Bench Pro, and, um, there's four hundred and eighty-one words per, per instruction in average.
- 3:55
That's a two pager, two-pager per task. That is not how people write prompts.
- 4:01
And to illustrate more of that, um, I took a couple examples here. The first one I looked at, I, I call the leaky prompt. It's a, uh, goal, um, task that's basically, um, that's bas-- trying to match in some regex and doing tests on, on some regex.
- 4:18
So in the first screenshot here, um, the instruction is pointing directly to the test file, which basically means that the LLM has all the ingredient it needs to go and find that test file and implement based on that.
- 4:31
The second one is, is even worse. Um, it's basically providing a complete interface of the implementation, basically locking the LLM from any kind of, uh, creativity, and it's forcing it to do it that way.
- 4:46
So that's the leaky prompt. The second example, it's the, the not economically valuable prompt. Uh, this is from SWE marathon, and this prompt is well-formed. Uh, it's, it's,
- 5:02
it's abstracted enough to allow for the LLM to do its work, but it's asking it to build a C compli- compiler in Rust. So I don't know if any of you ever tried to do that, but I don't think it's a good idea.
- 5:12
We should not do that. All right, moving on. The second problem, weak verifiers. Um, so the screenshot here is, is a, uh, is the work that DeepSWE, um, did, uh, to compare their, uh, their bench against SWE-Bench Pro.
- 5:32
And, um, let me just fix here so I can see the numbers.
- 5:39
In SWE-Bench Pro, eight point five of-- eight point five percent of all the tasks, uh, accepted wrong implementation in one hand, and more than twenty, twenty-four percent of the tasks, uh, rejected, um, correct implementations.
- 5:56
And so I kinda went again, dug a little bit, and I extracted one of the tasks, and I started looking at it. Um, and, and here's, here's what's happening in the example of, uh-
- 6:08
Reje-rejecting, um, possibly rejecting good, good answers. So in this example, the test is, is basically expecting a variable to exist. But that variable is first not specified in the instruction, and two, why would we expect an LLM to write the, the variable name this way?
- 6:28
So this test is cornering the LLM and basically, uh, causing, uh, those false negatives. In the other example, it's bas-- the test is basically checking functions that are unexported.
- 6:44
So if that was a PR in any of our projects and, uh, exposed these type of tests, we would not accept it. So this is what a weak verifier looks like.
- 6:58
All right, moving on. Re-reward hacking. So what's happening is models are becoming increasingly, increasingly able to optimize and figure out solutions to hard problems by going around the problem.
- 7:12
So instead of actually trying to, to fix the-- to, to, to apply a patch to a task, they try to go and find .git folders, or they look up the internet for any kind of traces that would allow 'em to, um, to do the task.
- 7:29
And this first graph here shows like-- shows that as models evolve, they are now more s-smarter and smarter in being able to do reward hacking. But
- 7:42
that's what we want. We want LLMs to be smart. The benchmarks are lacking behind, and they're not preventing from, from that to happen. Um,
- 7:52
more in detail, as you can see here, the more you go in time and the more you have new versions, the delta of, um, of, um, reward hacking is increasing.
- 8:06
So the conclusion here is there's a quality gap, and it's causing a trust gap. I have not met an engineer in the last six months that would choose a model or choose, um, an LLM based on the leaderboards.
- 8:21
Um, they look at them, there's a lot of hype, but then they move on, and they test things by themselves, and they apply that.
- 8:32
So how do we close the gap? Um, in the last two months, we've been working with our team at G2i to basically try to define a framework, uh, a set of principles that would allow us to build tasks for benchmarks that are, um, better than what we have today.
- 8:54
The first one, human instructions. Authored by humans, reviewed by humans. This is basically the entry point for any great tasks. The instructions given to an agent or an LLM should lean towards expressing desired behaviors, objectives, and hard constraints, not implement details
- 9:14
or try to guarantee self-containment when the task itself is, is expressing too much, uh, too, too much details.
- 9:24
The second principle is holistic graders, behavioral tests in one hand and then precision what ne-- where needed. This is very similar to how we approach, um, tests in engineering.
- 9:37
We want to have the most surface covered without being too prescriptive, but we also want to be precise where needed. So for security issues or business logic, we wanna have the whole stack unit, unit test, integration test, and then end-to-end tests.
- 9:53
But for the rest of the, the rest of the, the software, we don't wanna have a hundred percent coverage because that's, um, not efficient.
- 10:05
The third principle, production-grade. The tasks have to be--
- 10:10
that's-- have to have value, um, and they have to be eco-eco-economically valuable. Um, it is one thing to have a, a test, a task that is failing the LLM and proving that the LLM is not there yet.
- 10:24
It is another for-- It is another thing for an engineer to look at a task and say, "If the LLM is fixing this, I trust it to fix that."
- 10:33
Currently, we don't have that, so production-grade. The fourth principle, contamination-free by design.
- 10:44
We wanna do novel tasks only, and we wanna make sure that we keep pr-private holdout sets. This is
- 10:53
a principle that is very important as currently
- 10:58
the tasks that are existing in benchmarks are all pulled from GitHub repos or from, um, from, from public repos. So our approach here is that it should always be novel.
- 11:10
This way it's contamination-free by design. And the fifth and last principle here is information above leaderboards. Um, the benchmark needs to tell a story and needs to help people make decisions.
- 11:23
Leaderboards are what we see in benchmarks today. They tell you who wins, but they don't tell you why. And so we wanna basically put the X-axis back on, on, on, on the first page.
- 11:37
Uh, the idea here is that there's, um,
- 11:42
there's a lot of, um, data that we can extract from those-- these runs, and unfortunately, they're not being put, uh, in the forefront. And people have to dig, uh, a lot and do their own experiments to get to those data points.
- 11:58
And so finally, uh, initially I wanted to have a kind of a, a lofty like ending to this, but I think I, I, I pivoted to something more interesting.
- 12:08
Uh, this is a call to action to software engineers. Um,
- 12:13
benchmarks are not hard. We need to look under the hood, and we need to understand them and join the Discord because engineers' input is valuable.
- 12:24
And thank you. [audience applauding] [upbeat music]