# Benchmarks: The Good, the Bad, and the Ugly

Ali Khial

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/jWq-aZIU0kM-benchmarks-good-bad-ugly) · [Video](https://www.youtube.com/watch?v=jWq-aZIU0kM)

## Benchmarks: The Good, the Bad, and the Ugly

A prompt nobody would write exposes a larger problem: coding benchmarks earn trust only when their instructions, tests, and execution environments measure useful engineering work.

Familiarity with software tests and coding agents is helpful; no machine-learning background is required.

### Would an engineer write this prompt?

[0:26](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=26s)

Would an engineer ever write the instructions used to evaluate a coding agent? Ali Khial approaches that question as a software engineer. Introducing himself as director of AI/ML at G2i, he jokes about having no ML experience and more than fifty abandoned side projects. His planned classification of good, bad, and ugly benchmarks became an investigation into how benchmarks work. The opening exhibit is one task prompt spread across three screenshots. He shows it to three of his best engineers and asks whether they would write anything like it. All three say no.

![Slide stating “No one writes prompts like this. ever!” beside a dog reaction image labeled “WHAT?”](https://ai.engineer/api/talk-essay-media/bb699fdf2de723ea184bdbd54a11ad6b0057660a3f4114af9f36aeeb5d49e82e)

“No one writes prompts like this. ever!”

That mismatch raises a more basic question: what is a benchmark measuring? Before answering it, Khial encounters a wall of terminology—graders, long horizon, verifiers—and works with his team to reduce it to a small set of components.

### The machinery behind a score

[2:27](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=147s)

The benchmark pipeline is straightforward:

1. A **prompt or instruction** defines the task.
2. A model or agent receives it and produces a solution.
3. **Verifiers and rubrics** check and grade that solution.
4. A **harness** controls the environment around the process, limiting external factors.
5. The run produces trajectories, scores, and metadata that can support model rankings.

A trajectory records the path through the task; a score compresses the outcome. Both depend on what the agent was asked to do, how its work was judged, and what its environment allowed.

**A useful ranking depends on all three: good instructions, appropriate grading, and a controlled environment.** If those components work, the resulting measurements should be informative. Khial’s investigation follows the places where that expectation breaks down.

![Diagram connects prompts and instructions to models and agents, then solutions. Verifiers and rubrics point to solutions; a dashed harness encloses the pipeline, which outputs trajectories, scores and metadata.](https://ai.engineer/api/talk-essay-media/91345025e133697c128a86f229dc8196b205c457d628df5e68f75a085d9e8ef3)

The benchmark scaffold, with prompts, verifiers and the harness highlighted in green.

### Instructions can give away the work

[3:37](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=217s)

Khial reports finding an average of 481 words per instruction in a quick investigation of [SWE-Bench Pro](https://arxiv.org/abs/2509.16941). He characterizes that as roughly two pages per task. The dataset version, included fields, and counting method are unspecified, but the practical concern is clear: an unusually elaborate specification may test a different interaction from the one engineers expect to have with an agent.

His Go regular-expression example makes the concern more concrete. The instruction points directly to a test file, giving the model a route to discover what the evaluator expects and implement against it. A second excerpt supplies a complete implementation interface, sharply constraining the model’s design choices. Khial calls this a leaky prompt: part of the work of discovering requirements and choosing an implementation has already been done for the agent.

A task can avoid that problem and still raise a question about practical value. The [SWE-Marathon](https://www.swe-marathon.org/swe-marathon-paper.pdf) example asks for a C compiler written in Rust. Khial credits the prompt with being well formed and abstract enough to leave the model room to work. His objection is its economic relevance: he does not see building that compiler as a useful proxy for the work he wants to delegate. That is a judgment about task selection, separate from whether the task is difficult or clearly specified.

### When passing and correctness diverge

[5:12](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=312s)

Weak verification creates errors in both directions. Citing a comparison by [DeepSWE](https://deepswe.datacurve.ai/), Khial reports 8.5% false positives and an apparent 24% false negatives for SWE-Bench Pro. His spoken correction makes the second figure uncertain; [Datacurve’s research listing](https://datacurve.ai/research) instead summarizes the figures as 8% and 25%, without enough methodology to reconcile them or establish their denominators. The distinction between the two failure modes is more secure than the exact rates.

| Failure mode | What the verifier does | Consequence |
| --- | --- | --- |
| False positive | Accepts an incorrect implementation | Gives credit for work that is not correct |
| False negative | Rejects a correct implementation | Penalizes a valid solution |

Khial then examines how a false negative can arise. One test expects a particular variable to exist, even though the instruction never requires that name. An agent could satisfy the requested behavior and still fail because it chose a different internal representation.

The displayed excerpts highlight `re` and `AllowUnexported`. Khial also describes a test that checks unexported functions. His objection is that these checks bind the evaluation to internal implementation choices. He says his own projects would reject such tests in pull-request review. The engineering question is whether a test enforces a stated requirement or quietly adds one.

![Two code excerpts with red outlines around “re” and “AllowUnexported”; the lower excerpt lists matcher types and regexp.Regexp.](https://ai.engineer/api/talk-essay-media/4fc7bda1280ccf1336f14533bfa51f5251112fe48fe3a2bb573b126f817b7d4c)

SWE-Bench Pro test patch excerpts highlight implementation details.

For a small Go regex example, a behavioral test can leave the compiled expression’s variable name unconstrained:

```go
package matcher

import (
    "regexp"
    "testing"
)

func Match(pattern, value string) (bool, error) {
    compiled, err := regexp.Compile(pattern)
    if err != nil {
        return false, err
    }
    return compiled.MatchString(value), nil
}

func TestMatch(t *testing.T) {
    for _, tc := range []struct {
        value string
        want  bool
    }{
        {"aaa", true},
        {"bbb", false},
    } {
        got, err := Match("^a+$", tc.value)
        if err != nil || got != tc.want {
            t.Fatalf("Match(%q): got %v, %v; want %v",
                tc.value, got, err, tc.want)
        }
    }
    if _, err := Match("[", "aaa"); err == nil {
        t.Fatal("expected an error for an invalid pattern")
    }
}
```

Here the example contract covers matching, nonmatching, and invalid patterns. Renaming `compiled` does not affect the result. This illustrates the distinction in Khial’s critique: evaluate the required behavior without making an unstated internal name part of correctness.

### An agent can solve the evaluator’s problem instead

[6:58](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=418s)

**Reward hacking** exploits a gap between the intended task and the route to a favorable score. Instead of developing a patch, an agent might search `.git` directories or the internet for traces of an existing solution. The environment has then supplied a shortcut around the capability the benchmark intended to measure.

Khial describes the displayed graphs as showing increasing reward hacking across newer model versions. The graphs’ exact rates and the distinction between attempted and successful exploitation are not established here. His interpretation is that smarter agents are desirable, but benchmark safeguards are failing to keep pace with their ability to find alternative routes to a reward.

These quality problems become a trust problem. In Khial’s experience over the preceding six months, he had not met an engineer who selected a model on leaderboards alone. Engineers looked at the rankings, then ran their own experiments. G2i’s team had spent the preceding two months developing principles intended to make benchmark tasks more useful to those engineers.

### Express intent, then test behavior

[8:54](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=534s)

The first principle is **human instructions**: authored by humans and reviewed by humans. Instructions should express desired behaviors, objectives, and hard constraints. They should leave implementation decisions open unless those decisions are themselves requirements. Trying to make every task completely self-contained by adding more detail can erase the judgment the agent was supposed to exercise.

The second principle is **holistic graders**. Cover a broad behavioral surface without prescribing the internal solution, then add precision where failure is especially consequential:

- **Security and business logic:** use unit, integration, and end-to-end tests to check correctness at multiple levels.
- **Other software behavior:** choose coverage that provides useful confidence without insisting on every possible internal detail.

Khial rejects pursuing 100% coverage everywhere as inefficient. The goal is to direct testing effort toward meaningful failure modes, while allowing different valid implementations to pass.

### Make success useful and explain what it means

[10:05](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=605s)

The third principle, **production-grade tasks**, asks whether success would change an engineer’s willingness to delegate real work. Constructing a task that makes an LLM fail proves that the model has a limit. It does not establish the task’s economic value. A useful task supports a more practical inference: success on this problem gives an engineer reason to trust the agent with a related problem.

The fourth principle is **contamination-free by design**: create novel tasks and maintain private holdout sets. Khial contrasts that approach with tasks drawn from GitHub and other public repositories, where prior exposure or discoverable solutions can weaken the evaluation. His characterization of existing benchmarks as entirely public is too broad—SWE-Bench Pro also includes inaccessible held-out problems and a commercial set from proprietary repositories. The useful design requirement is to protect the evaluation from prior exposure, rather than assume a familiar repository task remains unseen.

The fifth principle is **information above leaderboards**. A benchmark should help someone make a decision, not merely announce a winner. Khial describes this as putting the X-axis back on the first page: make the context behind a score visible. Runs already contain trajectories and other useful data, but readers often have to dig for it or repeat experiments themselves. Surfacing that information is part of the benchmark’s job.

![Slide lists Human Instructions, Holistic Graders, Production Grade, Contamination Free, and Information above leaderboards. The fifth principle says “Tells a story. Helps decision making.”](https://ai.engineer/api/talk-essay-media/4fe79031979e30841564893a2aff5018ea1e55aad96c20b2f2a0572f387c04c9)

Five benchmark principles, including “Information above leaderboards.”

### Look under the hood

[11:58](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=718s)

Khial replaces a planned lofty ending with an invitation to software engineers: inspect benchmarks, understand their machinery, and contribute. His closing invitation to join the Discord rests on a practical point. Engineers already review requirements, question brittle tests, and judge whether a piece of work has value. Those same skills are needed inside benchmark design.

## Resources

- [SWE-Bench Pro](https://arxiv.org/abs/2509.16941) — The benchmark's original paper describes its task construction, repository coverage, and public, held-out, and commercial evaluation sets.
- [DeepSWE](https://deepswe.datacurve.ai/) — Datacurve's engineering benchmark presents original tasks and a live leaderboard with cost, output-token, and agent-step views.
- [SWE-Marathon](https://www.swe-marathon.org/swe-marathon-paper.pdf) — A project-scale coding benchmark covering compiler construction, application clones, and other extended tasks, with multiple verification methods.
- [Separating signal from noise in coding evaluations](https://openai.com/index/separating-signal-from-noise-coding-evaluations/) — OpenAI's July 2026 audit examines overly strict tests, missing requirements, weak coverage, and misleading prompts in SWE-Bench Pro.

## Slides

### Ali Khial introduction

Ali Khial introduces himself as G2i’s director of AI/ML and a software engineer at heart, joking that his 50+ abandoned side projects prove it.

[0:28](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=28s) [Ali Khial introduces himself as G2i’s director of AI/ML and a software engineer with over 50 abandoned side projects, alongside his portrait.](https://assets.ai.engineer/sites/aiecode2025/sha256/878cd96ce14940e1317d31eba474a221226e12601ac7af8d26d42d1e8fbd4007.webp)

Ali Khial
Director of AI/ML @ G2i

Software Engineer @ 💖

50+ abandoned side projects

G2i
G2i. All rights reserved.

### No one writes prompts like this

Khial showed a benchmark task prompt to three of his best engineers; all said they would not write a prompt like it.

[1:22](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=82s) [Large text reads, “No one writes prompts like this. ever!”](https://assets.ai.engineer/sites/aiecode2025/sha256/74ac0d44a2826c218da45698b5d73f4ec75f44c05e6d07902e5e31bad01039ba.webp)

No one writes prompts like this. ever!

G2i
G2i. All rights reserved.
2

### What are benchmarks?

The speaker encountered a wall of benchmark jargon, from graders to benchmaxxing, before working with the team to simplify the basics.

[1:55](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=115s) [The question “What are benchmarks?” appears beside terms including graders, LLM-as-judge, long horizon, evals and benchmaxxing.](https://assets.ai.engineer/sites/aiecode2025/sha256/33839b8eceb730da542c251b61d8db8c911f7ab0026d28d2d264c0329bc1a77e.webp)

What are benchmarks ?

Graders
LLM-as-judge
Long Horizon
Verfiers
N-gram
Fail-to-pass
Evals
Benchmaxxing
Pass-to-pass
Hill Climbing

G2i
G2i. All rights reserved.
3

### A simplistic view of a benchmark scaffold

A benchmark feeds prompts to models or agents, then verifies their solutions with rubrics. A harness surrounds the process and produces trajectories, scores, and metadata that can be used to rank models.

[2:29](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=149s) [A diagram shows prompts or instructions feeding into models or agents.](https://assets.ai.engineer/sites/aiecode2025/sha256/8f8eaafa8c46d68a66bbac610da481fcd3e06709e97b9669ca70ee2362d4ec06.webp)

Prompts /
Instructions

Models /
Agents

A simplistic view of a benchmark scaffold

G2i
G2i. All rights reserved.
4

[2:48](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=168s) [A harness encloses prompts, models, solutions, and verifiers, with trajectories, scores, and metadata shown as outputs.](https://assets.ai.engineer/sites/aiecode2025/sha256/3e54830d7bc6ddb3a3a41fc11ecdbe1b4b7fb394f24f32fba508cd5320d53f47.webp)

Prompts /
Instructions

Models /
Agents

Solutions

Verfiers /
Rubrics

Harness

Trajectories
Scores
Metadata

A simplistic view of a benchmark scaffold

G2i
G2i. All rights reserved.
4

### Highlighted benchmark scaffold

Prompts feed models that produce solutions, which verifiers assess. The harness outputs trajectories, scores and metadata for ranking models. Green highlights the prompts, verifiers and harness.

[3:14](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=194s) [The benchmark scaffold highlights prompts, verifiers, and the harness in green.](https://assets.ai.engineer/sites/aiecode2025/sha256/a22d5f73e4a1d39dd98fc8ab89b19107c9af2e733e6b8ff8c1275a15a04b6d4d.webp)

G2i

Prompts / Instructions
Models / Agents
Solutions
Verfiers / Rubrics

Harness

Trajectories
Scores
MetaData

A simplistic view of a benchmark scaffold

G2i. All rights reserved.
5

### Unrealistic instructions

The speaker found that SWE-bench Pro instructions average 481.6 words, or about two pages per task, and argues that this is unlike how people write prompts.

[3:34](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=214s) [A slide says instructions average 481.6 words, or about two pages per task.](https://assets.ai.engineer/sites/aiecode2025/sha256/ee21405d567bea4d219c0d6c60f43810f645e6f8478de954ea15eebd3a14ca35.webp)

When the instrument is measuring the wrong things

Unrealistic Instructions

481.6 words / instruction
~2 pages / task

G2i
G2i. All rights reserved.
8

Interface text:
6

### Leaky prompts and economic value

A SWE-Bench Pro prompt exposes tests and an interface, constraining implementation. A SWE Marathon task asks for a C compiler in Rust, which the speaker questions as economically valuable.

[4:49](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=289s) [Examples contrast a prompt exposing test and interface details with a task to build a C compiler in Rust.](https://assets.ai.engineer/sites/aiecode2025/sha256/bc2c3ca4b3625562823335bb520cd59aae983d357b1af3a891f4ccce10013e22.webp)

Leaky prompts

## Steps to Reproduce
1. Include an expression with {{regexp.match("foo")}} in lib/utils/parse.
1. Run the tests defined in parse_test.go (e.g., TestMatch or TestMatchers).
1. Notice that compilation fails with errors like undefined: Matcher or undefined: regexpMatcher.

Type: Interface
Name: Matcher
Path: lib/utils/parse/parse.go
Input: in string (for method Match)
Output: bool (indicating if the input matches)

Source: SWE-Bench Pro task
instance_gravitational__teleport-[illegible]

Not economically valuable

Build a C compiler in Rust from scratch. This includes:
• Lexer, preprocessor, parser, semantic analysis
• x86-64 code generation (emit assembly, use gcc/as/ld to assemble and link)
• Driver with CLI args
• Array subscript and lvalue assignments
• Short-circuit evaluation for logical operators
• Type-aware codegen for correct sized operations
• Global variable support with .data/.bss emission
• Full C preprocessor (macros, conditionals, token pasting, stringification)
• Switch/case/default lowering
• Typedef tracking and cast expressions
• Struct/union layout, member access, and codegen

Source: SWE Marathon task (rust-c-compiler)

G2i
G2i. All rights reserved.
7

### Weak verifiers

DeepSWE’s comparison reports false positive and false negative rates of 8.5% and 24.0% for SWE-Bench Pro, versus 0.3% and 1.1% for DeepSWE.

[5:16](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=316s) [A chart compares false positive and false negative rates for SWE-Bench Pro and DeepSWE.](https://assets.ai.engineer/sites/aiecode2025/sha256/f83627734ae82b8fa34bae2e0ca29b0314a34503591c17ee214cd8b4fd2d2ca4.webp)

When the instrument is miscalibrated
Weak Verifiers

False positive rate
Verifier accepted a wrong implementation
SWE-Bench Pro: 8.5%
DeepSWE: 0.3%
Axis ticks: 0%, 5%, 10%

False negative rate
Verifier rejected a correct implementation
SWE-Bench Pro: 24.0%
DeepSWE: 1.1%
Axis ticks: 0%, 15%, 30%

Source: https://deepswe.datacurve.ai/blog/deepswe

Interface text:
G2i
G2i. All rights reserved.
8
10

### SWE-Bench Pro test patch excerpts

The speaker points to a test expecting an unspecified variable name and comparisons of unexported fields as examples of how a verifier could reject valid solutions.

[5:59](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=359s) [Two code excerpts highlight a matcher field and a comparison option allowing unexported fields.](https://assets.ai.engineer/sites/aiecode2025/sha256/71452a1f3206464f72a645ec64051be28c8d04f6cfc4982bc44ed38acfda914a.webp)

{
    title: "string literal",
    in:    `foo`,
    out:   &regexpMatcher{re: regexp.MustCompile(`^foo$`)},
},

cmp.AllowUnexported(
    regexpMatcher{},
    prefixSuffixMatcher{},
    notMatcher{},
    regexp.Regexp{},
),

SWE-Bench Pro test patch excerpts

G2i
G2i. All rights reserved.
9

### Reward hacking

The strict harness scores lower across all three models, with drops of 9.1%, 7.5%, and 0.3%. The speaker describes models searching for .git folders or online traces instead of applying a patch.

[6:58](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=418s) [A chart compares standard and strict harness scores for three coding models.](https://assets.ai.engineer/sites/aiecode2025/sha256/35886d946fa025e7f6ffd29f5a0df02dfc8aae79ebae85da141b8203ee6b153a.webp)

When the instrument changes what is being measured

Reward Hacking

SWE-bench Multilingual score
Y-axis ticks: 0, 20, 40, 60, 80, 100%
Legend: Standard harness; Strict harness
Opus 4.8 Max: -9.1%
Composer 2.5: -7.5%
Opus 4.6 Max: -0.3%

Source: https://cursor.com/blog/reward-hacking-coding-benchmarks

G2i
G2i. All rights reserved.
10

### Standard and strict benchmark scores

Composer 2.5 falls from 74.74% to 54.04% under the strict harness, a 20.7-point gap. Opus 4.7 (high) shows a 4.9-point gap.

[7:50](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=470s) [A table lists model scores under standard and strict harnesses and their differences.](https://assets.ai.engineer/sites/aiecode2025/sha256/188198257387438cc94e5c8e95078df19585195b1162e2ed551783666326ddc8.webp)

MODEL | STANDARD | STRICT | Δ
1 | Composer 2.5 | 74.74% | 54.04% | +20.7
2 | Opus 4.8 (max) | 87.14% | 73.03% | +14.1
3 | Opus 4.8 (xhigh) | 84.67% | 70.86% | +13.8
4 | Opus 4.8 (medium) | 76.80% | 67.72% | +9.1
5 | Opus 4.8 (high) | 78.93% | 69.88% | +9.1
6 | GPT-5.4 (high) | 60.00% | 53.40% | +6.6
7 | Opus 4.8 (low) | 71.75% | 66.07% | +5.7
8 | Opus 4.7 (max) | 69.88% | 64.71% | +5.2
9 | Opus 4.7 (xhigh) | 67.99% | 62.86% | +5.1
10 | Opus 4.7 (high) | 67.65% | 62.72% | +4.9

Source: https://cursor.com/blog/reward-hacking-coding-benchmarks

G2i
G2i. All rights reserved.
11

### The quality gap becomes a trust gap

The speaker says a quality gap causes a trust gap: engineers he has met in the last six months look at leaderboards but test models themselves.

[8:07](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=487s) [Slide states that the quality gap leads to a trust gap.](https://assets.ai.engineer/sites/aiecode2025/sha256/8f3b087fb8c5d95d709afabe0a9e609fb0e584e2269a11cb4df0e77e449989f7.webp)

This quality gap ⇒ A trust gap.

G2i
G2i. All rights reserved.
12

### Closing the gap

G2i’s team has spent two months defining principles for building better benchmark tasks.

[8:30](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=510s) [Section slide introduces principles of better benchmarks.](https://assets.ai.engineer/sites/aiecode2025/sha256/4e29a2c2414732beadc3f7dfc0908f97191f48f391dc104c2e06b0d4b3be8ed9.webp)

Closing the gap

Principles of better benchmarks

G2i
G2i. All rights reserved.
13

### Principles of better benchmarks

Better benchmarks pair human-authored instructions with behavioral grading, production value and novel tasks held out privately. They should also explain results: leaderboards show who wins, but not why.

[11:12](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=672s) [Five principles: human instructions, holistic graders, production grade, contamination free, and information above leaderboards.](https://assets.ai.engineer/sites/aiecode2025/sha256/df2205dbb5e7424d1fe71fa762e0dd34636eea430b82b6110e0b10ed16d8b1da.webp)

1. Human Instructions
Authored by humans.
Reviewed by humans.

2. Holistic Graders
Behavioral tests.
Precision where needed.

3. Production Grade
Obvious value.
Economically valuable.

4. Contamination Free
Novel tasks.
Private holdout sets.

5. Information above leaderboards
Tells a story.
Helps decision making.

G2i. All rights reserved.

Interface text:
G2i
14
16

### Call to action to software engineers

The speaker asks software engineers to examine how SWE benchmarks work and join the Discord, saying their input is valuable.

[12:00](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=720s) [Slide asks software engineers to examine SWE benchmarks and join the Discord, emphasizing that their input is valuable.](https://assets.ai.engineer/sites/aiecode2025/sha256/0a02c7d10e384c49afdf8572cb1c80306c1b47d375d60353310bbf75a3510019.webp)

Call to action to software engineers

Look under the hood of SWE benchmarks and join the discord. Your input is valuable

G2i
G2i. All rights reserved.
15

## Full transcript

[0:00](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=0s) [outro jingle] Hello, everyone. Um, this is the last talk of this session, so hopefully it's gonna be short.

[0:17](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=17s) I know that you guys had to go through a long day, so try to keep it short and light for you all. Um, I'm gonna present myself. Um, I'm Ali.

[0:26](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=26s) I'm the director of AI/ML at G2i. Um, I have zero experience in ML, so I don't know why they put the ML in my title. I'm a software engineer, uh, at heart, and to prove that I have more than fifty abandoned side projects in my machine.

[0:40](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=40s) So, uh, you can know. So, uh, I'm gonna make a disclaimer. The, the title of the, the presentation is a little bit misleading. Uh, as I was working on it, I realized that it would be better if I presented my journey, uh, into benchmarks and what I learned instead of trying to find a dichotomy of the, the

[1:00](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=60s) bad, the ugly and, and the good. So, um, let's start with, um... I wanna grab your attention, and I invite you to look at this.

[1:10](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=70s) These beautiful three screenshots are a single prompt on one of the benchmark tasks.

[1:16](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=76s) As I was looking at it, I was like, "How can an engineer write a task like this?" So I said, "Nah, it's impossible. No one writes prompts like these ever."

[1:26](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=86s) But I wanted to double-check with my engineers, so I took three of our best engineers, I showed them the prompt, and I said, "Would you ever write a prompt like this?"

[1:34](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=94s) And the answer was no. And they're right. They shouldn't. And so

[1:43](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=103s) at that point, um, I was like, "What is a-- what are benchmarks anyway?" Uh, I needed to take a step back. I needed to look more. I need to understand.

[1:52](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=112s) And so as I was researching, I faced a wall of keywords. Um, graders, long horizon, verifiers, bench- benchmarking, and a lot of jargon. So I was like, either this is too complicated or, um, there's a lot of jargon and a lot of, um, words to, to, to work through here.

[2:16](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=136s) So, um, I worked through it, worked with my team. I have a lot of good researchers in the team, and we, uh, kinda like nailed-- like simplified to the most basics.

[2:27](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=147s) Um, and so the way I see it is that it starts as a prompt or an instruction. That prompt is fed to models and agents.

[2:40](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=160s) Agents provide solutions. Those solutions are verified, uh, and graded through verifiers and rubrics. All of that is wrapped in a harness that's, that's preventing it from, um, from the external factors.

[2:56](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=176s) And if it all goes good, uh, we have, um,

[3:00](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=180s) trajectories, scores, and, um, metadata that we can use, uh, to, to, to verif-- to

[3:09](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=189s) basically, uh, rank, uh, models. And so the equation is simple. If prompts and instructions are great, and verifiers are-- and rubrics are doing their job while the harness is preventing, uh, or creating an environment that is good for a benchmark, we should have amazing results.

[3:31](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=211s) Um, but that's not the reality. So what, what, what went wrong?

[3:37](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=217s) So the first thing is, when looking deeper in benchmarks, uh, most of the instructions are unrealistic. Um, I did a quick research on SWE-bench Pro, and, um, there's four hundred and eighty-one words per, per instruction in average.

[3:55](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=235s) That's a two pager, two-pager per task. That is not how people write prompts.

[4:01](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=241s) And to illustrate more of that, um, I took a couple examples here. The first one I looked at, I, I call the leaky prompt. It's a, uh, Go, um, task that's basically, um, that's bas-- trying to match in some regex and doing tests on, on some regex.

[4:18](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=258s) So in the first screenshot here, um, the instruction is pointing directly to the test file, which basically means that the LLM has all the ingredient it needs to go and find that test file and implement based on that.

[4:31](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=271s) The second one is, is even worse. Um, it's basically providing a complete interface of the implementation, basically locking the LLM from any kind of, uh, creativity, and it's forcing it to do it that way.

[4:46](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=286s) So that's the leaky prompt. The second example, it's the, the not economically valuable prompt. Uh, this is from SWE marathon, and this prompt is well-formed. Uh, it's, it's,

[5:02](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=302s) it's abstracted enough to allow for the LLM to do its work, but it's asking it to build a C compli- compiler in Rust. So I don't know if any of you ever tried to do that, but I don't think it's a good idea.

[5:12](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=312s) We should not do that. All right, moving on. The second problem, weak verifiers. Um, so the screenshot here is, is a, uh, is the work that DeepSWE, um, did, uh, to compare their, uh, their bench against SWE-bench Pro.

[5:32](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=332s) And, um, let me just fix here so I can see the numbers.

[5:39](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=339s) In SWE-bench Pro, eight point five of-- eight point five percent of all the tasks, uh, accepted wrong implementation in one hand, and more than twenty, twenty-four percent of the tasks, uh, rejected, um, correct implementations.

[5:56](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=356s) And so I kinda went again, dug a little bit, and I extracted one of the tasks, and I started looking at it. Um, and, and here's, here's what's happening in the example of, uh-

[6:08](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=368s) Reje-rejecting, um, possibly rejecting good, good answers. So in this example, the test is, is basically expecting a variable to exist. But that variable is first not specified in the instruction, and two, why would we expect an LLM to write the, the variable name this way?

[6:28](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=388s) So this test is cornering the LLM and basically, uh, causing, uh, those false negatives. In the other example, it's bas-- the test is basically checking functions that are unexported.

[6:44](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=404s) So if that was a PR in any of our projects and, uh, exposed these type of tests, we would not accept it. So this is what a weak verifier looks like.

[6:58](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=418s) All right, moving on. Re-reward hacking. So what's happening is models are becoming increasingly, increasingly able to optimize and figure out solutions to hard problems by going around the problem.

[7:12](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=432s) So instead of actually trying to, to fix the-- to, to, to apply a patch to a task, they try to go and find .git folders, or they look up the internet for any kind of traces that would allow 'em to, um, to do the task.

[7:29](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=449s) And this first graph here shows like-- shows that as models evolve, they are now more s-smarter and smarter in being able to do reward hacking. But

[7:42](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=462s) that's what we want. We want LLMs to be smart. The benchmarks are lacking behind, and they're not preventing from, from that to happen. Um,

[7:52](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=472s) more in detail, as you can see here, the more you go in time and the more you have new versions, the delta of, um, of, um, reward hacking is increasing.

[8:06](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=486s) So the conclusion here is there's a quality gap, and it's causing a trust gap. I have not met an engineer in the last six months that would choose a model or choose, um, an LLM based on the leaderboards.

[8:21](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=501s) Um, they look at them, there's a lot of hype, but then they move on, and they test things by themselves, and they apply that.

[8:32](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=512s) So how do we close the gap? Um, in the last two months, we've been working with our team at G2i to basically try to define a framework, uh, a set of principles that would allow us to build tasks for benchmarks that are, um, better than what we have today.

[8:54](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=534s) The first one, human instructions. Authored by humans, reviewed by humans. This is basically the entry point for any great tasks. The instructions given to an agent or an LLM should lean towards expressing desired behaviors, objectives, and hard constraints, not implement details

[9:14](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=554s) or try to guarantee self-containment when the task itself is, is expressing too much, uh, too, too much details.

[9:24](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=564s) The second principle is holistic graders, behavioral tests in one hand and then precision what ne-- where needed. This is very similar to how we approach, um, tests in engineering.

[9:37](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=577s) We want to have the most surface covered without being too prescriptive, but we also want to be precise where needed. So for security issues or business logic, we wanna have the whole stack unit, unit test, integration test, and then end-to-end tests.

[9:53](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=593s) But for the rest of the, the rest of the, the software, we don't wanna have a hundred percent coverage because that's, um, not efficient.

[10:05](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=605s) The third principle, production-grade. The tasks have to be--

[10:10](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=610s) that's-- have to have value, um, and they have to be eco-eco-economically valuable. Um, it is one thing to have a, a test, a task that is failing the LLM and proving that the LLM is not there yet.

[10:24](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=624s) It is another for-- It is another thing for an engineer to look at a task and say, "If the LLM is fixing this, I trust it to fix that."

[10:33](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=633s) Currently, we don't have that, so production-grade. The fourth principle, contamination-free by design.

[10:44](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=644s) We wanna do novel tasks only, and we wanna make sure that we keep pr-private holdout sets. This is

[10:53](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=653s) a principle that is very important as currently

[10:58](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=658s) the tasks that are existing in benchmarks are all pulled from GitHub repos or from, um, from, from public repos. So our approach here is that it should always be novel.

[11:10](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=670s) This way it's contamination-free by design. And the fifth and last principle here is information above leaderboards. Um, the benchmark needs to tell a story and needs to help people make decisions.

[11:23](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=683s) Leaderboards are what we see in benchmarks today. They tell you who wins, but they don't tell you why. And so we wanna basically put the X-axis back on, on, on, on the first page.

[11:37](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=697s) Uh, the idea here is that there's, um,

[11:42](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=702s) there's a lot of, um, data that we can extract from those-- these runs, and unfortunately, they're not being put, uh, in the forefront. And people have to dig, uh, a lot and do their own experiments to get to those data points.

[11:58](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=718s) And so finally, uh, initially I wanted to have a kind of a, a lofty like ending to this, but I think I, I, I pivoted to something more interesting.

[12:08](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=728s) Uh, this is a call to action to software engineers. Um,

[12:13](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=733s) benchmarks are not hard. We need to look under the hood, and we need to understand them and join the Discord because engineers' input is valuable.

[12:24](https://www.youtube.com/watch?v=jWq-aZIU0kM&t=744s) And thank you. [audience applauding] [upbeat music]
