Benchmarks: The Good, the Bad, and the Ugly

Talk transcript

1. 0:00

   [outro jingle] Hello, everyone. Um, this is the last talk of this session, so hopefully it's gonna be short.

2. 0:17

   I know that you guys had to go through a long day, so try to keep it short and light for you all. Um, I'm gonna present myself. Um, I'm Ali.

3. 0:26

   I'm the director of AI/ML at G2i. Um, I have zero experience in ML, so I don't know why they put the ML in my title. I'm a software engineer, uh, at heart, and to prove that I have more than fifty abandoned side projects in my machine.

4. 0:40

   So, uh, you can know. So, uh, I'm gonna make a disclaimer. The, the title of the, the presentation is a little bit misleading. Uh, as I was working on it, I realized that it would be better if I presented my journey, uh, into benchmarks and what I learned instead of trying to find a dichotomy of the, the

5. 1:00

   bad, the ugly and, and the good. So, um, let's start with, um... I wanna grab your attention, and I invite you to look at this.

6. 1:10

   These beautiful three screenshots are a single prompt on one of the benchmark tasks.

7. 1:16

   As I was looking at it, I was like, "How can an engineer write a task like this?" So I said, "Nah, it's impossible. No one writes prompts like these ever."

8. 1:26

   But I wanted to double-check with my engineers, so I took three of our best engineers, I showed them the prompt, and I said, "Would you ever write a prompt like this?"

9. 1:34

   And the answer was no. And they're right. They shouldn't. And so

10. 1:43

    at that point, um, I was like, "What is a-- what are benchmarks anyway?" Uh, I needed to take a step back. I needed to look more. I need to understand.

11. 1:52

    And so as I was researching, I faced a wall of keywords. Um, graders, long horizon, verifiers, bench- benchmarking, and a lot of jargon. So I was like, either this is too complicated or, um, there's a lot of jargon and a lot of, um, words to, to, to work through here.

12. 2:16

    So, um, I worked through it, worked with my team. I have a lot of good researchers in the team, and we, uh, kinda like nailed-- like simplified to the most basics.

13. 2:27

    Um, and so the way I see it is that it starts as a prompt or an instruction. That prompt is fed to models and agents.

14. 2:40

    Agents provide solutions. Those solutions are verified, uh, and graded through verifiers and rubrics. All of that is wrapped in a harness that's, that's preventing it from, um, from the external factors.

15. 2:56

    And if it all goes good, uh, we have, um,

16. 3:00

    trajectories, scores, and, um, metadata that we can use, uh, to, to, to verif-- to

17. 3:09

    basically, uh, rank, uh, models. And so the equation is simple. If prompts and instructions are great, and verifiers are-- and rubrics are doing their job while the harness is preventing, uh, or creating an environment that is good for a benchmark, we should have amazing results.

18. 3:31

    Um, but that's not the reality. So what, what, what went wrong?

19. 3:37

    So the first thing is, when looking deeper in benchmarks, uh, most of the instructions are unrealistic. Um, I did a quick research on SWE-Bench Pro, and, um, there's four hundred and eighty-one words per, per instruction in average.

20. 3:55

    That's a two pager, two-pager per task. That is not how people write prompts.

21. 4:01

    And to illustrate more of that, um, I took a couple examples here. The first one I looked at, I, I call the leaky prompt. It's a, uh, goal, um, task that's basically, um, that's bas-- trying to match in some regex and doing tests on, on some regex.

22. 4:18

    So in the first screenshot here, um, the instruction is pointing directly to the test file, which basically means that the LLM has all the ingredient it needs to go and find that test file and implement based on that.

23. 4:31

    The second one is, is even worse. Um, it's basically providing a complete interface of the implementation, basically locking the LLM from any kind of, uh, creativity, and it's forcing it to do it that way.

24. 4:46

    So that's the leaky prompt. The second example, it's the, the not economically valuable prompt. Uh, this is from SWE marathon, and this prompt is well-formed. Uh, it's, it's,

25. 5:02

    it's abstracted enough to allow for the LLM to do its work, but it's asking it to build a C compli- compiler in Rust. So I don't know if any of you ever tried to do that, but I don't think it's a good idea.

26. 5:12

    We should not do that. All right, moving on. The second problem, weak verifiers. Um, so the screenshot here is, is a, uh, is the work that DeepSWE, um, did, uh, to compare their, uh, their bench against SWE-Bench Pro.

27. 5:32

    And, um, let me just fix here so I can see the numbers.

28. 5:39

    In SWE-Bench Pro, eight point five of-- eight point five percent of all the tasks, uh, accepted wrong implementation in one hand, and more than twenty, twenty-four percent of the tasks, uh, rejected, um, correct implementations.

29. 5:56

    And so I kinda went again, dug a little bit, and I extracted one of the tasks, and I started looking at it. Um, and, and here's, here's what's happening in the example of, uh-

30. 6:08

    Reje-rejecting, um, possibly rejecting good, good answers. So in this example, the test is, is basically expecting a variable to exist. But that variable is first not specified in the instruction, and two, why would we expect an LLM to write the, the variable name this way?

31. 6:28

    So this test is cornering the LLM and basically, uh, causing, uh, those false negatives. In the other example, it's bas-- the test is basically checking functions that are unexported.

32. 6:44

    So if that was a PR in any of our projects and, uh, exposed these type of tests, we would not accept it. So this is what a weak verifier looks like.

33. 6:58

    All right, moving on. Re-reward hacking. So what's happening is models are becoming increasingly, increasingly able to optimize and figure out solutions to hard problems by going around the problem.

34. 7:12

    So instead of actually trying to, to fix the-- to, to, to apply a patch to a task, they try to go and find .git folders, or they look up the internet for any kind of traces that would allow 'em to, um, to do the task.

35. 7:29

    And this first graph here shows like-- shows that as models evolve, they are now more s-smarter and smarter in being able to do reward hacking. But

36. 7:42

    that's what we want. We want LLMs to be smart. The benchmarks are lacking behind, and they're not preventing from, from that to happen. Um,

37. 7:52

    more in detail, as you can see here, the more you go in time and the more you have new versions, the delta of, um, of, um, reward hacking is increasing.

38. 8:06

    So the conclusion here is there's a quality gap, and it's causing a trust gap. I have not met an engineer in the last six months that would choose a model or choose, um, an LLM based on the leaderboards.

39. 8:21

    Um, they look at them, there's a lot of hype, but then they move on, and they test things by themselves, and they apply that.

40. 8:32

    So how do we close the gap? Um, in the last two months, we've been working with our team at G2i to basically try to define a framework, uh, a set of principles that would allow us to build tasks for benchmarks that are, um, better than what we have today.

41. 8:54

    The first one, human instructions. Authored by humans, reviewed by humans. This is basically the entry point for any great tasks. The instructions given to an agent or an LLM should lean towards expressing desired behaviors, objectives, and hard constraints, not implement details

42. 9:14

    or try to guarantee self-containment when the task itself is, is expressing too much, uh, too, too much details.

43. 9:24

    The second principle is holistic graders, behavioral tests in one hand and then precision what ne-- where needed. This is very similar to how we approach, um, tests in engineering.

44. 9:37

    We want to have the most surface covered without being too prescriptive, but we also want to be precise where needed. So for security issues or business logic, we wanna have the whole stack unit, unit test, integration test, and then end-to-end tests.

45. 9:53

    But for the rest of the, the rest of the, the software, we don't wanna have a hundred percent coverage because that's, um, not efficient.

46. 10:05

    The third principle, production-grade. The tasks have to be--

47. 10:10

    that's-- have to have value, um, and they have to be eco-eco-economically valuable. Um, it is one thing to have a, a test, a task that is failing the LLM and proving that the LLM is not there yet.

48. 10:24

    It is another for-- It is another thing for an engineer to look at a task and say, "If the LLM is fixing this, I trust it to fix that."

49. 10:33

    Currently, we don't have that, so production-grade. The fourth principle, contamination-free by design.

50. 10:44

    We wanna do novel tasks only, and we wanna make sure that we keep pr-private holdout sets. This is

51. 10:53

    a principle that is very important as currently

52. 10:58

    the tasks that are existing in benchmarks are all pulled from GitHub repos or from, um, from, from public repos. So our approach here is that it should always be novel.

53. 11:10

    This way it's contamination-free by design. And the fifth and last principle here is information above leaderboards. Um, the benchmark needs to tell a story and needs to help people make decisions.

54. 11:23

    Leaderboards are what we see in benchmarks today. They tell you who wins, but they don't tell you why. And so we wanna basically put the X-axis back on, on, on, on the first page.

55. 11:37

    Uh, the idea here is that there's, um,

56. 11:42

    there's a lot of, um, data that we can extract from those-- these runs, and unfortunately, they're not being put, uh, in the forefront. And people have to dig, uh, a lot and do their own experiments to get to those data points.

57. 11:58

    And so finally, uh, initially I wanted to have a kind of a, a lofty like ending to this, but I think I, I, I pivoted to something more interesting.

58. 12:08

    Uh, this is a call to action to software engineers. Um,

59. 12:13

    benchmarks are not hard. We need to look under the hood, and we need to understand them and join the Discord because engineers' input is valuable.

60. 12:24

    And thank you. [audience applauding] [upbeat music]