Coding Evals: From Code Snippets to Codebases – Naman Jain, Cursor

Talk transcript

1. 0:00

   [on-hold music] Hi, everyone.

2. 0:21

   So I'll be talking about, uh, like, some work on evaluations, particularly evaluations across, like, I guess I've done in the last four years. So let's get started.

3. 0:32

   So, uh, I'll be talking about co- uh, coding evaluations across varying time horizons. So I've been, uh, working on, like, in the code space for about four years now.

4. 0:39

   Like, it was right before, like, early Copilot came out, and my first project was actually working on generating, like, single-line pandas snippets, and my last project was generating an entire code base, so the field has, like, really progressed fa-- very quickly.

5. 0:53

   So I'll be talking about, like, uh, different stages of evaluations we have considered and some f-- uh, learnings across these other projects and how I see evaluations going forward.

6. 1:01

   So the first work I did was on, uh, like, uh, evaluating, uh, coding models in like second, uh, work doing in seconds of time, like generating single-line snippets, your Copilot code completions.

7. 1:12

   Then I work- did some work on, like, uh, evaluating on, like, interview-style competition programming problems, uh, which, uh, where models can work up to minutes. Uh, then we worked on some work on, like, uh, repository question answering, uh, which required like maybe, uh, more, uh, multiple minutes, tens of minutes.

8. 1:28

   Uh, and finally, like, uh, pushing the, uh, frontier forward, we are, uh, thinking about, uh, evaluating models on very complex tasks which can take hours or, like, multiple hours of work, like code optimization and, like, even further.

9. 1:40

   So let's get started. Uh, so first work I'll be talking about is, like, CodeBench, uh, which is, uh, like, uh, uh, evaluation work on, uh, models for, like, competition coding.

10. 1:52

    So here, uh, like, this is what a problem would look like. This is, like, very standard LeetCode problem, and don't worry, you don't need to solve something like this.

11. 1:59

    So, uh, like, uh, here, uh, as you can see, there's a problem, uh, statement. And, and the nice thing about these interview-style problems is that these problems are very well, uh, defined.

12. 2:09

    You have like, uh, good natural language specifications, some example input, output examples, so you can very, uh, reliably evaluate if the models are doing a good job or not.

13. 2:17

    So what is the motivation behind this and how, uh, we improve the frontier here. So the first challenge in, uh, evaluating, uh, language models these days is, like, data contamination.

14. 2:27

    These models are trained on, like, the entire Internet, and, uh, like on Stack Overflow, you'll find, uh, like very, uh, similar programming problems, puzzles. Uh, similarly, uh, like you'll find, uh, like, uh, very similar programming problem sources on GitHub, uh, or on the Internet.

15. 2:42

    So, uh, like contamination is a big, uh, deal. Uh, another very, uh, challenging factor we just struggled with the field is, like, insufficient test suites. So, uh, you'll see that, uh, like th- in this program, uh, like the goal was to return a sorted unique, uh, common e-elements between the two lists.

16. 2:59

    But, uh, like even a solution which does not do the sorting and just returns the set actually works because the tests were brittle and were not catching this mistake.

17. 3:06

    So, uh, like test suites is another, uh, like, uh, very challenging factor and how do we generate good and diverse tests. And finally, uh, difficulty distributions, which is something which people did not-- do not really, uh, reliably, uh, like calibrate.

18. 3:20

    Uh, like when I first was working, uh, in, uh, this space, uh, like there were two benchmarks available. On one benchmark, the performance was eighty percent or ninety percent, and on the other one it was one percent, and there was nothing in between.

19. 3:32

    And, uh, like as, uh, like benchmark users, what you care about is having some signal from the benchmark to, like, basically headline to make progress, to measure progress. And in, uh, either of these regimes, when if the problems are too easy or too hard, you don't get a lot of signals.

20. 3:46

    So it is very important when you're designing benchmarks to think about, like, the kinds of problems you are taking, and will it provide enough signal for the users of your benchmark.

21. 3:55

    So, uh, like in LiveCodeBench, we pioneered like dynamic evaluations. Uh, particularly, uh, like we can periodically update, uh, the evaluation sets, uh, and this gives you two, uh, very nice factors.

22. 4:06

    First is you can combat contamination. So you can evaluate the models on problems that were released after the model was trained, so it has likely not seen the problem, something like that.

23. 4:14

    Uh, and, uh, then you can also modify the problem difficulty distributions over time. So as we have talked about, models are incre-- uh, like improving very rapidly. Uh, so what was difficult, uh, for the model six months back might not be now.

24. 4:27

    So you can, uh, if you are updating your evaluation sets constantly, you can actually, uh, keep calibrate-- uh, the difficulty distributions calibrated, so you still get more signal out of your benchmarks.

25. 4:37

    So how we did that here, like, we had like an automated approach for curation of these problems, and, uh, similarly, we could, uh, automatically construct these test cases in an automated manner.

26. 4:47

    And, uh, this allows a very nice thing. When-- Since we are, uh, uh, like collecting problems over time, we have time as a control knob. So, like, we have these problem release months, uh, on LeetCode, and if you evaluate the model performances, like the pass@ rate one metric, uh, like, uh, on problems released over different months, you

27. 5:05

    will see that after, uh, like, uh, these model release dates, you would see stark drop in model performance. So like after DeepSeek was, uh, released in like September 2023, uh, the performance starkly drops from like maybe fifty percent average to like over like twenty percent or thir- uh, fifteen percent average.

28. 5:21

    So like, uh, based on these sliding windows, you can, uh, evaluate performance, measure contamination, and even combat contamination.

29. 5:28

    Um, and, uh, we have the running leaderboard, which is like very well maintained. And, uh, on this leaderboard you can actually, uh, like, uh, like view performances by, uh, scrolling this, uh, horizontal time bar.

30. 5:39

    And you'll see that as you are scrolling, uh, the contaminated models, which are the red bars, actually go down, which does highlight that, uh, like problem does, uh, like model performance does change on, uh, these newer kind of problems.

31. 5:52

    Um, finally, for, uh, test generation, we, uh, maintain, uh, like these, uh, test generation, uh, te-test generators. So if you have worked on fuzzing, you would have like input generators where you'd gen-generate diverse inputs.

32. 6:04

    And each of the problems are supported by like thirties or fifty inputs, so you can, uh, reliably find mistakes and bugs in, uh, incorrect code. And these are all automatically generated, uh, using an LLM-driven approaches.

33. 6:16

    And these problems, uh, have been like continuously being released and updated. So we have released like sift different versions of, uh, LiveCodeBench and these, uh, ne-- One of the nice things or one of the worrying things for me at the start was that, uh, like if you're constantly updating the eval sets, will, uh, like people be able

34. 6:31

    to keep track of them? Will, will people be using them, or will they just restrict to a single version? Uh, it turned out that these newer eval sets were constantly be update-- uh, like, adopted by different foundation model labs.

35. 6:41

    And, uh, like, uh, since we updated the problem difficulty over time, uh, the evaluation sets continue to provide strong signal to compare, uh, different models.

36. 6:51

    Um, so this was like LiveCodeBench. Let's talk about, uh, like something which is, uh, a more encoding agents, like more real-world programs. And this is, uh, work on, like, uh, software optimization.

37. 7:01

    So this is a problem I'm very excited about, and I'll ta- I'll talk about a few factors why you should maybe be excited about this. So, uh, here we are trying to, uh, measure model capabilities in generating high-performance software.

38. 7:12

    And, uh, I feel that this, uh, like problem domain, uh, like mixes two, uh, factors, like the algorithmic coding, uh, uh, field I talked about, which is like LiveCodeBench setting, but also like globa-global software editing, like, uh, SWE-bench and other like software op-- uh, ge- uh, general software engineering benchmarks.

39. 7:28

    Uh, uh, in high-performance, uh, software, you will have to do algorithmic work, you have to do deep analysis, and find, uh, uh, generate so-software with like right, uh, runtime.

40. 7:38

    So, uh, one of the key, uh, principles when we are trying to build this benchmark was like ensuring construct validity, because when you see a lot of benchmarks today, uh, we, uh, get very high benchmark scores, but at a lot of the times they don't really translate to real-world performance gains.

41. 7:53

    So construct validity refers to how close, uh, a measurement reflects the underlying, uh, concept it's meant to measure. So like here we are measuring code optimization, and we want something which is, uh, like, uh, reliably evaluates real-world, uh, takes.

42. 8:06

    So this usually re- requires like two aspects. First is like the task distribution. Your task should be natural and sourced from the real world, and then you should be able to reliably grade them.

43. 8:16

    So let me talk about like what steps we take to, uh, make this happen and how we construct this benchmark. So let's say we take a code base like llama.cpp.

44. 8:25

    Uh, we take, uh, uh, we crawl over all the commits of the code base, and we find the commits which are opt-- uh, like doing something, uh, related to performance optimization.

45. 8:33

    So here there was this commit which is optimizing the quantized performance of, uh, like, uh, certain kinds of models. Uh, for all of these, uh, commit-- uh, performance optimizing commits, we would, uh, like generate performance test cases.

46. 8:45

    Um, and, uh, these performance test cases would look like some workloads. And, uh, once we have these workloads, uh, we have a very, uh, nice and precise way to specify the problem statement that, uh, given this workload of, let's say, uh, running a Qwen, uh, seven-B model, uh, can, uh, we give this, uh, problem to, uh, SWE-agent

47. 9:03

    and ask the model to optimize the code, uh, the llama.cpp repository, so this code runs faster. So as you can imagine, this co-- uh, task is like fairly challenging.

48. 9:10

    You need to understand, like low level, uh, implementation details, uh, and, uh, like how quantized models, uh, behave, how we can, uh, improve the runtime. And so models can generate a patch, and the evaluation is done on whether the patch is correct.

49. 9:23

    So does it pass the equivalence check with the human patch? And, uh, is there a, a valid optimization over the, uh, reference human patch? Uh, that is, uh, whether you can, uh, generate a better runtime than what a human could do.

50. 9:36

    S- so, uh, like, uh, this is a very challenging task. We have like hundred plus optimization tasks sourced in this manner, and this is like fairly, uh, like important in, uh, like, uh, like high-p-performance settings.

51. 9:47

    So think about like data science, uh, like ML visualization scenarios. Uh, our, uh, benchmark, uh, like comprises of like various, uh, low-level, uh, code, like C/C++ Rust. And the very nice thing is like these are precise problem statements.

52. 10:01

    You can, uh, easily specify to the model what is the goal in the form of a performance test which the model has access to, and it can continuously iterate over it for a long time.

53. 10:09

    So here we can scale the test-time compute and pick the best solution based on, uh, the test cases that we have. S-- And this can happen like synchronously or asynchronously.

54. 10:19

    So, uh, like, uh, we generate these performance test cases, and, uh, that worked, uh, reasonably well, but, uh, we found that there were, uh, like cases of reward hacking here.

55. 10:28

    So what do I mean by reward hacking? Like friendly models would write non-idiomatic code to like actively exploit the evaluation infrastructure or overfit the test distributions. So one funny example we saw was like our models would add like LLU cache to pan-- uh, like arbitrary pandas methods when we were, uh, trying to optimize pandas and the, uh,

56. 10:46

    efficient solution should have required changing something in the internals. Uh, so we try to pass this by changing our evaluation in-infrastructure, so it's like more robust to this kind of hacking, uh, approaches.

57. 10:57

    But then we saw something, you know, like even more drastic. Models would sometimes completely hijack the infra where, uh, they would add, uh, like site-customized .py file where-- which runs at the start of Python runtime, and it would basically change the NumPy library, uh, like which was i-installed in the code base or something it crawled from, uh,

58. 11:15

    source. And there is-- Like I think you can do some, uh, ways to, uh, like take some measures to make your evaluation infra, which is robust to these kind of, uh, like adversarial, uh, like attacks.

59. 11:26

    But, uh, here, uh, like there could be myriad ways in, uh, in which models can hack these kind of scenarios. And here, uh, we propose, uh, like HackDetector, where, uh, which is a detection system that leverages GPTF like code analysis capabilities and test-time compute to like basically identify these kind of hacking behaviors at runtime.

60. 11:44

    So you don't have to imagine all the possible failure scenarios at the start. So what it would take is like a model patch, the expert patch and test cases, and we'll ask GPT-5 to give like verdicts on like whether it's reward hacking with some kind of explanation.

61. 11:57

    Uh, we'll do, uh, do this a few times and take the consensus. And based on this con-census, we'll determine if, uh, this is, uh, doing some like a non-idiomatic coding patterns or not.

62. 12:07

    And, uh, we did some fai-failure analysis based on this. So now you can detect mistakes using test cases, whether the code is correct or not, whether it is optimizing or not.

63. 12:15

    But you can also detect reward hacks using this, like, LLM-as-a-judge, uh, factor. And wh- uh, what you see is kind of surprising. Uh, like, models make a lot of, like, uh, correctness mistakes that you can catch by tests.

64. 12:27

    But even if the code passes the test cases, like, O3 attempted reward hacking patterns in, like, 30% of the problems it tried. And this fraction is, like, going down, uh, for the newer models to some degree, but it is still existing.

65. 12:38

    And as we go to more and more real world tasks, uh, this is going to, uh, get more challenging, and we need to figure, uh, like, ways to combat these kind of reward hacking patterns by using LLM judge and other, uh, ways to make this evaluation infra more, uh, reliable.

66. 12:53

    So next I'll talk about, like, uh, s- like, specifically some of our new work on, like, uh, like, pushing the b-boundary of code evals even further and, uh, taking a look at more challenging tasks.

67. 13:03

    So here, uh, we were, uh, thinking about, like, can, uh, like, these language models translate, uh, like, a entire code base? Uh, specifically given a specification as a C program, can you generate a C++ implementation for the same?

68. 13:17

    And we took a fairly complex, uh, code base. So Zopfli is a, like, highly efficient compression library from Google. Like, it has about, like, 4,000 lines of code, hundreds of functions, and complex data structures.

69. 13:28

    Uh, and, uh, we want, like, um, like, very precise and correct code, so we, uh, generated, like, a million compression inputs, and your test case was to generate a Rust implementation that, uh, maintains correctness over those million test cases.

70. 13:40

    And when I did this work back in, uh, like, uh, last year, it took us 12 hours to actually do this translation. Now, perhaps with better models, this can be done in two hours.

71. 13:48

    But still, I think, uh, this is pushing the frontier of, like, what the models can do currently. Um, so what is one of the key findings when we are trying to make progress in s- uh, something like this?

72. 13:58

    Like, end-to-end correctness is i-important, but it only gives you, like, one bit of feedback. But for these very long horizon tasks, one thing which, uh, will become, uh, more important going forward is, like, having some s- uh, measures of in-intermediate correctness.

73. 14:11

    So, like, for our case, we could, uh, measure, like, fraction of code translated, fraction of code refactored, and based on these kind of settings, you can, uh, understand, like, h- if you're making progress or not, and how you can, uh, scale the systems better.

74. 14:25

    Um, so, like, uh, as we're closing, I'll talk about, uh, like... I'll quickly talk about some of the work I did on, like, in-the-wild evals. So this work was done in collaboration with, uh, Llama Arena folks.

75. 14:35

    And, uh, like, I'll talk about two settings here. First is Copilot Arena. So this is, like, evaluating in IDE, uh, code completion assistance. So what we will do here is we'll, uh, have an IDE plugin with, uh, like, uh, similar to, uh, GitHub Copilot setting, uh, we'll generate a completion for you, but instead of just a single

76. 14:51

    completion, you'll have, uh, two completions appearing, like, um, top and, uh, down. And you can, uh, pick either one of them via shortcuts like Tab or Shift + Tab.

77. 15:00

    And, um, based on the, uh, like, acceptance rates, we can pairwise compare what the code completion assistants are doing.

78. 15:07

    S- uh, uh, we also did some work on RepoChat, where, uh, like, uh, to evaluate, uh, like, code question answering capabilities of models, uh, we, uh, built a system where you can provide a GitHub URL, uh, and you can ask a natural language query about the code base, which could be something about explain the code base to

79. 15:23

    as complex as, "Let's try to solve this issue. Let's give me... give me a model patch that could s-uh, solve this issue." And, uh, uh, we integrated a very basic and simple, uh, like, SWE-agent system that fetches the code base, resolves user queries in, like, multi-turn, uh, code assistant, uh, conversations.

80. 15:40

    So, uh, one thing that stood out to me in these kind of things, uh, is, like, like, how human-centric experiment design, uh, needs to be. So, uh, like, for code, uh, like, Copilot Arena in particular, we realized that, like, uh, latency is a big concern for acceptance rates.

81. 15:57

    So if you look at the acceptance... uh, like, latency below and acceptance rates, like, if it is, like, anything more than one second, uh, like, the acceptance rates drop very starkly.

82. 16:06

    So people care a lot about latency, so you have to... so we had to, uh, design an experiment so that it's robust to these kind of, like, uh, latency differences between models, balanced latency across different models.

83. 16:15

    So, like, if you're doing, uh, like, anything in the wild, having this human-centering component, understanding human behaviors is very important to do anything meaningful. So, uh, at the end, I think, uh, just to, uh, recap, like, I think I talked about a bunch of works.

84. 16:30

    Uh, like, what are some, uh, big takeaways? So I think, uh, dynamic, uh, dynamically updating evaluation sets to, like, prevent contamination, like, modify the problem distributions, like, in terms of difficulty, in terms of distribution of tasks we care about.

85. 16:43

    As we, like, uh, improve... uh, as the language model capabilities will improve over time, the, uh, types of tasks will, uh, start to do with model chains. You can even, uh, think of this, like, uh, we were doing, like, code completion where we were generating, like, few tokens, few lines, and now we are generating, like, uh, tens

86. 16:58

    of lines, hundreds of lines. And to some degree, this, uh, will, uh, continuously change, and we have to update our evaluation sets, uh, so that it reflects the real world usage and, uh, kinds of things people need.

87. 17:08

    Um, the second very, uh, important thing is, like, ensuring reliable grading in this domain. And, like, tests are very good for ensuring correctness and, uh, provide a lot of reliable feedback, but, uh, once we go to real world settings, like, models can, uh, start doing, like, lot of non-idiomatic coding patterns.

88. 17:24

    They will add try catches everywhere to just prevent any kind of bug from occurring. So having these kind of LLM judges to detect non-idiomatic coding patterns, code quality, and just any, uh, like, arbitrary hacks, uh, will be very important.

89. 17:36

    And finally, like, as I talked about in the last work, uh, like, intermediate grading signals so that you can measure, like, incremental progress, uh, is, uh, like, another key factor here.

90. 17:46

    So I think that's, uh, the end of my talk. Thank you. [upbeat music]