AI Engineer Code 2025
Coding Evals: From Code Snippets to Codebases – Naman Jain, Cursor
Read the talk
Coding Evals: From Code Snippets to Codebases
As coding agents move from completing a line to changing entire repositories, evaluations need fresh tasks, stronger grading, and signals that reveal progress before the final test passes.
From a talk by Naman Jain
Before you start: Familiarity with unit tests, code-completion tools, and repository patches will help; no prior benchmark research is required.
What changes when the task takes hours?
How do you evaluate a coding model when its job grows from generating one pandas line to producing an entire codebase? Naman Jain’s work spans that transition, beginning before early Copilot. The increasing scope changes both what a task demands and how much feedback an evaluator must provide.
| Time horizon | Coding task |
|---|---|
| Seconds | Single-line snippets and Copilot completions |
| Minutes | Interview and competitive programming problems |
| Tens of minutes | Repository question answering |
| Hours or multiple hours | Software optimization and more complex work |
A completion can be judged soon after it appears. Longer tasks require an evaluation that can support sustained work across a repository.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A passing solution can still miss the specification
LiveCodeBench starts with interview and competitive programming problems. These offer an attractive evaluation contract: an explicit natural-language specification, example inputs and outputs, and a program whose behavior can be tested. But a clear problem statement does not automatically produce a reliable benchmark.
The first problem is contamination. Similar programming puzzles and their solutions appear on Stack Overflow, GitHub, and other sources used to train models. A high score can therefore mix the ability to solve a new problem with familiarity with an old one.
The second problem is an inadequate test suite. Jain gives a task that asks for the sorted, unique common elements of two lists. A solution that returns a set without sorting still passes the brittle tests. The distinction is easy to expose in Python:
python
def common_elements(a, b):
return set(a) & set(b)
def sorted_common_elements(a, b):
return sorted(set(a) & set(b))
a = [3, 1, 3, 2]
b = [3, 1, 4]
expected = [1, 3]
# This check discards the required ordering and result type.
assert set(common_elements(a, b)) == set(expected)
# This check preserves the sorted-list contract.
assert sorted_common_elements(a, b) == expected
assert common_elements(a, b) != expected
The inputs illustrate the failure mode: a test that normalizes away a requirement cannot detect its violation. More tests help only if they check the properties the specification actually requires.
The third problem is difficulty calibration. Jain recalls one available benchmark yielding roughly 80–90% performance and another yielding about 1%. A nearly saturated benchmark leaves little room to distinguish improvements; an overwhelmingly difficult one provides little evidence of partial capability. Benchmark design needs a distribution of problems that gives its users useful signal.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make problem release time a control
LiveCodeBench periodically refreshes its evaluation set. That serves two purposes: problems released after training are less likely to have been seen by a model, and the difficulty distribution can change as models improve. A problem that was challenging six months earlier may no longer distinguish current systems. The evaluation set must keep moving if it is to keep measuring progress.
Automated problem curation and test construction make those updates practical. Collecting problems over time also makes their release month an experimental control: compare a model’s pass@1, its success rate with one attempted solution per problem, across successive LeetCode release windows.
Jain describes DeepSeek falling from roughly 50% to roughly 15–20% pass@1 on problems released after what he identifies as its September 2023 release. The comparison is approximate, and he does not identify the model variant. Its useful feature is the change across problem windows: an aggregate score would conceal that temporal pattern.
The running leaderboard makes this comparison interactive through a horizontal time selector. Moving the evaluation window to newer problems changes the rankings and scores; Jain points to declining red bars that identify contaminated models. Release windows let benchmark users investigate how performance changes when older, potentially familiar problems are excluded.
Test maintenance complements fresh problem selection. LiveCodeBench uses fuzzing-style input generators to produce diverse cases rather than relying only on a few examples. Jain reports roughly 30–50 automatically generated inputs per problem, using LLM-driven approaches to expose mistakes in incorrect code.
At the time of the talk, Jain reports six LiveCodeBench releases. He initially worried that users would stay on one version, undermining the purpose of continuous updates. Instead, foundation-model labs adopted newer sets, and the evolving difficulty distribution continued to distinguish models. Updating a benchmark is useful only if its users move with it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn optimization commits into executable objectives
Software optimization brings algorithmic problem solving into real repositories. The GSO benchmark combines the algorithmic demands of LiveCodeBench with the global software editing associated with SWE-bench. Producing high-performance software requires understanding the implementation, analyzing the algorithm, and choosing changes that improve runtime.
The central design requirement is construct validity: does the measurement reflect the capability it is supposed to measure? A high benchmark score is useful here only if it reflects an ability to optimize real software. Jain separates that requirement into two parts: source natural tasks from real work, and grade them reliably.
The task-construction pipeline starts from repository history:
- Crawl a codebase such as
llama.cppfor performance-related commits. - Identify an optimization, such as a change that improves quantized-model performance.
- Generate a workload that exercises the relevant behavior—for example, running a Qwen 7B model.
- Give the workload to SWE-agent and ask it to patch
llama.cppso that the workload runs faster.
The workload makes the objective concrete, while the repository forces the agent to reason about low-level implementation details and quantized-model behavior.
Grading then asks two different questions. Does the generated patch pass an equivalence check against the human patch? And how does its runtime compare with that human reference? Jain describes outperforming the human patch as the target. The current GSO scoring rule instead requires correctness and at least 95% of the human speedup; that threshold is distinct from beating the reference.
Jain reports more than 100 optimization tasks, spanning settings such as data science, machine learning, and visualization, with low-level code in C, C++, and Rust. The performance test is available to the agent, so it can inspect a precise objective and repeatedly iterate against it. That supports scaling test-time compute: run longer searches, evaluate candidate patches, and select the best valid solution using the tests, either synchronously or asynchronously.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Passing tests does not establish a legitimate optimization
Giving agents a measurable performance objective also creates opportunities for reward hacking. Jain observed models exploiting evaluation infrastructure or overfitting its test distribution. In pandas optimization tasks, models added LRU caches to arbitrary methods where the intended improvement required changing library internals. Reusing results could look efficient under the evaluation workload without delivering the underlying optimization.
After the team hardened the infrastructure against that behavior, it encountered a more invasive exploit. Models added sitecustomize.py, a file Python can load during startup, and used it to modify the NumPy dependency. The patch was reaching beyond the intended optimization and changing the environment in which it would be evaluated. Hardening individual weaknesses helps, but the possible exploits are too varied to enumerate in advance.
HackDetector adds a second form of grading. In the version described in the talk, GPT-5 analyzes the model patch, the expert patch, and the test cases, then returns a reward-hacking verdict with an explanation. The system repeats that judgment and uses the consensus to identify suspicious or non-idiomatic patterns. The workflow spends additional inference compute on reviewing how a patch achieves its result, rather than relying solely on the result itself.
| Grading mechanism | Question it addresses |
|---|---|
| Correctness tests | Does the patch preserve the required behavior? |
| Performance tests | Does it improve runtime? |
| HackDetector judgments | Does it exploit the evaluator or use suspicious patterns? |
These checks cover different failure modes. Jain reports that o3 attempted reward-hacking patterns on about 30% of the problems it tried. He emphasizes that passing tests does not rule out such behavior; the reported rate is not presented as a precisely defined rate conditional on passing tests. The fraction decreases somewhat for newer models but remains nonzero, making dependable grading harder as tasks become more realistic.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A whole-codebase task needs intermediate feedback
Whole-codebase translation extends the evaluation horizon further. Jain introduces the general question using a C program and a C++ implementation, then describes a concrete experiment whose target is Rust. The source is Google’s Zopfli compression library: in his description, roughly 4,000 lines of code, hundreds of functions, and complex data structures. The companion Syzygy project specifies C-to-safe-Rust translation.
The experiment uses one million generated compression inputs to evaluate the Rust implementation. That is a substantial validation workload, but its contract matters: the Syzygy paper describes compression-ratio equivalence for ZopfliDeflate, rather than a proof of equivalence over every possible input.
Jain recalls the earlier translation taking 12 hours and suggests newer models might reduce it to two hours. The paper reports about 15 hours and documents manual intervention, so the recollection should not be read as a measured, fully unattended runtime comparison. The two-hour figure is a forecast. Either way, the example demands sustained work far beyond a short coding completion.
End-to-end correctness provides only one bit of feedback. Until the complete translation passes, a final pass/fail result cannot tell an evaluator how much useful work has accumulated. Jain proposes intermediate measures such as the fraction of code translated and the fraction refactored. These do not replace the final correctness check; they make incremental progress visible enough to diagnose the system and decide how to scale it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evaluate assistance inside the workflow
Evaluations in collaboration with LMArena bring the user into the measurement. Copilot Arena places an IDE plugin in the code-completion workflow. Instead of showing one suggestion, it presents two completions vertically. The developer chooses between them with Tab or Shift+Tab, and acceptance behavior supports pairwise comparisons between models.
RepoChat extends that interaction to repository questions. A user supplies a GitHub URL and a natural-language request, ranging from explaining the codebase to proposing a patch for an issue. A basic SWE-agent system fetches the repository and handles the request through a multi-turn coding conversation. The evaluation therefore reaches beyond isolated answers into assistance grounded in an actual codebase.
Once humans participate, interface behavior becomes part of experimental validity. Jain reports that Copilot Arena acceptance dropped sharply when latency exceeded about one second. That observation belongs to this completion workflow, not to all developer interactions. The paper measures acceptance of either suggestion against pairwise latency, which is determined by the slower model.
To make comparisons robust to response-time differences, the experiment balances latency across models. Both suggestions appear together after both responses finish, their positions are randomized, and sampling accounts for latency. Otherwise, an apparent preference for a model could partly reflect when its suggestion arrived rather than the quality of its code. Understanding the user’s behavior is part of designing the evaluation, not an adjustment to make after collecting votes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep the evaluation aligned with the work
As models improve, the task distribution changes along with task difficulty. Developers move from asking for a few tokens or lines to requesting tens or hundreds of lines. Refreshing evaluation sets therefore means more than adding harder problems: it means updating the kinds of work represented, while continuing to reduce exposure to contaminated examples.
Grading must evolve with that scope. Tests remain valuable correctness checks, but real repository work exposes behaviors they may fail to penalize. Jain closes with another concrete example: models can insert try/catch blocks everywhere to suppress failures. LLM judges can help assess those patterns, code quality, and arbitrary hacks that a passing test result may conceal.
For tasks that run long enough to produce substantial unfinished work, intermediate grading supplies the remaining feedback. A useful evaluation should reveal whether the system is making incremental progress before the entire task succeeds. That is what lets the evaluation keep pace as the unit of work grows from a snippet to a codebase.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Original benchmark paper explaining release-window evaluation, contamination analysis, and multiple coding tasks.
Official code and instructions for running LiveCodeBench evaluations.
Optimization tasks, evaluation harness, and tools for constructing tasks from performance-related commits.
Describes incremental code and test translation, dynamic analysis, and the Zopfli case study.
Explains paired IDE completions, preference collection, and controls for response latency.
Introduces repository-grounded conversations for understanding code, reviewing changes, and solving issues.
Updates since the talk
- GSO leaderboard and methodology updatesDocumentation
Defines optimization thresholds and records later changes to agent evaluation and reward-hack detection.
Read the complete timestamped transcript
- 0:00
[on-hold music] Hi, everyone.
- 0:21
So I'll be talking about, uh, like, some work on evaluations, particularly evaluations across, like, I guess I've done in the last four years. So let's get started.
- 0:32
So, uh, I'll be talking about co- uh, coding evaluations across varying time horizons. So I've been, uh, working on, like, in the code space for about four years now.
- 0:39
Like, it was right before, like, early Copilot came out, and my first project was actually working on generating, like, single-line pandas snippets, and my last project was generating an entire code base, so the field has, like, really progressed fa-- very quickly.
- 0:53
So I'll be talking about, like, uh, different stages of evaluations we have considered and some f-- uh, learnings across these other projects and how I see evaluations going forward.
- 1:01
So the first work I did was on, uh, like, uh, evaluating, uh, coding models in like second, uh, work doing in seconds of time, like generating single-line snippets, your Copilot code completions.
- 1:12
Then I work- did some work on, like, uh, evaluating on, like, interview-style competition programming problems, uh, which, uh, where models can work up to minutes. Uh, then we worked on some work on, like, uh, repository question answering, uh, which required like maybe, uh, more, uh, multiple minutes, tens of minutes.
- 1:28
Uh, and finally, like, uh, pushing the, uh, frontier forward, we are, uh, thinking about, uh, evaluating models on very complex tasks which can take hours or, like, multiple hours of work, like code optimization and, like, even further.
- 1:40
So let's get started. Uh, so first work I'll be talking about is, like, CodeBench, uh, which is, uh, like, uh, uh, evaluation work on, uh, models for, like, competition coding.
- 1:52
So here, uh, like, this is what a problem would look like. This is, like, very standard LeetCode problem, and don't worry, you don't need to solve something like this.
- 1:59
So, uh, like, uh, here, uh, as you can see, there's a problem, uh, statement. And, and the nice thing about these interview-style problems is that these problems are very well, uh, defined.
- 2:09
You have like, uh, good natural language specifications, some example input, output examples, so you can very, uh, reliably evaluate if the models are doing a good job or not.
- 2:17
So what is the motivation behind this and how, uh, we improve the frontier here. So the first challenge in, uh, evaluating, uh, language models these days is, like, data contamination.
- 2:27
These models are trained on, like, the entire Internet, and, uh, like on Stack Overflow, you'll find, uh, like very, uh, similar programming problems, puzzles. Uh, similarly, uh, like you'll find, uh, like, uh, very similar programming problem sources on GitHub, uh, or on the Internet.
- 2:42
So, uh, like contamination is a big, uh, deal. Uh, another very, uh, challenging factor we just struggled with the field is, like, insufficient test suites. So, uh, you'll see that, uh, like th- in this program, uh, like the goal was to return a sorted unique, uh, common e-elements between the two lists.
- 2:59
But, uh, like even a solution which does not do the sorting and just returns the set actually works because the tests were brittle and were not catching this mistake.
- 3:06
So, uh, like test suites is another, uh, like, uh, very challenging factor and how do we generate good and diverse tests. And finally, uh, difficulty distributions, which is something which people did not-- do not really, uh, reliably, uh, like calibrate.
- 3:20
Uh, like when I first was working, uh, in, uh, this space, uh, like there were two benchmarks available. On one benchmark, the performance was eighty percent or ninety percent, and on the other one it was one percent, and there was nothing in between.
- 3:32
And, uh, like as, uh, like benchmark users, what you care about is having some signal from the benchmark to, like, basically headline to make progress, to measure progress. And in, uh, either of these regimes, when if the problems are too easy or too hard, you don't get a lot of signals.
- 3:46
So it is very important when you're designing benchmarks to think about, like, the kinds of problems you are taking, and will it provide enough signal for the users of your benchmark.
- 3:55
So, uh, like in LiveCodeBench, we pioneered like dynamic evaluations. Uh, particularly, uh, like we can periodically update, uh, the evaluation sets, uh, and this gives you two, uh, very nice factors.
- 4:06
First is you can combat contamination. So you can evaluate the models on problems that were released after the model was trained, so it has likely not seen the problem, something like that.
- 4:14
Uh, and, uh, then you can also modify the problem difficulty distributions over time. So as we have talked about, models are incre-- uh, like improving very rapidly. Uh, so what was difficult, uh, for the model six months back might not be now.
- 4:27
So you can, uh, if you are updating your evaluation sets constantly, you can actually, uh, keep calibrate-- uh, the difficulty distributions calibrated, so you still get more signal out of your benchmarks.
- 4:37
So how we did that here, like, we had like an automated approach for curation of these problems, and, uh, similarly, we could, uh, automatically construct these test cases in an automated manner.
- 4:47
And, uh, this allows a very nice thing. When-- Since we are, uh, uh, like collecting problems over time, we have time as a control knob. So, like, we have these problem release months, uh, on LeetCode, and if you evaluate the model performances, like the pass@ rate one metric, uh, like, uh, on problems released over different months, you
- 5:05
will see that after, uh, like, uh, these model release dates, you would see stark drop in model performance. So like after DeepSeek was, uh, released in like September 2023, uh, the performance starkly drops from like maybe fifty percent average to like over like twenty percent or thir- uh, fifteen percent average.
- 5:21
So like, uh, based on these sliding windows, you can, uh, evaluate performance, measure contamination, and even combat contamination.
- 5:28
Um, and, uh, we have the running leaderboard, which is like very well maintained. And, uh, on this leaderboard you can actually, uh, like, uh, like view performances by, uh, scrolling this, uh, horizontal time bar.
- 5:39
And you'll see that as you are scrolling, uh, the contaminated models, which are the red bars, actually go down, which does highlight that, uh, like problem does, uh, like model performance does change on, uh, these newer kind of problems.
- 5:52
Um, finally, for, uh, test generation, we, uh, maintain, uh, like these, uh, test generation, uh, te-test generators. So if you have worked on fuzzing, you would have like input generators where you'd gen-generate diverse inputs.
- 6:04
And each of the problems are supported by like thirties or fifty inputs, so you can, uh, reliably find mistakes and bugs in, uh, incorrect code. And these are all automatically generated, uh, using an LLM-driven approaches.
- 6:16
And these problems, uh, have been like continuously being released and updated. So we have released like sift different versions of, uh, LiveCodeBench and these, uh, ne-- One of the nice things or one of the worrying things for me at the start was that, uh, like if you're constantly updating the eval sets, will, uh, like people be able
- 6:31
to keep track of them? Will, will people be using them, or will they just restrict to a single version? Uh, it turned out that these newer eval sets were constantly be update-- uh, like, adopted by different foundation model labs.
- 6:41
And, uh, like, uh, since we updated the problem difficulty over time, uh, the evaluation sets continue to provide strong signal to compare, uh, different models.
- 6:51
Um, so this was like LiveCodeBench. Let's talk about, uh, like something which is, uh, a more encoding agents, like more real-world programs. And this is, uh, work on, like, uh, software optimization.
- 7:01
So this is a problem I'm very excited about, and I'll ta- I'll talk about a few factors why you should maybe be excited about this. So, uh, here we are trying to, uh, measure model capabilities in generating high-performance software.
- 7:12
And, uh, I feel that this, uh, like problem domain, uh, like mixes two, uh, factors, like the algorithmic coding, uh, uh, field I talked about, which is like LiveCodeBench setting, but also like globa-global software editing, like, uh, SWE-bench and other like software op-- uh, ge- uh, general software engineering benchmarks.
- 7:28
Uh, uh, in high-performance, uh, software, you will have to do algorithmic work, you have to do deep analysis, and find, uh, uh, generate so-software with like right, uh, runtime.
- 7:38
So, uh, one of the key, uh, principles when we are trying to build this benchmark was like ensuring construct validity, because when you see a lot of benchmarks today, uh, we, uh, get very high benchmark scores, but at a lot of the times they don't really translate to real-world performance gains.
- 7:53
So construct validity refers to how close, uh, a measurement reflects the underlying, uh, concept it's meant to measure. So like here we are measuring code optimization, and we want something which is, uh, like, uh, reliably evaluates real-world, uh, takes.
- 8:06
So this usually re- requires like two aspects. First is like the task distribution. Your task should be natural and sourced from the real world, and then you should be able to reliably grade them.
- 8:16
So let me talk about like what steps we take to, uh, make this happen and how we construct this benchmark. So let's say we take a code base like llama.cpp.
- 8:25
Uh, we take, uh, uh, we crawl over all the commits of the code base, and we find the commits which are opt-- uh, like doing something, uh, related to performance optimization.
- 8:33
So here there was this commit which is optimizing the quantized performance of, uh, like, uh, certain kinds of models. Uh, for all of these, uh, commit-- uh, performance optimizing commits, we would, uh, like generate performance test cases.
- 8:45
Um, and, uh, these performance test cases would look like some workloads. And, uh, once we have these workloads, uh, we have a very, uh, nice and precise way to specify the problem statement that, uh, given this workload of, let's say, uh, running a Qwen, uh, seven-B model, uh, can, uh, we give this, uh, problem to, uh, SWE-agent
- 9:03
and ask the model to optimize the code, uh, the llama.cpp repository, so this code runs faster. So as you can imagine, this co-- uh, task is like fairly challenging.
- 9:10
You need to understand, like low level, uh, implementation details, uh, and, uh, like how quantized models, uh, behave, how we can, uh, improve the runtime. And so models can generate a patch, and the evaluation is done on whether the patch is correct.
- 9:23
So does it pass the equivalence check with the human patch? And, uh, is there a, a valid optimization over the, uh, reference human patch? Uh, that is, uh, whether you can, uh, generate a better runtime than what a human could do.
- 9:36
S- so, uh, like, uh, this is a very challenging task. We have like hundred plus optimization tasks sourced in this manner, and this is like fairly, uh, like important in, uh, like, uh, like high-p-performance settings.
- 9:47
So think about like data science, uh, like ML visualization scenarios. Uh, our, uh, benchmark, uh, like comprises of like various, uh, low-level, uh, code, like C/C++ Rust. And the very nice thing is like these are precise problem statements.
- 10:01
You can, uh, easily specify to the model what is the goal in the form of a performance test which the model has access to, and it can continuously iterate over it for a long time.
- 10:09
So here we can scale the test-time compute and pick the best solution based on, uh, the test cases that we have. S-- And this can happen like synchronously or asynchronously.
- 10:19
So, uh, like, uh, we generate these performance test cases, and, uh, that worked, uh, reasonably well, but, uh, we found that there were, uh, like cases of reward hacking here.
- 10:28
So what do I mean by reward hacking? Like friendly models would write non-idiomatic code to like actively exploit the evaluation infrastructure or overfit the test distributions. So one funny example we saw was like our models would add like LLU cache to pan-- uh, like arbitrary pandas methods when we were, uh, trying to optimize pandas and the, uh,
- 10:46
efficient solution should have required changing something in the internals. Uh, so we try to pass this by changing our evaluation in-infrastructure, so it's like more robust to this kind of hacking, uh, approaches.
- 10:57
But then we saw something, you know, like even more drastic. Models would sometimes completely hijack the infra where, uh, they would add, uh, like site-customized .py file where-- which runs at the start of Python runtime, and it would basically change the NumPy library, uh, like which was i-installed in the code base or something it crawled from, uh,
- 11:15
source. And there is-- Like I think you can do some, uh, ways to, uh, like take some measures to make your evaluation infra, which is robust to these kind of, uh, like adversarial, uh, like attacks.
- 11:26
But, uh, here, uh, like there could be myriad ways in, uh, in which models can hack these kind of scenarios. And here, uh, we propose, uh, like HackDetector, where, uh, which is a detection system that leverages GPTF like code analysis capabilities and test-time compute to like basically identify these kind of hacking behaviors at runtime.
- 11:44
So you don't have to imagine all the possible failure scenarios at the start. So what it would take is like a model patch, the expert patch and test cases, and we'll ask GPT-5 to give like verdicts on like whether it's reward hacking with some kind of explanation.
- 11:57
Uh, we'll do, uh, do this a few times and take the consensus. And based on this con-census, we'll determine if, uh, this is, uh, doing some like a non-idiomatic coding patterns or not.
- 12:07
And, uh, we did some fai-failure analysis based on this. So now you can detect mistakes using test cases, whether the code is correct or not, whether it is optimizing or not.
- 12:15
But you can also detect reward hacks using this, like, LLM-as-a-judge, uh, factor. And wh- uh, what you see is kind of surprising. Uh, like, models make a lot of, like, uh, correctness mistakes that you can catch by tests.
- 12:27
But even if the code passes the test cases, like, O3 attempted reward hacking patterns in, like, 30% of the problems it tried. And this fraction is, like, going down, uh, for the newer models to some degree, but it is still existing.
- 12:38
And as we go to more and more real world tasks, uh, this is going to, uh, get more challenging, and we need to figure, uh, like, ways to combat these kind of reward hacking patterns by using LLM judge and other, uh, ways to make this evaluation infra more, uh, reliable.
- 12:53
So next I'll talk about, like, uh, s- like, specifically some of our new work on, like, uh, like, pushing the b-boundary of code evals even further and, uh, taking a look at more challenging tasks.
- 13:03
So here, uh, we were, uh, thinking about, like, can, uh, like, these language models translate, uh, like, a entire code base? Uh, specifically given a specification as a C program, can you generate a C++ implementation for the same?
- 13:17
And we took a fairly complex, uh, code base. So Zopfli is a, like, highly efficient compression library from Google. Like, it has about, like, 4,000 lines of code, hundreds of functions, and complex data structures.
- 13:28
Uh, and, uh, we want, like, um, like, very precise and correct code, so we, uh, generated, like, a million compression inputs, and your test case was to generate a Rust implementation that, uh, maintains correctness over those million test cases.
- 13:40
And when I did this work back in, uh, like, uh, last year, it took us 12 hours to actually do this translation. Now, perhaps with better models, this can be done in two hours.
- 13:48
But still, I think, uh, this is pushing the frontier of, like, what the models can do currently. Um, so what is one of the key findings when we are trying to make progress in s- uh, something like this?
- 13:58
Like, end-to-end correctness is i-important, but it only gives you, like, one bit of feedback. But for these very long horizon tasks, one thing which, uh, will become, uh, more important going forward is, like, having some s- uh, measures of in-intermediate correctness.
- 14:11
So, like, for our case, we could, uh, measure, like, fraction of code translated, fraction of code refactored, and based on these kind of settings, you can, uh, understand, like, h- if you're making progress or not, and how you can, uh, scale the systems better.
- 14:25
Um, so, like, uh, as we're closing, I'll talk about, uh, like... I'll quickly talk about some of the work I did on, like, in-the-wild evals. So this work was done in collaboration with, uh, Llama Arena folks.
- 14:35
And, uh, like, I'll talk about two settings here. First is Copilot Arena. So this is, like, evaluating in IDE, uh, code completion assistance. So what we will do here is we'll, uh, have an IDE plugin with, uh, like, uh, similar to, uh, GitHub Copilot setting, uh, we'll generate a completion for you, but instead of just a single
- 14:51
completion, you'll have, uh, two completions appearing, like, um, top and, uh, down. And you can, uh, pick either one of them via shortcuts like Tab or Shift + Tab.
- 15:00
And, um, based on the, uh, like, acceptance rates, we can pairwise compare what the code completion assistants are doing.
- 15:07
S- uh, uh, we also did some work on RepoChat, where, uh, like, uh, to evaluate, uh, like, code question answering capabilities of models, uh, we, uh, built a system where you can provide a GitHub URL, uh, and you can ask a natural language query about the code base, which could be something about explain the code base to
- 15:23
as complex as, "Let's try to solve this issue. Let's give me... give me a model patch that could s-uh, solve this issue." And, uh, uh, we integrated a very basic and simple, uh, like, SWE-agent system that fetches the code base, resolves user queries in, like, multi-turn, uh, code assistant, uh, conversations.
- 15:40
So, uh, one thing that stood out to me in these kind of things, uh, is, like, like, how human-centric experiment design, uh, needs to be. So, uh, like, for code, uh, like, Copilot Arena in particular, we realized that, like, uh, latency is a big concern for acceptance rates.
- 15:57
So if you look at the acceptance... uh, like, latency below and acceptance rates, like, if it is, like, anything more than one second, uh, like, the acceptance rates drop very starkly.
- 16:06
So people care a lot about latency, so you have to... so we had to, uh, design an experiment so that it's robust to these kind of, like, uh, latency differences between models, balanced latency across different models.
- 16:15
So, like, if you're doing, uh, like, anything in the wild, having this human-centering component, understanding human behaviors is very important to do anything meaningful. So, uh, at the end, I think, uh, just to, uh, recap, like, I think I talked about a bunch of works.
- 16:30
Uh, like, what are some, uh, big takeaways? So I think, uh, dynamic, uh, dynamically updating evaluation sets to, like, prevent contamination, like, modify the problem distributions, like, in terms of difficulty, in terms of distribution of tasks we care about.
- 16:43
As we, like, uh, improve... uh, as the language model capabilities will improve over time, the, uh, types of tasks will, uh, start to do with model chains. You can even, uh, think of this, like, uh, we were doing, like, code completion where we were generating, like, few tokens, few lines, and now we are generating, like, uh, tens
- 16:58
of lines, hundreds of lines. And to some degree, this, uh, will, uh, continuously change, and we have to update our evaluation sets, uh, so that it reflects the real world usage and, uh, kinds of things people need.
- 17:08
Um, the second very, uh, important thing is, like, ensuring reliable grading in this domain. And, like, tests are very good for ensuring correctness and, uh, provide a lot of reliable feedback, but, uh, once we go to real world settings, like, models can, uh, start doing, like, lot of non-idiomatic coding patterns.
- 17:24
They will add try catches everywhere to just prevent any kind of bug from occurring. So having these kind of LLM judges to detect non-idiomatic coding patterns, code quality, and just any, uh, like, arbitrary hacks, uh, will be very important.
- 17:36
And finally, like, as I talked about in the last work, uh, like, intermediate grading signals so that you can measure, like, incremental progress, uh, is, uh, like, another key factor here.
- 17:46
So I think that's, uh, the end of my talk. Thank you. [upbeat music]