SWE-rebench: Lessons from Evaluating Coding Agents on Real Software Engineering Tasks — Ibragim Badertdinov, Nebius

Talk transcript

1. 0:00

   [upbeat music] Okay.

2. 0:16

   Okay, great. Th-thank you. Then I think we could start. So my name is Ibragim. I will share with you the lessons that we learned, uh, through our evals of coding agents and different models on the real-world software engineering task using as the example our SWE-rebench Leaderboard.

3. 0:33

   Um, I want to share some practical lessons mostly, and I think that evals matter now even more than before because we have a lot of models, closed source, open weight models that are doing really great in the software engineering domain.

4. 0:47

   And of course, you can rely on your gut feeling, vibe checks, or maybe one or two your most favorite questions to choose between the options. But everything is fine until you roll out something into the production and it just breaks down and clients are unhappy.

5. 1:02

   So I think that we n-need to evaluate everything. And before we will deep dive, uh, I want to share a small fact about me. Uh, so actually, I have a very non-traditional background for AI research.

6. 1:15

   I'm a dentist by training. That's me ten years ago. And that's why on my Google Scholar, I have papers from like NeurIPS and ICML about RL and test-time scaling, along with some psychotherapy or medical insurance problems in dentistry.

7. 1:29

   And in the medicine, cost of every mistake is really high. And I think that, uh, for the AI domain, we also could say that, uh, cost of each mistake is higher than traditional software engineering.

8. 1:42

   And actually, I should say that I believe that, like, dental pain and infrastructural pain are kind of similar because both of them, uh, will not let you sleep at night.

9. 1:51

   But with dental pain, you could go to the dentist and he will, uh, cure you. But with infrastructural pain, you need to do with it something by yourself. So about our leaderboard.

10. 2:03

    Let's break down word by word what do we do. So SWE-rebench, it's fresh real-world software engineering task on thirty models evaluated every month. So what does it mean fresh?

11. 2:13

    Most of the benchmarks, uh, during their release, they release questions and solutions. So implicitly or explicitly, this data can become a part of the pre-training of the next generation of models.

12. 2:25

    So if you want to build some open, truly decontaminated benchmark, um, time splits are the only way. That's why every month we collect only fresh problems from the previous month and then assess the model's capabilities.

13. 2:40

    In terms of the real world, in pre-LLM era, there were a lot of benchmark about, for example, um, some bracket sequence or ordering correctly adjectives in English. But now we need some natural problems that people could ask systems to do, and even more, some well-paid problems like software engineering, for example.

14. 2:59

    Also, software engineering problems and tasks are not about just simple question answering. They are truly subtasks, so it means that to solve the issue or implement the feature, you need to understand the structure of repository.

15. 3:14

    You need to try to write some tests, implement the solution, run the test, uh, reproduce the mistakes or, uh, bugs. And also, it is some multi-turn and naturally long context task.

16. 3:26

    So it's not just concatenating some text or books. No, it's truly long context task. And also, it is about tool use, harnesses. So that's why I believe that software engineering domain is really valuable for evaluations.

17. 3:38

    We also evaluate something like thirty models with the same hardness, simple same hardness. And for the reference, we also give, uh, some numbers for Claude Code, Codex, and Juni harnesses, and we'll add actually more in the report, a lot of stuff.

18. 3:53

    And I always read all the comments on, uh, local Llama subreddit and X and try to add most actual and interesting models. Of course, we get requests like, "Okay, can you please evaluate some obliterated roleplay, sixty-nine billion parameters agent?"

19. 4:08

    But we mostly stick to the most popular ones.

20. 4:12

    About the tasks. Um, for any verifiable software engineering task, actually, we have three main components. It's similar for SWE-bench, SWE-rebench, other domains, TerminalBench. You have some task description. For us, it's just original issue title and description, uh, from the given timeframe, um, from some permissive but popular open source repository.

21. 4:34

    Uh, for the sandbox, you can call it environment, RL environment, sandbox, snapshot, but basically it's just an executable Docker image with the installed dependencies so we could run the test of the project.

22. 4:46

    And the third one is a verifier. Basically, it's just a test from the pull request that solved some issue or implemented, uh, some feature. And here I could say that there is actually two sets of tests fail to pass.

23. 5:00

    It is the test that should be failed before, uh, solving the issue, for example, and should be passed after. And pass to pass, it's something like regression test. And also, uh, it's important to say that every test is not just a question, but mostly it's some Docker image, one or ten gigabyte, so you need a good infrastructure

24. 5:19

    actually to run everything. I think that this is one of the most important, uh, slides. I will share the presentation on X or could send you. But the thing is that every month we verify every task and, uh, we have a really big, uh, bank of the problems with the task because I believe that it is not

25. 5:39

    too easy to say what is-- it look like a perfect task, but we can say what makes it bad. So for problem description, you actually need something balanced, not too vague, not too overspecified, not too easy, not too hard, because for too easy problems, all the models will solve it and your effective size of benchmark- Uh, will

26. 6:00

    be less. For the verifier and test, here is one of the examples. So usually, software engineers write the test after implementing some solution, so they may be some kind of overfitted.

27. 6:12

    Here, for example, test require the agent to generate exact substring in the error, uh, message. So even with the correct solution, this passed will... Uh, this test will not be passed.

28. 6:22

    And you need a stable infrastructure because you need to minimize the infrastructural noise, uh, during your runs. Uh, for example, your test could connect to some external resources, and it will be some dependency.

29. 6:35

    Or we had a problem in one of pipelines, so several images just, uh, get some default time, like 1970s, and some tests were relied on that. So we just, uh, get some problems with these kind of evaluations.

30. 6:50

    Uh, in my opinion, for our benchmark, um, collection is mostly a filtering problem because we have a really good source of task and information, like GitHub. We use GitHub Archive, um, as main source for pull requests and issues for large scale projects and just GitHub API for the smaller ones.

31. 7:08

    Here, one hundred percent is, uh, number of pull request linked with some issues. So for example, if you need a lot more data for pre-training runs, for example, or post-training runs, uh, if you will use just pull request, it will be eight times, uh, bigger data set.

32. 7:25

    We use interactive agent to install all the dependencies and project, so we could use this Docker image. And we also have some several steps of, uh, just LLM message filtering with the, uh, like, most common problems.

33. 7:40

    But at the end, we try to choose s-sample that is ten percent bigger than we need in our final runs because after running some models, you could face, uh, problems in terms of task quality that could be visible only after, uh, agents will try to solve it.

34. 7:58

    And for the final set of task, we manually verify. I think it's one full-time, uh, day of work to manually verify each, uh, task, so we could make sure that they are solvable but quite challenging.

35. 8:12

    Here is the slide about our hardness and agent. I believe that it is better to have some minimalistic agent with strong infrastructure than having over-engineering agent with weak infrastructure.

36. 8:25

    It's an example of the most popular tools and bash commands in our scaffold with, uh, Claude Opus 4.6. Uh, so with upper case, it is, uh, agent's tools, and lower case, it's bash commands.

37. 8:39

    And actually, the most popular ones is quite simple. And we also run our agent in a YOLO setup, so it means that we don't want to... Our agent to ask some clarification questions or something like that to just, uh, need to solve the issue.

38. 8:53

    And we start with some simple React plus, uh, demonstration that you have in your prompt demonstration how to use your tools. But nowadays, uh, every model is, uh, quite good in tool calling, so we just minimize our context as well.

39. 9:09

    So about what breaks in practice with the agents. I think that every month we have one or two, uh, model runs that just became invalid because some problems. Uh, first of all, I-- You need to define your retry policy.

40. 9:24

    You actually want to separate your errors of the model and some infrastructural errors. So you need to define what exit stats. For example, uh, too long context or too many tool calls or your provider errors.

41. 9:38

    Will you rerun these runs or not? For the caching, it actually really improves, uh, your cost efficiency. I hope you know about that. Here is an example with our simple agent.

42. 9:48

    It's very similar to software e-engineering agent or Minitry agent by SWE-bench creators. So with the caching included, your, uh, your cost will be, like, four times less. But for Claude Code, it actually spends a lot of tokens.

43. 10:02

    So even with journal caching and, like, Haiku sub-agents for some sub-task will actually cost quite a lot. And we-- After one of the runs, we saw that, uh, during the updates of the models, even within a same, uh, family, for example, like GPT-5.2, GPT-5.4 or the longer or more older versions, there could be some, uh, default

44. 10:27

    parameters drifting for the reasoning level, for the caching level or other stuff that you also need to make sure that is, uh, relevant and work in your infrastructure. That's why I believe that, first of all, you need to try to run some external benchmark, like SWE-bench and any other TerminalBench on your infrastructure to make sure that actually

45. 10:47

    your numbers and, uh, reported numbers match, and only then start to do your experiments. Here's the most, uh, favorite slides. Uh, so, uh, we found at least two ways how, how models cheat.

46. 11:01

    First one is a well-known issue. It is all about Claude Code here, but it will be also about Codex and other models as well. So the thing is that, uh, during our runs before-- When we build our Docker image, we do a checkout to the base commit before the solution was implemented.

47. 11:19

    So agent will start doing something there. And if you will run command Git Log with all flag, then you will get an access to the overall Git history. So that's how, for example, Claude Code just, uh, look up to the future, to the solution patch, and copy-paste it, and so successfully solved this issue.

48. 11:40

    After that, we remove all the future Git history because, uh, previous Git history might be helpful to get some context working with the issue, but we need to remove the future one.

49. 11:50

    After that, Claude Code came up with the WebPage tool. It has a WebPage tool, so it just went to GitHub, uh, uh, repository, original one, to see the conversation in the original issue pull request and solved it.

50. 12:03

    Okay. After that, we restricted WebPage tool. So Claude Code, okay, I have curl. Let's just use bash command with curl. We'll go to the original issue. Here you can see that actually Claude Code also formatted the conversation to be more convenient, and then just check the original test in the main and solve the issue.

51. 12:24

    So when models, uh, get better, actually, I believe that they may- might, like, tend to cheat even more and do some reward hacking. So we solve only with some kind of post-processing and trajectory analysis and try to come up with, uh, new solutions as well.

52. 12:41

    I think that one of the main reasons why we made, uh, this benchmark and, uh, maintain it, we want to share some practical value with the real AI engineers and AI creators.

53. 12:53

    So that's why we report not only some mean resolved metric, but also, um, tokens per problem, price per problem, and we do five, uh, runs for each, uh, task to report some confidence intervals and also pass at five something like if, uh, model solved, uh, each test at least, we, uh, think that it's successful to give some

54. 13:15

    kind of potential of the model. Also, you can check something like pass all five if you need reliability. So you will, uh, mark, uh, the task as successful only if agent solve it in all five runs.

55. 13:27

    Um, after some analytics in terms of economics tokens and, uh, price per problem, we also want to do something on trajectory level because I think that it is a source of a lot of insights about how some models, uh, work in our or external harnesses.

56. 13:46

    And the next one is about if you know how to make a, well, evaluation or benchmark, you could use the same pipeline to collect some validation set, for example, and to think about training.

57. 13:59

    And I don't say about, like, SFT or RL. At first, you could just try with choosing between models, harnesses, and parameters on your validation set, and then maybe do some kind of auto research or just update your prompts and tools.

58. 14:14

    Then do some, uh, simple rejection sampling, fine-tuning, or distillation from the, uh, bigger models, and then move to more complex strategies like GRPO. So we use the same pipeline that we use for SWE-rebench Leaderboard to make two big open source releases.

59. 14:30

    First one is, uh, SWE-rebench. We released it, uh, last year. It is something like thirty thousands of RL environments, like real-world software engineer tasks with Docker images, and it was used by some frontier labs to train better models.

60. 14:46

    And now we also release SWE-rebench V2. It is something about, uh, software engineering tasks on twenty programming languages. Also, a lot of Docker images, a lot of tasks that could be used for training.

61. 14:57

    I will work on adoption for it. We also have an adapter for Harbor, our TerminalBench, which is quite convenient format to run any evaluations or the training. And I think that, uh, for the future, we need to think about more long-horizon tasks, more about something complex, and something about code quality as well.

62. 15:16

    Because if you will check any patch from SWE-bench submission or SWE-rebench submission, you will see, um, some problems that actually, uh, the real developers will not do. And during the review, uh, you will say that, "Okay, it's not how things work actually."

63. 15:31

    For example, Gemini, Dual LM, GPD models, uh, they tend to produce some reproduced tests or files and then just don't remove it. We also can talk about some code quality during the poll request.

64. 15:45

    So yeah, I think that we need to come up with some long-horizon task, more trajectory analysis, and then move on to training better models.

65. 15:54

    So yeah, that's it. Please check the Leaderboard, SWE-rebench Leaderboard. Update every month. I will be here. Uh, feel free to reach out. This is my X handle, and I will release, like, new open source project and also will share these slides, I think, tomorrow.

66. 16:09

    Yeah. Thank you for your attention. [audience applauding] [upbeat electronic music]