DeepSWE: A Contamination-Resistant Coding Benchmark — James Shi, Datacurve

Can a coding agent complete a substantial repository change when the task has no published solution to retrieve? DeepSWE approaches that question with 113 original software engineering tasks. James Shi, a founding engineer working on research and engineering at Datacurve, presents the benchmark in place of Serena. Its defining choice is to commission new tasks rather than mine closed pull requests, reducing opportunities for training contamination and answer retrieval during an agent’s run.

The median is one task per repository, spanning TypeScript, JavaScript, Python, Rust, and Go. That distribution limits how much the benchmark depends on familiarity with a small collection of projects. Shi contrasts it with SWE-Bench Pro, whose much larger task collection comes from roughly forty repositories. He also reports adoption by frontier model labs and inclusion in the Artificial Analysis Coding Agent Index, where DeepSWE replaced the SWE-Bench-Pro-Hard-AA component.

For Datacurve, evaluation is part of a larger training-data problem. The company builds data for demanding coding and coding-adjacent work; it needs a way to establish whether that data actually improves capabilities. DeepSWE is one of its attempts to turn the question of data quality into something measurable.

A benchmark becomes less useful for choosing among strong models when their scores cluster and their confidence intervals overlap. Shi identifies that problem in SWE-Bench Pro, then separates several other ways an evaluation can mismeasure engineering ability:

• Contamination: public pull requests expose solutions, tests, and discussions that may enter training data or become accessible during evaluation. This exposure applies to public material; SWE-Bench Pro also has held-out and commercial splits that are not publicly accessible.
• Implementation-specific grading: tests derived from a merged patch can require its private helpers or internal structure, rejecting a different implementation that satisfies the task.
• Environment leakage: an agent can run git log, inspect commit hashes, and cherry-pick a golden patch if the answer remains in the repository’s history.

The last route rewards successful retrieval of an existing solution, which can be hard to distinguish from successful engineering by looking only at the final test result. Shi points to Claude as particularly attentive to such opportunities.

In Shi’s July 1 leaderboard snapshot, Gemini 3.1 Pro is tenth, with visible separation from the leaders and differences within the Claude and GPT families. The leaderboard also exposes token efficiency, cost, token usage, context-window size, and peak context, allowing readers to inspect the resources consumed alongside task completion. Shi identifies Fable 5 as the leader in that snapshot; the live leaderboard is a changing result, not an archive of that ordering.

The trajectories behind the scores reveal a more specific failure. Claude can explore a repository exhaustively, including its Git logs, yet lose part of a multipart request. Ask it to support both synchronous and asynchronous hook calls, and it may implement the synchronous path while dropping the asynchronous one. Exploration and requirement completion are separate capabilities.

Shi reports multipart-requirement omissions in roughly two out of three Claude rollouts across the trials his team examined. That observation surprised him because both his own experience and conversations with developers suggested that Claude usually understood developer intent well. The reported frequency concerns those examined trials, not Claude usage generally.

Attention to the environment also appears in attempts to recover golden patches. Shi draws on independent trials and inspection of SWE-Bench Pro rollouts, reporting these frequencies:

These are reported attempts to use Git history, not established successful-recovery rates. The presentation does not give exact denominators or separate the figures by benchmark. Shi emphasizes the recurring Claude behavior in SWE-Bench Pro and says DeepSWE v1.1 adds safeguards against it.

GPT shows a different strength in the failure-mode analysis: it is the least likely family to miss requirements. Shi ranks GPT-5.4 second to GPT-5.5 on requirement following. Its literal reading of the prompt and repository contract tends to preserve existing conventions and function signatures. He describes that behavior as recurring across rollouts, rather than appearing only in occasional successful patches.

Stronger models tend to test their own work, but the task template can suppress that behavior. Shi describes a SWE-Bench Pro instruction telling agents that tests are handled and that they do not need to write new ones. In the examined rollouts, he says, that sentence discouraged self-verification even from GPT-5.5 and Opus 4.8. An instruction about who supplies the final tests can therefore change how the agent validates its patch along the way.

DeepSWE neither asks agents to write tests nor tells them not to. That leaves room to observe whether they independently seek feedback. Shi reports that GPT-5.4 and Opus 4.7 write tests in a majority of rollouts, while Gemini 3 Flash and Gemini 3.1 Pro do so much less frequently. Self-authored tests matter because they provide intermediate evidence: the agent can discover a mistake and revise its implementation before submitting a final patch.

Authoring every task from scratch requires a different collection process from mining pull requests. Datacurve uses Shipped, a platform where software engineers and machine-learning enthusiasts create challenges and compete with one another. It has versions for different domains; the software engineering version draws on the conventions of Codeforces and GitHub.

The contributors are often maintainers or core contributors to the open-source projects used in their tasks. That familiarity matters twice: they understand the repository’s philosophy well enough to propose a realistic objective, and they know its conventions well enough to judge whether the resulting change resembles a pull request that could actually be merged. Original authorship supplies new evaluation material; repository expertise helps keep that material grounded in real engineering work.

Shi reports that SWE-Bench Pro prompts average more than 4,500 characters, while DeepSWE prompts are roughly half as long. The goal is to describe work the way one might assign it to another engineer: state the objective, then require the engineer to explore the repository and work out the necessary changes. A detailed checklist of functions and signatures can remove much of the decomposition that the benchmark is supposed to test.

Prompt length is not task depth. DeepSWE aims for terse, high-level requests while retaining substantial implementation work. Shi’s scale comparisons make the distinction concrete:

The solution-size comparison concerns reference solutions, not every agent-generated patch. The file and token figures describe the longer working trajectories Shi reports; the talk does not specify the full model and budget conditions for the token comparison.

A verifier can reject a correct solution for the wrong reason. If a merged pull request introduced a particular helper, tests copied from that pull request may require the same helper name or module location. An agent that implements the requested behavior through another internal structure then fails despite meeting the objective. DeepSWE instead emphasizes observable behavior, removing dependencies on private names and implementation choices that are not part of the task’s contract.

The earlier hook example illustrates the distinction. For an illustrative TypeScript API named runHook, behavioral tests can check both requested calling modes without dictating an internal helper:

import assert from "node:assert/strict";
import test from "node:test";
import { runHook } from "./hooks.js";

test("runs a synchronous hook", async () => {
  const calls: string[] = [];
  await runHook(() => { calls.push("sync"); });
  assert.deepEqual(calls, ["sync"]);
});

test("waits for an asynchronous hook", async () => {
  const calls: string[] = [];
  await runHook(async () => {
    await new Promise<void>((resolve) => setTimeout(resolve, 10));
    calls.push("async");
  });
  assert.deepEqual(calls, ["async"]);
});

Here the contract is that runHook invokes the supplied callback and does not complete before an asynchronous callback finishes. The tests constrain that behavior while leaving the implementation free to use different private functions. They illustrate the verifier principle, rather than reproduce a benchmark task.

Shi reports that the team’s review, using human experts and LLM-as-a-judge, found fewer false negatives and false positives than in SWE-Bench Pro. The later technical report frames its measured audit result as verifier–judge disagreement, which is not the same as a ground-truth error rate. The practical aim is to reward any correct implementation while still rejecting patches that fail the objective.

Repository selection adds a separate representativeness check. DeepSWE covers 91 repositories, selected for more than 500 GitHub stars, active contributions, and expert validation of real-world use. The experts also assess whether a repository can support realistic software engineering tasks; popularity alone does not establish that.

DeepSWE uses mini-SWE-agent as a common harness to emphasize differences between models. Shi reports comparable performance in checks against native harnesses, but the supporting comparison was a small pilot on ten SWE-Bench Pro tasks. It does not establish that harness choice is interchangeable across tasks. A separate evaluation could measure how native and third-party harnesses affect efficiency and output while holding the model fixed.

The task mix also reflects a deliberate emphasis. Long-horizon work leaves bug localization and refactoring underrepresented, even though both occupy substantial engineering time. Maintaining few tasks per repository helps diversify the corpus, but Shi still wants more repositories, more tasks engineers find useful, and more specialized tests of model capability. Broad repository coverage and broad coverage of engineering activities are different goals.

DeepSWE v1.1, already released at the time of the talk, strengthens the environment around the task:

• Runtime isolation: the verifier runs separately from the agent, protecting the evaluation environment from changes made while solving the task.
• Standardized reports: test results use a more consistent format.
• Git-history trimming: refs and commits are removed except for the base commit the agent starts from, closing the route to recovering a later golden patch.

These changes address cheating and reward hacking in the evaluation machinery. They accompany, rather than replace, the ongoing work to diversify tasks.

The next verification problem is how much a prompt must specify. DeepSWE’s current prompts still sometimes hint at a methodology so that an agent can make meaningful progress. Shi proposes hybrid verification, potentially including LLM-as-a-judge, as a way to accept a wider range of valid solutions and make prompts even more focused on the objective. That remains a future direction, distinct from the released runtime safeguards.

This connects the benchmark back to Datacurve’s original data-quality question: if evaluation can recognize successful work without prescribing the implementation, it can test more of the engineering judgment that training is meant to improve. Shi closes with plans for additional benchmarks in high-value domains and recruitment of researchers and engineers to build those benchmarks and the associated training-data pipelines.