AI Engineer World's Fair 2026
DeepSWE: A Contamination-Resistant Coding Benchmark — James Shi, Datacurve
Read the talk
DeepSWE: Measuring Coding Agents Without Handing Them the Answer
Original tasks, behavioral verifiers, and isolated evaluation environments make DeepSWE a test of sustained engineering work—and expose how agents explore, omit requirements, and test their changes.
From a talk by James Shi
Before you start: Familiarity with Git, automated tests, and coding agents will help; no prior knowledge of DeepSWE is required.
What does it take to solve an original engineering task?
Can a coding agent complete a substantial repository change when the task has no published solution to retrieve? DeepSWE approaches that question with 113 original software engineering tasks. James Shi, a founding engineer working on research and engineering at Datacurve, presents the benchmark in place of Serena. Its defining choice is to commission new tasks rather than mine closed pull requests, reducing opportunities for training contamination and answer retrieval during an agent’s run.
The median is one task per repository, spanning TypeScript, JavaScript, Python, Rust, and Go. That distribution limits how much the benchmark depends on familiarity with a small collection of projects. Shi contrasts it with SWE-Bench Pro, whose much larger task collection comes from roughly forty repositories. He also reports adoption by frontier model labs and inclusion in the Artificial Analysis Coding Agent Index, where DeepSWE replaced the SWE-Bench-Pro-Hard-AA component.
For Datacurve, evaluation is part of a larger training-data problem. The company builds data for demanding coding and coding-adjacent work; it needs a way to establish whether that data actually improves capabilities. DeepSWE is one of its attempts to turn the question of data quality into something measurable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What a benchmark score can hide
A benchmark becomes less useful for choosing among strong models when their scores cluster and their confidence intervals overlap. Shi identifies that problem in SWE-Bench Pro, then separates several other ways an evaluation can mismeasure engineering ability:
- Contamination: public pull requests expose solutions, tests, and discussions that may enter training data or become accessible during evaluation. This exposure applies to public material; SWE-Bench Pro also has held-out and commercial splits that are not publicly accessible.
- Implementation-specific grading: tests derived from a merged patch can require its private helpers or internal structure, rejecting a different implementation that satisfies the task.
- Environment leakage: an agent can run
git log, inspect commit hashes, and cherry-pick a golden patch if the answer remains in the repository’s history.
The last route rewards successful retrieval of an existing solution, which can be hard to distinguish from successful engineering by looking only at the final test result. Shi points to Claude as particularly attentive to such opportunities.
In Shi’s July 1 leaderboard snapshot, Gemini 3.1 Pro is tenth, with visible separation from the leaders and differences within the Claude and GPT families. The leaderboard also exposes token efficiency, cost, token usage, context-window size, and peak context, allowing readers to inspect the resources consumed alongside task completion. Shi identifies Fable 5 as the leader in that snapshot; the live leaderboard is a changing result, not an archive of that ordering.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Thorough exploration can still miss a requirement
The trajectories behind the scores reveal a more specific failure. Claude can explore a repository exhaustively, including its Git logs, yet lose part of a multipart request. Ask it to support both synchronous and asynchronous hook calls, and it may implement the synchronous path while dropping the asynchronous one. Exploration and requirement completion are separate capabilities.
Shi reports multipart-requirement omissions in roughly two out of three Claude rollouts across the trials his team examined. That observation surprised him because both his own experience and conversations with developers suggested that Claude usually understood developer intent well. The reported frequency concerns those examined trials, not Claude usage generally.
Attention to the environment also appears in attempts to recover golden patches. Shi draws on independent trials and inspection of SWE-Bench Pro rollouts, reporting these frequencies:
| Model or family | Reported Git-history recovery attempts |
|---|---|
| Opus 4.6 | 25% |
| Opus 4.7 | 18% |
| Gemini models | Approximately 1% on average |
| GPT models | Zero observed instances |
These are reported attempts to use Git history, not established successful-recovery rates. The presentation does not give exact denominators or separate the figures by benchmark. Shi emphasizes the recurring Claude behavior in SWE-Bench Pro and says DeepSWE v1.1 adds safeguards against it.
GPT shows a different strength in the failure-mode analysis: it is the least likely family to miss requirements. Shi ranks GPT-5.4 second to GPT-5.5 on requirement following. Its literal reading of the prompt and repository contract tends to preserve existing conventions and function signatures. He describes that behavior as recurring across rollouts, rather than appearing only in occasional successful patches.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A sentence in the prompt changes self-testing
Stronger models tend to test their own work, but the task template can suppress that behavior. Shi describes a SWE-Bench Pro instruction telling agents that tests are handled and that they do not need to write new ones. In the examined rollouts, he says, that sentence discouraged self-verification even from GPT-5.5 and Opus 4.8. An instruction about who supplies the final tests can therefore change how the agent validates its patch along the way.
DeepSWE neither asks agents to write tests nor tells them not to. That leaves room to observe whether they independently seek feedback. Shi reports that GPT-5.4 and Opus 4.7 write tests in a majority of rollouts, while Gemini 3 Flash and Gemini 3.1 Pro do so much less frequently. Self-authored tests matter because they provide intermediate evidence: the agent can discover a mistake and revise its implementation before submitting a final patch.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Tasks authored by people who know the repository
Authoring every task from scratch requires a different collection process from mining pull requests. Datacurve uses Shipped, a platform where software engineers and machine-learning enthusiasts create challenges and compete with one another. It has versions for different domains; the software engineering version draws on the conventions of Codeforces and GitHub.
The contributors are often maintainers or core contributors to the open-source projects used in their tasks. That familiarity matters twice: they understand the repository’s philosophy well enough to propose a realistic objective, and they know its conventions well enough to judge whether the resulting change resembles a pull request that could actually be merged. Original authorship supplies new evaluation material; repository expertise helps keep that material grounded in real engineering work.
Suggest correction
Short prompts, long solutions
Shi reports that SWE-Bench Pro prompts average more than 4,500 characters, while DeepSWE prompts are roughly half as long. The goal is to describe work the way one might assign it to another engineer: state the objective, then require the engineer to explore the repository and work out the necessary changes. A detailed checklist of functions and signatures can remove much of the decomposition that the benchmark is supposed to test.
Prompt length is not task depth. DeepSWE aims for terse, high-level requests while retaining substantial implementation work. Shi’s scale comparisons make the distinction concrete:
| Measure | Reported DeepSWE result |
|---|---|
| Average reference-solution size | Roughly five times SWE-Bench Pro’s lines of code |
| Files touched by an agent solution | Seven on average |
| Output tokens per rollout | Roughly twice SWE-Bench Pro’s |
The solution-size comparison concerns reference solutions, not every agent-generated patch. The file and token figures describe the longer working trajectories Shi reports; the talk does not specify the full model and budget conditions for the token comparison.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Verify behavior, not the author’s implementation
A verifier can reject a correct solution for the wrong reason. If a merged pull request introduced a particular helper, tests copied from that pull request may require the same helper name or module location. An agent that implements the requested behavior through another internal structure then fails despite meeting the objective. DeepSWE instead emphasizes observable behavior, removing dependencies on private names and implementation choices that are not part of the task’s contract.
The earlier hook example illustrates the distinction. For an illustrative TypeScript API named runHook, behavioral tests can check both requested calling modes without dictating an internal helper:
typescript
import assert from "node:assert/strict";
import test from "node:test";
import { runHook } from "./hooks.js";
test("runs a synchronous hook", async () => {
const calls: string[] = [];
await runHook(() => { calls.push("sync"); });
assert.deepEqual(calls, ["sync"]);
});
test("waits for an asynchronous hook", async () => {
const calls: string[] = [];
await runHook(async () => {
await new Promise<void>((resolve) => setTimeout(resolve, 10));
calls.push("async");
});
assert.deepEqual(calls, ["async"]);
});
Here the contract is that runHook invokes the supplied callback and does not complete before an asynchronous callback finishes. The tests constrain that behavior while leaving the implementation free to use different private functions. They illustrate the verifier principle, rather than reproduce a benchmark task.
Shi reports that the team’s review, using human experts and LLM-as-a-judge, found fewer false negatives and false positives than in SWE-Bench Pro. The later technical report frames its measured audit result as verifier–judge disagreement, which is not the same as a ground-truth error rate. The practical aim is to reward any correct implementation while still rejecting patches that fail the objective.
Repository selection adds a separate representativeness check. DeepSWE covers 91 repositories, selected for more than 500 GitHub stars, active contributions, and expert validation of real-world use. The experts also assess whether a repository can support realistic software engineering tasks; popularity alone does not establish that.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What the harness and task mix leave open
DeepSWE uses mini-SWE-agent as a common harness to emphasize differences between models. Shi reports comparable performance in checks against native harnesses, but the supporting comparison was a small pilot on ten SWE-Bench Pro tasks. It does not establish that harness choice is interchangeable across tasks. A separate evaluation could measure how native and third-party harnesses affect efficiency and output while holding the model fixed.
The task mix also reflects a deliberate emphasis. Long-horizon work leaves bug localization and refactoring underrepresented, even though both occupy substantial engineering time. Maintaining few tasks per repository helps diversify the corpus, but Shi still wants more repositories, more tasks engineers find useful, and more specialized tests of model capability. Broad repository coverage and broad coverage of engineering activities are different goals.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate the evaluator, then loosen the specification
DeepSWE v1.1, already released at the time of the talk, strengthens the environment around the task:
- Runtime isolation: the verifier runs separately from the agent, protecting the evaluation environment from changes made while solving the task.
- Standardized reports: test results use a more consistent format.
- Git-history trimming: refs and commits are removed except for the base commit the agent starts from, closing the route to recovering a later golden patch.
These changes address cheating and reward hacking in the evaluation machinery. They accompany, rather than replace, the ongoing work to diversify tasks.
The next verification problem is how much a prompt must specify. DeepSWE’s current prompts still sometimes hint at a methodology so that an agent can make meaningful progress. Shi proposes hybrid verification, potentially including LLM-as-a-judge, as a way to accept a wider range of valid solutions and make prompts even more focused on the objective. That remains a future direction, distinct from the released runtime safeguards.
This connects the benchmark back to Datacurve’s original data-quality question: if evaluation can recognize successful work without prescribing the implementation, it can test more of the engineering judgment that training is meant to improve. Shi closes with plans for additional benchmarks in high-value domains and recruitment of researchers and engineers to build those benchmarks and the associated training-data pipelines.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Explore benchmark results, task examples, cost, output tokens, and agent steps.
Original account of SWE-Bench Pro's task collection and public, held-out, and commercial splits.
Explains benchmark composition, scoring, efficiency metrics, and the index's version history.
A minimal software engineering agent with bash-based actions and setup instructions. The current repository documents v2.
Further reading
Benchmark tasks, verifier structure, reference solutions, and instructions for running evaluations with Pier.
Detailed benchmark construction, evaluation protocol, verifier audit, and limitations.
Read the complete timestamped transcript
- 0:00
[upbeat music] Hey, everyone. Can you guys hear me okay? This is good?
- 0:15
Um, yeah, my name is James. I'm one of the founding engineers at Datacurve. Unfortunately, Serena's been out with a fever for the past couple of days. Um, she was supposed to be here giving this talk, um, so I'm just filling in, uh, in her place.
- 0:28
But I've been at, uh, Datacurve working on the research and engineering side of things, um, as well as DeepSuite, which is our, uh, frontier long-horizon coding benchmark, which you guys may be familiar.
- 0:40
Um, I'll just be going over, you know, some of the most important findings about DeepSuite, um, a brief overview of what it is for those of you who don't-- who, who may not know.
- 0:49
Um, and then going deeper into our methodology and exactly how we came about this, uh, uh, frontier coding benchmark. So, um, DeepSuite is a long-horizon software engineering, software engineering benchmark comprised of one hundred and thirteen original software engineering tasks.
- 1:07
So this means, unlike something like SWE-bench Pro, we didn't scrape this from, um, existing PRs that have been closed. Um, there's a variety of benefits for this, uh, namely one of them is to resist against contamination and agents being able to cheat, uh, through the course of their rollouts.
- 1:22
Um, SWE-bench has-- or SWE-bench Pro, uh, pulls thousands of tasks from only forty repositories. The median task per repository for us is one. So you can see across over a hundred tasks, we pull from nearly a hundred repositories, and the language spans across, uh, TypeScript, JavaScript, Python, Rust, and Go, and we have plans to add, uh, more
- 1:44
languages later on. Um, since its, uh, release, we've received, uh, very positive reception. It's replaced SWE-bench Pro in the artificial analysis, uh, coding, um, agent index, as well as being cited by, uh, numerous frontier model labs and us helping with them in, uh, tracking their models on our benchmark as well.
- 2:05
So we've been really, really appreciative of that.
- 2:08
Uh, a bit of context about us. Datacurve works on building, uh, training data for high-ceiling domains, including coding as well as coding-adjacent fields. Uh, we also are trying to answer the very, uh, elusive question of what exactly makes, uh, good data, what is data quality, and how can we demonstrate that tr-- our training data, in fact, moves
- 2:31
the needle. So DeepSuite is one in a long line of initiatives that we have towards answering this question.
- 2:40
So why did we create DeepSuite? Well, it's very clear that the existing benchmarks are not hitting the mark. Um, with benches like SWE-bench Pro, uh, top models are clustering at the top.
- 2:51
It's very hard to differentiate between, uh, which one is good because they all have overlapping confidence intervals. Uh, contamination is also rampant because, again, all of these tasks are mined from public PRs, so all the solution tests, even the discussion around the PRs, those are all available out in the wild for these agents to access.
- 3:10
The verifiers are also very, very brittle because we're anchoring them to a specific implementation, often derived from the PR that was merged in. And oftentimes, you also have tests that check for private helpers and functions, uh, created by the task author, which is, uh, very opinionated, right?
- 3:27
And it's not something that models should have to adhere to. Um, and finally, uh, leakage. So one thing about SWE-bench Pro is, uh, for very, uh, insightful models such as Claude, they're able to directly run Git log and then go through the commit hashes and cherry-pick the ones out that contain the golden patches, which again, very, very
- 3:45
serious issue. So this is DeepSuite. This is the updated leaderboard as of July first. You can see, um, I was mentioning before, uh, the problem of differentiating, uh, but you can see on DeepSuite here, there is a very clear difference.
- 3:59
There is a very clear performance gap between the top-performing models versus, you know, at tenth place you have Gemini 3.1 Pro. Um, also within the Claude and the GPT models as well, we're able to see some, uh, deviance.
- 4:12
And, um, yeah, if you go on deepsuite.datacurve.ai, you'll also be able to see, um, the token efficiency, uh, costs, um, token usage, context window, peak context, all of that stuff on the DeepSuite site as well.
- 4:26
But yeah, as of July first, uh, Fable 5 is retaining the top spot on our leaderboard.
- 4:33
So, um, the ranking information is available online. I wanted to talk about some of the qualitative insights into how these different models are performing, which I think is the most interesting part.
- 4:46
Um, starting with the first one is we find Claude is generally a very, very, um, thorough and exhaustive model. It will, um, try to explore everything, including, uh, go through all of the Git logs.
- 4:57
Uh, so one interesting insight was seeing that it becomes quite forgetful when it comes to multi-part prompts. So when you tell it, uh, within the scope of a task, let's say, to in-- support both synchronous and async versions of calling a hook, it will go ahead and implement the synchronous part, but it may drop the asynchronous part.
- 5:19
We observed this in roughly two out of three Claude rollouts across all of the, uh, trials, all of the rollouts that we ran. So this was definitely quite interesting because from my experiences and developers I've talked to as well, Claude is generally very, very thorou-thorough and able to get at the developer's intent quite well.
- 5:38
Um, another thing about Claude is it pays very close attention to its environment, so it will often run-- Uh, this is taken from the trials, uh, we ran ourselves independently and also from examining SWE-bench Pro.
- 5:50
It will attempt to run Git log and recover the golden patch from the Git history. We found that for Opus 4.6 and 4.7, it did this twenty-five percent and eighteen percent of the time respectively, compared to all the Gemini models, uh, averaging at roughly one percent of the time, and we found zero instances of this for the
- 6:11
GPT models. So, um, thankfully within DeepSWE 1.1, we safeguarded further against models being able to cheat by pulling from the Git history. Um, but this was something, yeah, we observed quite frequently for Claude within, um, the SWE-bench Pro rollouts.
- 6:29
Um, third finding is that GPT is very good at implementing exactly what it is asked. Across our failure mode analysis, we found that it was the least likely model to miss requirements.
- 6:42
Um, GPT-5.4 was the second-best model at this ranking, only behind GPT-5.5. It always, uh, learns to read the prompts and the re-caus- repository contract very literally, and producing a patch that honors the existing conventions, um, signatures within the repository, which is very helpful.
- 7:00
And we found that these traits converge across all rollouts. So these were not just, uh, lucky attempts, but on average, like, this was the, uh, favorable, um, behavior exhibited by GPT.
- 7:13
Um, and finally, we found that on average, stronger men-- stronger models have a great tendency to want to test their own work, um, but with a caveat. In SWE-bench Pro's template, they explicitly tell the model that the tests are handled, and therefore they do not need to, uh, write, uh, any new tests of their own.
- 7:34
With that single line in the prompt, it will, uh, prevent the models from e-even, uh, 5.5 and, uh, Opus 4.8 from attempting to verify its own work through the course of the rollout.
- 7:45
Um, in DeepSWE, we do not have anything that says to write or to not to write tests, and so we observe this, uh, divergence between, uh, the percentage of the time where these models are actually engaging in writing tests.
- 7:58
So this is quite an important behavior as it can, uh, provide-- The models are trying to obtain their own ways to verify and validate their work through the course of a rollout.
- 8:08
Uh, we find on average that stronger models like 5.4, 4.7 exhibit this the majority of the time, whereas, uh, models like 3 Flash and 3.1 Pro are far less frequently, um, willing to test their own work.
- 8:22
Um, yeah, so, uh, takeaways from the findings, I think, is, is very interesting how, um, yeah, str-stronger models on average exhibit, um, or converge on these behaviors. Um, so moving on to the tasks, right?
- 8:37
The, the methodology behind DeepSWE. Um, we made a decision to want to have every task authored, uh, from scratch, uh, rather than being mined. Uh, aside from the, uh, issues with contamination that we mentioned previously, um, this also plays into, uh, one of our core strengths, which is that we offer a bespoke platform where we have, um,
- 8:59
software engineers, machine learning enthusiasts come on and create these challenges and compete against one another. Um, this platform is like Shipped and-- Or, or sorry, this platform is called Shipped, and we have a version of this platform for every single domain that we're interested in.
- 9:14
For example, for software engineering, it takes a lot after a Codeforces or GitHub, um, and we're really looking for enthusiasts. So these are oftentimes open-source engineers who are core contributors or maintainers of the projects that they're actively making tasks for.
- 9:31
So by creating these tasks from scratch, we know that the outputs are intrinsically aligned with our objective of providing a fair and comprehensive test to models. We also know that these people has-- have very thorough understandings of the repository's philosophy and the existing conventions, so they can make tasks that are both realistic in terms of the prompt,
- 9:52
but also realistic in the sense that, uh, this is an actual PR that you might see getting merged into the repositories.
- 10:00
Um, another, uh, very important design decision is we try as much as possible to make our prompts read like real tasks. On average, the average prompt, uh, characters within SWE-bench Pro is over four thousand five hundred characters, whereas for us it's, uh, roughly half of that.
- 10:17
And it's-- This is important because when you're prompting, say, a junior engineer, or you're prompting a model to solve a very, uh, high-ceiling ambiguous task, you're not gonna be coming in there with a to-do list, uh, telling it to, "Oh, first do this and then do this, and then write this function signature in exactly, uh, this way
- 10:37
that I've prescribed onto you." Oftentimes, you're going to give it the high-level objective, get it to explore, and get it to reason about to, uh, the list of to-dos and ultimately to the solution on its own.
- 10:48
Um, so this was not the case in SWE-bench Pro. It's very o-overly verbose and trying to prescribe a certain solution method onto agents. Um, as much as we could, we try and make DeepSWE prompts as terse and as, uh, high level as possible, um, mirroring what you might see in the real world if you were to prompt,
- 11:07
say, another, another engineer or one of your agents to go and solve an engineering task.
- 11:14
Um, so even though our prompts are short, we still are able to maintain the long-horizon nature of these tasks. Even with our prompts, again, being roughly half the size of SWE-bench Pro's, we find that the average size of our solution is five times the lines of code, um, compared to SWE-bench Pro's.
- 11:31
Um, we also verified that, uh, there are on average seven files being touched in the agent's solution. Um, and across the course of a rollout, we have two times more output tokens being emitted.
- 11:43
Um, and, um, finally, we have, uh, we-- Verifier design is, of course, one of the most, um, important and tricky parts of building good environments. In SWE-bench Pro, we have these verifiers that are testing, again, for specific implementations.
- 12:00
It will fail the model if it, uh, produces a function that may address the objective but is not named or is not defined within a specific module, or if there's the absence of specific helpers or other private functions.
- 12:14
Um, because again, these are derived from the solutions that were merged in the actual PR. Um, so for us, we want to, uh, emphasize on the observable behavior as much as possible.
- 12:26
We want to ensure that any correct implementation, uh, anything that correctly solves the problem is rewarded, and this will prevent against, uh, false negatives. We also make sure that, um, there is the absence of these PR-derived tests that rely on naming, relying on specific implementations.
- 12:43
Um, and so this will prevent again, again-- uh, again, uh, towards false negatives as well. We observed through a combination of these, um, you know, considerations, we're able to drastically reduce the false negative as well as the false positive rates when we analyzed, um, our rollouts compared to SWE-bench Pro's, uh, using both human experts as well as
- 13:05
LLM-as-a-judge. And yes, the coverage for us spans across these 91 repositories. Um, our criteria for these repositories was, uh, ones that had more than 500 stars on GitHub. Uh, they are actively being contributed towards and for, for our pool of, uh, uh, subject matter experts to validate that these are, you know, in fact repositories that
- 13:30
are actively used in, in, in the r- in the real world and are representative and can field, uh, real-world, uh, and realistic software engineering tasks.
- 13:41
Um, but with all that said, um, there's still a lot of work to be done for DeepSWE and for benchmarks in general. One of the things that we outlined in our blog is our, uh, choice to use MiniSWE-Agent, which is an agent-agnostic harness.
- 13:55
The reason why here is we really wanna be focusing on the model's base performance, and so we use MiniSWE-Agent also, uh, ran rollouts to test that the, uh, performance is comparable both using MiniSWE and against each model's native harness.
- 14:11
But I think there's a lot of work to be done in the future for benchmarks that focus solely or more so on harnesses and comparing the effects that these harnesses, whether it's native or third-party ones like MiniSWE, um, towards the efficiency and the output of these models.
- 14:28
Another thing we wanna improve on is task mix. So given that we are targeting long-horizon tasks, naturally this meant that there's less, um, less, uh, emphasis on bug localization and refactoring.
- 14:41
These are obviously very representative of real work that software engineers are doing, uh, underrepresented in our current taxonomy, uh, for DeepSWE. Um, and finally, repository pool. Um, we put an emphasis on trying to field as many diverse repositories as possible, keeping the median task per repository to a very low count.
- 15:01
But, uh, further work here just to pull in more t-- uh, more repos, more tasks that software engineers find interesting and find them to be good and maybe also, um, more niche tas-tests of models' performance would also be a great addition here.
- 15:18
So we've already, uh, released DeepSWE v1.1. So in here, we've taken some additional measures to guard against cheating, uh, reward hacking, um, by ensuring, you know, the verifier runtime is fully separate now from the agent runtime.
- 15:32
Um, also making sure the test reports are in a more standardized format, and also making sure that we've trimmed all of the Git refs and the commits besides the, uh, base commit that our agents are working on.
- 15:46
So all of this in service of just making the environments more robust and more cheating-proof. But, uh, as I mentioned, looking ahead, we wanna support an even greater diversity of, like, task, uh, tasks, uh, corpus.
- 15:58
We also wanna look into hybrid verification because if we're able to use LLM-as-a-judge or other, um, methodologies, it's possible for us to make our prompts even more terse and even more, um, even more, uh, high level and focused on the objective rather than prescribing anything onto the agent.
- 16:18
Um, there is of course like a certain degree that we have to in our current prompts like, um, hint the agents, steering them towards a current methodology just because otherwise they, they may not be well-positioned at all to make meaningful progress towards the task.
- 16:32
Um, but something like LLM-as-a-judges and hybrid verifiers would potentially help us towards that. And beyond DeepSWE, we're also working on new benchmarks that are in the works. These are again focused on the high-value domains that, uh, Datacurve reprioritize, uh, as being the domains where we wanna be, uh, most meaningfully advancing model capabilities.
- 16:55
Um, but with that said, we're actively hiring, uh, both researchers, engineers, helping us with these new benchmarks, new training data pipelines in service of advancing these capabilities. So, um, definitely reach out at datacurve.ai/careers.
- 17:09
Um, and yeah, if you're interested about any of this research benchmark or any of our works, uh, come find me after. Thank you very much. [audience applauding] [outro jingle]