Agents reported thousands of bugs, how many were real? - Ian Butler and Nick Gregory

AI agents for software engineering have been exploding in popularity over the last year, and we wanted to find out just how good are they at finding and fixing bugs.

Can you even rely on them for maintenance? Well, that's what we're gonna tell you here today at AI Engineers World Fair. Uh, hey guys, we're Bismuth, and we've been working on software agents for the last year and some change.

My name is Ian, and I'm the CEO of Bismuth. So my background was in data engineering, machine learning, and search. Previously, I was at Zillow as a senior eng on the AV testing platform, uh, and this is my second time working deeply on dev tooling at a startup.

Um, I started a technical documentation search in 2019, along with Nick here, building an index of hundreds of millions of technical documents for inline IDE use, along with AI summarization for internal knowledge bases.

Hey, so I'm Nick, CTO of Bismuth. Um, I was primarily in the software security space before starting Bismuth with Ian. Um, I was at Google just prior on an internal tools team building endpoint security software, and before that, I was a research scientist focused on detecting software exploitation.

Um, there's a lot of tools and techniques from that space that transfer to building intelligent agentic code tools, it turns out.

Before we get started, just wanna shout out Baseten for helping provide credits and compute for running our benchmark across both DeepSeek R1 and Llama 4 Maverick. Thanks, guys. We really appreciate it.

Thank you. So to dive right in, we wanted to share with you all the benchmark we've been building for a few months now to explore just how good software agents are at coding tasks outside of the normal feature development.

There's a handful of benchmarks already to measure how effective LLMs are for writing code, HumanEval, SWE-bench, aider's Polyglot benchmark, uh, and LiveCodeBench just to name a few. But of course, that's only one part of the software development life cycle and what developers do.

So what about the rest of the SDLC? Well, uh, initial scoping and planning is kind of an entirely different task, right? It requires broader business context, knowledge of existing systems and designs, exploring potential existing solutions, and so on.

Um, it's basically a completely different task from development. Um, next we have the feature dev and testing, which are already pretty well covered by the existing benchmarks. Um, and then there's the code review process, which is pretty much un-benchmarked by existing work, even though we're seeing more and more LLM-based tools in the space.

Uh, and this is actually the first part of our benchmark. Deployment is a pretty separate task requiring not just writing configuration but also setting up monitoring, integrating with other existing systems, et cetera.

It's also pretty distinct. Um, uh, last and las- last and not least though, we have software maintenance tasks, so think things like bug fixes, dependency upgrades, migrations, et cetera.

Um, this is another part of the SDLC that at its core is still generally writing code but is in a distinctly different way than feature development.

So the ability to reason through a code base deeply to find bugs is directly transferable to being able to produce features. You can think about how, like, when you're dividing a feature, you're gonna learn the architecture of the code base.

You're gonna learn, you know, how everything connects, the data flow, and stuff like that. So that's the same for finding bugs. In both cases, you need an understanding of the system architecture and its connectedness.

Um, in the case of finding bugs, you even often need to understand the system more deeply than when you originally wrote the feature in the first place. Um, to that end, maintenance requires the ability to first deeply reason through a system and identify if bugs are potentially present.

We found agents struggle with holistic evaluation of files and systems, only finding subsets of bugs per run. Thinking models alleviate this somewhat, but reasoning appears to be somewhat narrow, only exploring a limited number of potential avenues at a single time.

This is translated directly to LLMs missing bugs that human developers would pick up almost immediately and confirming bugs that human developers would discard almost immediately. While there is value here, the ability is clearly still in its infancy.

When it comes to patching of these bugs, because of the relative simplicity that we have seen from them, they usually get it, uh, without much of an effort. But again, because the bugs themselves are not particularly complex, this isn't really saying much.

Yeah. So there are some existing bug detection benchmarks from the software security space, but there are some big limitations that don't really make them suitable for evaluating these new agentic AI systems, since they were built to benchmark kind of classic static analysis or program repair tools.

Uh, they focus on relatively simplistic bugs in common patterns, things like null pointer dereferences, buffer overflows, maybe SQL injection in a web app, which could all be found statically.

Um, a lot of the benchmarks are also really limited in the languages that they pull from, so many of them are only Java, for instance, because that's where the vast majority of enterprise complicated software was and honestly still is written.

Um, there's also a bias towards security issues in the existing benchmarks, partly as a result of that being where the classic static analysis tools are focused. Um, but bugs appear in many more ways than just security defects, right?

Copy-paste bugs are a really simple example where they don't often result in an auth bypass, for example, but it breaks the software to end users all the same, right?

So how does our benchmark, SM100, compare?

So we painstakingly gathered 100 bugs that we triaged, validated, and classified in over 84 public repositories. These are bugs that have all been remediated already but are represented in the wild and work in open source by various developers across all levels.

Our goal when assembling these bugs was to provide a range of issue types from obvious low specific domain knowledge all the way to senior staff level engineering knowledge with significant depth of understanding for a given project.

We also wanted to make sure we provide a multi-language benchmark, as LLMs have notably different performance across them. Here we focused on Python, TypeScript, JavaScript, and Go, first because Python and TypeScript, uh, slash JavaScript are languages that are the most popular, and LLMs are supposedly better at performing on them.

Then we chose Go as kind of a control to balance performance on a low-level systems engineering language.

So looking ahead, here's an example of what we mean by an objective bug. Um, this is anything, an explicit security issue or logical issue that could cause data loss or system crashes.

The reason we went for these types of bugs is it removes anything, uh, that is ambiguous, um, or harmless in the context of the system, uh, or performs correctly if you look one level up and the thing that calls it kind of bounds the problem anyway.

Um, an example of this, uh, is a function that doesn't check bounds, uh, but the code is called, uh, but the code that it's called from ensures the input never goes over those bounds anyway.

So in isolation, that's a bug, but in the context of the system, there's, there's no failure point. Um, we explicitly did not include feature requests, optimization, style formatting, or design decisions.

Uh, this is also to reduce ambiguity. Uh, frankly, humans still debate these to today. There's no actual objective kind of discourse around them. It's whatever your, like, uh, cadre decides they want to use.

It also helps make it reproducible across evaluations, and as alluded to earlier, uh, we annotated each bug with metadata such as the severity and context, where it was defined and called, uh, how much system-specific domain knowledge a human would require to find the bug, how difficult even with said knowledge it would be to find it, so even

if you're an expert in the system, how long it would take you to navigate and get that bug, and finally, the implication of the bug itself. Was it a data loss?

Was it a crash? Is it a security exploit? Et cetera.

These classifications allow us to understand what level of bugs these AI agents are capable of finding regularly. We note that every once in a while they surprise us, such in that recent blog post where o3 was able to find, um, a zero-day exploit.

Uh, but notably there it took about 100 runs to actually get it over the same context each time. Um, so while that's still useful to know they're possible, it's more useful to know in everyday usage, how are they capable, um, on just everyday tasks?

So for each system that we benchmarked, there's four numbers to get out of the benchmark. The first is just can the system discover the bugs without any prior knowledge?

Uh, we call this the needle in the haystack result, and we'll talk a bit more about the specifics of our methodology for this in just a second. Um, of course, though, when the agents produce a list of bugs, there's a lot more than just the one that we're looking for, so we also manually measure the false positive

rate of the bugs that come out to get an overall sense of how effective it is, uh, broadly. Um, we also look at whether the system can find the bug at the time of introduction.

So given the pull request or commit that introduces the bug, can the agent point it out at that point? Um, of course, here the agent doesn't have to go searching to find issues and has a lot more kind of immediate context around it, so with a bit more of an optimistic starting point, how good can they be?

Um, and finally, we still do wanna see how the agents, uh, do with suggesting remediations for the things that it identifies. Uh, and so for each bug that it discovers, we ask it to fix the bug, uh, and see if the result does indeed fix the bug without breaking the rest of the code base.

So jumping back, uh, to needle in the haystack now, we don't wanna just tell agents, you know, go find bugs somewhere in the repository because frankly, most of the things that we're measuring are big enough, we're exploring all of the code to find the single bug that we're interested in would take way too long.

Uh, and we also don't though want to hint at what the actual bug is in the evaluation. We don't wanna bias them in any way. So how do we solve this?

Uh, well, we've broken up the repositories into subsystems containing files that are likely all interrelated, so think, you know, maybe part of a front end or some specific, uh, API, uh, point.

We then feed the list of files in that subsystem, uh, or sorry, we then filter the subsystems that were c- that contain the files that were modified kind of in the golden PR commit and just feed the files in those subsystems to the LLM.

So it's getting a reduced list of files in the entire repository. This way we're not biasing it in any way, but we're still scoping it down so it doesn't have to look through everything, uh, and we can still get an idea of what this works or how well this works in like a non-benchmark capacity.

It's just looking at one part of the code but that in its entirety.

Okay, so what does it take to actually get a system up to a level of beating some of the biggest companies and labs in the world? So basic implementations are trivial.

Give it a shell tool, surreplace, think, report bug, you know, a finish tool, put it in a loop and press play. So you do have an agent here. It's running, it's looking at code files, um, and you might even find some bugs.

We actually found five or six in the basic loop here, uh, but they also had a 97% false positive rate. So almost every basic agent that's just equipped with these tools from like the Anthropic model card releases or OpenAI methodology releases, um, are not up to the task of really finding bugs and triaging systems.

So it's actually really hard to build a good agent for identifying bugs and working on code. It's a combination of model, system, prompting, information, the way you feed that information to a model, and navigation strategy.

Um, if you look at our numbers for performance comparisons, you can see right off the bat that Bismuth leads the pack for needle in a haystack. Um, we found ten of our needles, uh, the next leading solution finding seven.

So we do note off the bat there's a lot of room to grow here, but if anything, we're excited that we found an unsaturated benchmark to work on, uh, as we build our product and as other labs build theirs.

I will say we are starting to see some exciting things on true positive rate and detection though, uh, with Claude Code starting out at sixteen percent, us at twenty-five percent in second place, and Codex at forty-five percent.

You'll also note that for these three solutions I just mentioned, we find significantly less, like, random nonsense compared to the rest of the pack. Like, we just have a tighter scoping.

Uh, Devin, Cursor Agent, uh, and Cosine all found between nine hundred and twelve hundred items or thirteen hundred items for Cursor, uh, with between a three and ten percent true positive rate.

So there's a lot of, um- headway for those agents to go. On PR review, uh, Codex was quite strong, uh, with 20% ... uh, 27% of the needle in haystack found, then Devin at [REDACTED:age], and then us at [REDACTED:age].

So one thing to note there is that those do not include any false positives that had been found, uh, during those PR reviews. This is just did it find the needle in the haystack?

And even there, you'll note that the best model only got 27, so about a third. Uh, there's a long way to go on both PR review and bug detection.

I wanna note that given the models here, uh, Bismuth itself is agnostic, uh, but we typically run on top of Anthrodi- uh, Anthropic models, and they've been easier to serve for our customers from Vertex.

Uh, in this instance, our solution was able to beat, uh, their own solution, Claude Code, in multiple categories, uh, while improving on their base model.

So let's move on to looking at what we call basic agents. Uh, these are agents that are the simple loop we described earlier, and we were able to actually look at a couple open source models here as well.

And we'll note just off the bat, um, open source has a long way to come in this space. Um, R1 only had a 1% true positive rate, um, over what?

Like, hundreds?

Uh, yeah.

Yeah, over hundreds of bugs. Um, same with Llama Maverick at a 2%. Um, R1 did manage to find, um, a needle in a haystack, um, but both of these were lapped by Sonnet 4 and o3 in a loop, um, where Sonnet 4 found six, uh, and o3 found two, with a 3% and 6%, uh, false ...

uh, true positive rate respectively. The hard truth here is that the highest popular agent score outside of us, who are just launching now, uh, scored 7% on SM100. Um, that means the most used agents in the world are the worst at finding and fixing complex bugs.

Uh, this is a problem for the industry that we're hoping to bring to light, and we're hoping to see that these teams will begin to work on this to increase their score in what represents the back 90% of work for software engineering.

It's also important to note that three out of six agents got 10% or less true positives out of a massive number of reports. Uh, on one issue, an agent gave us an astounding 70 reports, and nobody's going to actually sift through that many, right?

Like, no engineer's gonna look through 70 bugs and be like, "Oh, yeah, that one for sure is the right one." So there's a long way to go on tightening how much information these agents are reporting and the accuracy of that information itself.

I think that, you know, in context, that even some simple bugs are missed. Um, only two agents, Bismuth and Codex, found this state issue, and the consequence of the state issue is pretty simple, right?

When you submit a form, um, usually they clear. In this particular case, uh, isDirty was never set to false, uh, so it would never clear the form, and it would say you still had things filled out.

You know, that's not the biggest bug in the world, but that has real consequences for users and user experience, and that's the type of things a human developer would immediately catch.

So we think there is a large way to go.

So after examining, uh, all of the performance across all these different agents, there was a notable commonality between them. These agents are very narrow in their thinking, even when using thinking models.

This is actually a problem. It's more of a problem on non-thinking models, but thinking models themselves are very narrow in the types of things they're evaluating, and we find that even with that narrowness, they don't go deep enough.

So from our perspective, broader thinking chains and deeper thinking along selected chains is going to be required to actually find bugs in these files. It's also interesting to note that on a per run basis, the total number of bugs remains roughly consistent, but the bugs themselves change.

So that means from any one particular run, an LLM is not looking holistically at a file and inventorying everything that's happening. Um, we suspect there's different biases occurring, uh, in that and that the models themselves, you know, through whatever context had been provided, are looking at a file one way and not another way.

Um, there's methodologies we thought of to solve that, but right now it seems like that is a pervasive problem across every agent in the industry, and no one has quite solved that.

So there's a broader problem here, right? Despite 60 to 70%, 80% in some cases, scores on SWE-bench, agents still struggle with SM100. What does that mean? Existing agents are able to create software upfront, but to manage and fix software after it's been deployed will be a major struggle as far as we see it.

And this is a hard problem that teams are working on across the board, us included, but it demands targeted search, better program comprehension, cross-file reasoning, bug pattern recognition that we just don't see in deep capacity from many of these solutions.

Now, we're hopeful here because Codex, for instance, and us are starting to show tighter, narrow ... uh, tighter, more narrow bands of the bugs that it's solving. We do get some complex issues.

We are one of the two agents that found that bug I mentioned earlier. So as we progress here, we hope the industry evolves and focuses more on this problem.

So to wrap things up, we know that the most frequently used agents today have a high risk of introducing bugs, though as we just said, a few of the newer agents as part of this new generation, including us, are starting to show an increased ability to reason through code and more effectively use their context to evaluate concerns.

This ability is nascent, but so far what we've seen has been encouraging, uh, and with this benchmark, we can now clearly show as things progress. Um, with effort and some different techniques, you can do better, and improvements here will have benefits across the entire industry.

We have many things that we wanna try, and I'm sure other labs do, uh, and we're really excited to be on the forefront of this new generation of agents.

So thanks for watching. Uh, you can dive into numbers and get more details on the benchmark, uh, at the website, uh, and be sure to check us out at bismuth.sh.

Thank you.

Thanks