AI Engineer World's Fair 2025
AI-powered entomology: Lessons from millions of AI code reviews
Read the talk
AI code review needs two tests: can it catch the issue, and is the comment welcome?
Diamond’s development shows why finding bugs is only the beginning: an AI reviewer must choose useful comments, then measure whether developers act on them.
From a talk by Tomas Reimers
Before you start: Familiarity with pull requests and code review is sufficient; no machine learning background is required.
Can AI find the bugs AI helps create?
As AI writes more code, can it also find the bugs that come with it? That question led Graphite co-founder Tomas Reimers and his team to build Diamond, an AI reviewer that connects to GitHub. The project began about a year before the talk, as the team saw both AI-generated code and bugs increasing. Reimers calls the work AI-powered entomology: the study of bugs.
The initial experiment was straightforward: give Claude a pull request and ask it to find bugs. The results included a concrete failure in Graphite’s own codebase: certain paths returned a database ORM class without instantiating it, which Reimers says would crash the server. Another example, shared on Twitter, involved border-radius arithmetic that he described as dividing by a negative number and crashing the frontend. The latter example establishes the kind of issue the reviewer flagged, but the verbal description alone does not explain the crash—division by a negative number is not inherently an error.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A correct comment can still be unwelcome
Finding real bugs did not make the reviewer consistently useful. Alongside the successful catches came suggestions to change code to do what it already did, assertions that valid CSS behavior was invalid, and requests to restore old behavior simply because the code used to work that way. These comments eroded the confidence earned by the useful ones.
The first explanation was a capability boundary: there are issues an LLM can catch and issues it cannot. Asking a model what review comments someone would leave on a PR encourages it to imitate the whole activity of code review. That includes comments within its capabilities and comments beyond them. Categorizing those capabilities helped, but it left another class of frustrating output: requests to document a class, extract logic into a function, or add tests. These could be technically defensible without being useful interruptions.
The distinction became clearer when developers and designers reviewed historical comments from both humans and the bot. Developers largely agreed about which comments they would accept from an LLM. Designers found some of those judgments puzzling: superficially similar comments received different reactions. The developers were evaluating more than correctness. Advice they might welcome from a colleague could feel pedantic or annoying from an automated reviewer. Model capability and developer receptiveness are separate axes. A useful reviewer needs to satisfy both.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Best-practice advice needs a reason to appear
The opposite problem is advice that models can readily produce but developers do not necessarily want: comment this function, add tests, extract a type, or move this logic into a function. A human reviewer applies a local judgment. Is this logic unusually tricky for this codebase? Is someone likely to misunderstand it? Would extraction actually help? Without that judgment, the bot can keep producing defensible cleanliness suggestions almost anywhere. Technical correctness alone does not decide whether a comment belongs in the review.
More context can change the boundary. Repository code, past history, style guides, and explicit rules can help the reviewer determine which advice fits the project. Reimers suggests that the region of comments a model can produce and developers will welcome expands as that context grows. The practical target is the intersection supported by the context available now.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep checking the boundary as the reviewer changes
An offline taxonomy provides an initial direction, but it cannot establish that a changing reviewer stays useful. Graphite updated its prompts to request only comments within the model’s capabilities that developers wanted to receive. Reimers reports that reception improved anecdotally. The next problem was maintaining that behavior as the system evolved.
Moving to Claude 4, or choosing Opus instead of Sonnet, raises the question again: does the reviewer still operate in the useful part of the chart? Adding context raises a related opportunity: perhaps it can now produce valuable categories of comments that were previously out of reach. Graphite began by inspecting the distribution of comment types the reviewer actually produced. Reimers presents that distribution as specific to Graphite’s observations. It describes the reviewer’s output, but does not by itself measure whether that output succeeds.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use reactions to detect overreach
Graphite added emoji upvotes and downvotes to review comments as a feedback channel for capability failures. A spike in downvotes could signal hallucinations or an attempt to push the reviewer beyond what it could reliably assess. The operational response was to narrow its behavior again. That made reactions useful for detecting trouble, while leaving the second axis—whether developers wanted the comments—harder to measure.
Reimers reports a downvote rate below 4% at the time of the talk. He does not specify the denominator, so the figure should not be read as a false-positive or hallucination rate. More fundamentally, a lack of downvotes does not establish that a comment helped someone improve the code. That requires looking at what happens after the comment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Measure whether the requested change happens
A review comment usually asks someone to change code. Graphite therefore began measuring the percentage of comments followed by the change they described, examining open-source repositories and repositories available through its code review product. This turns usefulness into an observable outcome: did the requested change occur?
Reimers reports that about 50% of human review comments led to changes within the same PR. That supplied a practical target for the bot: reach the level of action associated with human feedback. The observation window matters. A developer may agree with a comment but defer the fix to a follow-up PR; advice may concern future work rather than the current change; and a preference may reasonably remain unimplemented. Healthy review cultures leave room for disagreement, so an unacted-on comment is not automatically a failed review.
Reimers reports that Diamond reached a 52% within-PR action rate as of March. He attributes reaching the human baseline to prompting the reviewer appropriately. These are reported action rates, not bug-detection accuracy; the talk does not provide sample sizes or enough comparison details to establish statistical superiority over human reviewers. The result supports a narrower, useful conclusion: a reviewer constrained to appropriate comments can get developers to make the changes it requests at roughly the observed human rate.
Diamond is the production product Reimers presents for applying these findings. Its underlying approach connects scope to feedback: select comment categories the model can handle and developers welcome, then measure whether those comments produce changes. That is what turns the ability to spot an occasional bug into a code review product people can use.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Graphite's introduction to Diamond, including GitHub integration, codebase context, custom rules, comment filtering, and review insights.
Updates since the talk
Current Graphite Agent documentation explaining review categories and dashboards for accepted issues, acceptance rates, and downvotes.
Configure which feedback Graphite Agent produces or suppresses, then track the effectiveness of individual rules and exclusions.
Read the complete timestamped transcript
- 0:00
[upbeat music] Thank you all so much for coming to this talk.
- 0:16
Um, thank you for being at this conference, generally. My name's Tomas. I'm one of the co-founders of Graphite, and I'm here to talk to you around AI-powered entomology. If you don't know, entomology is the study of bugs.
- 0:26
It's something that we do-- we-- It's very near and dear to our heart and part of what our product does. So Graphite, for those of you that don't know, builds a product called Diamond.
- 0:34
Diamond's an AI-powered code reviewer. You go ahead, you, uh, you connect it to your GitHub, and it goes ahead and finds bugs. The project started about a year ago.
- 0:42
What we started to notice was that the amount of code being written by AI was going up and up and up, but so was the amount of bugs. And after really thinking about it, we thought that, you know, this might actually be part and parcel, and what we need to do is we need to find a way
- 0:55
to better address these bugs in general. Uh, given the technological advances, the first thing we turned to was AI itself, and we started to ask, "Well, maybe AI is creating the bugs, but can it also find the bugs?
- 1:08
Can it help us?" Um, we started to go ahead and do things like ask Claude, "Hey, here's a PR, can you find bugs on this PR?" And we were pretty impressed with the early results.
- 1:18
Here's an example actually pulled from this week from our codebase, where it turns out that in certain instances we'd be returning one of our database ORM classes uninstantiated, which would go ahead and crash our server.
- 1:29
Here's an example that came up on Twitter this week from our bot that found that, um, in certain instances, there would be math being done around border radiuses that would lead to a division by a negative number and would then go ahead and crash the front end.
- 1:42
So to answer the question, it turns out AI can find bugs. That's the end of the talk. I'm kidding. If you've tried this, [laughing] um, you know you've probably had a really, really frustrating experience.
- 1:51
Um, we also went ahead and saw things like, "You should update this code to do what it already does," "CSS doesn't work this way when it does," or my favorite, "You should revert this code to what it used to do because it used to do it." [laughing]
- 2:04
Um, getting those lost us a lot of confidence, but we started to think, well, we're seeing some really good things, and we're seeing some really bad things, a-and maybe there's actually more than one type of bug.
- 2:15
Maybe there's more than one type of thing an LLM can find. And so we started with the most basic division of, well, there's probably stuff that LLMs are good at catching and things that they're not good at catching.
- 2:25
At the end of the day, LLMs ultimately try and mimic the thing that you're asking them to do, and if you ask them, "Hey, what kind of code review comments would be left on this PR?"
- 2:33
It goes ahead and leaves everything, both those that are within its capability and things that are not within its capability. And so we started to categorize those. What we found, though, was even when we categorized those, the LLM would start to leave comments like this:
- 2:46
"You should add a comment describing what this class does," "You should extract this logic out into a function," or, "You should make sure this code has tests." While these are technically correct, to developers they're really frustrating, and I think this was actually one of the most, like, insightful moments for us in building this project, was when we
- 3:03
sat down with our design team and we started to actually go through past bugs, um, both those left by, uh, our bot and by humans in our own codebase.
- 3:13
The developers were all pretty much on the same page of like, "Yep, I'd be okay if an LLM left that. No, I would not be okay if an LLM left that.
- 3:19
Yes, I'd be okay." And our designers were actually kind of baffled by it. They were like, "Well, but, like, that kinda looks like that other comment." And I think that what's happening here in the mind of the developer is if you go ahead and you read a type of comment like this, um, maybe you find it pedantic,
- 3:34
frustrating, annoying when it comes from an LLM, and you're much more welcoming to it when it comes from a human. And so as we started to think more around sort of that classification of bugs, we started to think around actually a second axis here, which was there's stuff LLMs can catch and LLMs can't catch, but there's also
- 3:49
stuff that humans want to receive from an LLM and humans don't want to receive from an LLM.
- 3:55
And so what we went ahead and we, uh, what we went ahead and did was we went ahead, and we actually took ten thousand comments from our own codebase, from open source codebases, uh, open source codebases, and we fed them to various LLMs, and we asked them to car-- uh, categorize them.
- 4:10
And we did that not just once, but we did that quite a few times, and then we went ahead, and we summarized those comments. And what we ended up with was actually this chart, uh, where it says there's actually quite a few different types of bugs that you see left on codebases in the wild.
- 4:24
Ignoring LLMs for a second, just talking around humans, you see things which are bugs. Those are logical inconsistencies that lead the code to behave in a way it doesn't want to behave.
- 4:32
There's also accidentally committed code. This actually shows up more than you would expect. Um, there are performance and security concerns. There's documentation where the code says one thing and does another, and it's not clear which one's right.
- 4:43
Um, there's stylistic changes. Things like, "Hey, um, you should update this comment," or, "In this codebase we follow this other pattern." And then there's a lot of stuff outside of sort of that top right quadrant.
- 4:53
So in the bottom right, where humans want to receive it, but the LLMs don't seem to be able to get there yet, are things like tribal knowledge. Uh, one class of comment that you'll see a lot in PRs is, "Hey, we used to do it this way.
- 5:06
We don't do it this way anymore because of blank." This documentation doesn't exist. It exists in the heads of your senior developers. And that's wonderful, but it's really hard for an AI to be able to mind-read to that.
- 5:18
On the left side, where LLMs definitely can catch it but humans don't want to receive, are those things I showed you earlier, code cleanliness and best practice. Examples of these that we've found are, uh, comment this function, add tests, extract this type out into a different type, extract this logic out into a function.
- 5:34
While this is always correct to say, I think it's really hard to know when to apply to an LLM. I think as a human, you're applying some kind of barometer of, well, in this codebase, this logic is particularly tricky, and I think someone's gonna get tripped up, so we should extract it out, versus, well, in this codebase
- 5:50
it's actually fine. But what-- A bot can pretty much always leave this comment. I'd actually make the argument a human can pretty much always leave this comment and it be technically correct.
- 5:58
The question is whether it's welcome in the codebase. And one thing I'm gonna say sort of like outside of all of this is as you add more-
- 6:06
This area seems to become larger of what people are comfortable with. But for now, given the context that we have, given the code base, the past history, the, uh, your style guide and rules, we are what we have.
- 6:18
We have what we have. And so we end up with this idea of, well, it turns out that these are basically the classes of comments that we think that human-- that LLMs can both create and humans want to receive.
- 6:33
Now, if you've worked with LLMs, you know that these kinds of offline passes and first passes are great for initial categorizations. But the much harder question is: How do you know that you're right continuously, right?
- 6:44
We can-- So a-as the story goes, we went ahead, we basically started to characterize comments that LLMs leave. We updated our prompts to only prompt the LLM to do things that were in its capacity and that humans wanted to receive.
- 6:56
And people anecdotally started to like it a lot more. But as we started to then think around, well, how can we get this LLM to... How do we know that this is going right?
- 7:05
As we think around new LLMs, as we got into Claude 4 or Opus instead of Sonnet, how do we know that we're actually staying in this top right quadrant?
- 7:13
And as we increase the context, how do we know that this, that this isn't growing on us? And actually, maybe there are even more types of comments that we could be leaving that we're not leaving already.
- 7:24
And so first and foremost, we started by just looking at what kinds of comments is the thing currently leaving. Your mileage may vary. For us, this is roughly the proportion we see of comments being left by the LLM right now based just on what we've seen.
- 7:39
But the, the deeper question for us was how do we, how do we measure the success, right? Like, given this quadrant, how do we know that in the-- we're in the top right?
- 7:47
The first one was easy for us. So think around what they can catch and they can't catch. What we started to do was we started to actually add upvotes and downvotes to the product.
- 7:55
So we let you go ahead and emoji react in these comments, and they pretty much tell us when the LLM's hallucinating. When we start to see an upvote-- When we start to see a, when we start to see a downvote spike, we know that, okay, we might be trying to extend this thing beyond its capabilities.
- 8:08
We, we need to tone it down. But the second one was a lot harder. That humans want to receive and humans don't want to receive was something that we weren't really sure how to get at.
- 8:18
And so upvote, downvote, we implemented it. We see about a less than a four percent downvote rate these days. We felt pretty good about that. The second one, as we started to think around it, well, what we realized was, well, what's the point of a comment?
- 8:30
Why do you leave a comment in code review? You leave a comment in code review ultimately so that someone actually updates the code to reflect that. And so our question was, well, can we measure that?
- 8:38
Can we measure what percent of comments actually lead to the change that they describe? And so we started to do that. We started to ask that question of, on open source repos and on the variety of repos that Graphite, which is a code review tool, has access to, can we actually start to measure that number?
- 8:52
And I think one of the most fascinating things we found, um, was that only about fifty percent of human comments lead to changes. And so we started to ask the question of, well, could we get the LLM to, to at least this, right?
- 9:04
Because if we get it to at least this, it's at least leaving, uh, comments on the level of fidelity that humans are. Now, you might be sitting in the audience and being like, "Well, why don't a hundred percent of, uh, comments lead to, lead to action?"
- 9:15
I wanna, I wanna caveat this number. I'm saying lead to action within that PR itself. And so a lot of comments are sometimes fixed forward, where people are like, "Hey, I, I hear you, and I'm gonna fix this in a follow-up."
- 9:26
A lot of comments are also like, "Hey, as a, a heads-up, in the future, if you do this, maybe you can do it this other way," but don't need to be acted on then.
- 9:34
I think there's a, there's a fair-- And some of them are just purely preferential of, "I would do it this way." Someone disagrees. In healthy code review cultures, that space for disagreement exists.
- 9:43
And so we started to measure this, and we started to say, "Could we get the bot here?" Um, over time, we actually have. So as of March, we're at fifty-two percent, which is to say that if you start to actually prompt it correctly, you can get there.
- 9:54
And I think our, our sort of broader thesis is that, um, this measuring, uh, getting, uh, bugs via an LLM does actually work. Um, if you wanna try any of these findings in production, um, Diamond is our product that, uh, offers it.
- 10:10
We have a booth over there. Um, thank you. [audience applauding] [upbeat music]
Build the taxonomy from real reviews
Graphite repeatedly asked various LLMs to categorize 10,000 comments from its own and open-source codebases, then summarized the categorizations. This produced a vocabulary for discussing what reviewers actually say, rather than treating every comment as a bug report.
These categories require different judgments. A documentation mismatch, for example, establishes a disagreement but leaves open whether the implementation or the prose is wrong. A style suggestion may depend on knowing which pattern the repository follows.
Some highly desirable comments require information unavailable to the model. A senior developer might explain that the team previously used an approach but abandoned it for a particular reason. If that history exists only in developers’ heads, the reviewer cannot recover it from the code. This tribal knowledge belongs on the wanted-but-difficult side of the taxonomy.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.