AI-powered entomology: Lessons from millions of AI code reviews

As AI writes more code, can it also find the bugs that come with it? That question led Graphite co-founder Tomas Reimers and his team to build Diamond, an AI reviewer that connects to GitHub. The project began about a year before the talk, as the team saw both AI-generated code and bugs increasing. Reimers calls the work AI-powered entomology: the study of bugs.

The initial experiment was straightforward: give Claude a pull request and ask it to find bugs. The results included a concrete failure in Graphite’s own codebase: certain paths returned a database ORM class without instantiating it, which Reimers says would crash the server. Another example, shared on Twitter, involved border-radius arithmetic that he described as dividing by a negative number and crashing the frontend. The latter example establishes the kind of issue the reviewer flagged, but the verbal description alone does not explain the crash—division by a negative number is not inherently an error.

Finding real bugs did not make the reviewer consistently useful. Alongside the successful catches came suggestions to change code to do what it already did, assertions that valid CSS behavior was invalid, and requests to restore old behavior simply because the code used to work that way. These comments eroded the confidence earned by the useful ones.

The first explanation was a capability boundary: there are issues an LLM can catch and issues it cannot. Asking a model what review comments someone would leave on a PR encourages it to imitate the whole activity of code review. That includes comments within its capabilities and comments beyond them. Categorizing those capabilities helped, but it left another class of frustrating output: requests to document a class, extract logic into a function, or add tests. These could be technically defensible without being useful interruptions.

The distinction became clearer when developers and designers reviewed historical comments from both humans and the bot. Developers largely agreed about which comments they would accept from an LLM. Designers found some of those judgments puzzling: superficially similar comments received different reactions. The developers were evaluating more than correctness. Advice they might welcome from a colleague could feel pedantic or annoying from an automated reviewer. Model capability and developer receptiveness are separate axes. A useful reviewer needs to satisfy both.

Graphite repeatedly asked various LLMs to categorize 10,000 comments from its own and open-source codebases, then summarized the categorizations. This produced a vocabulary for discussing what reviewers actually say, rather than treating every comment as a bug report.

These categories require different judgments. A documentation mismatch, for example, establishes a disagreement but leaves open whether the implementation or the prose is wrong. A style suggestion may depend on knowing which pattern the repository follows.

Some highly desirable comments require information unavailable to the model. A senior developer might explain that the team previously used an approach but abandoned it for a particular reason. If that history exists only in developers’ heads, the reviewer cannot recover it from the code. This tribal knowledge belongs on the wanted-but-difficult side of the taxonomy.

The opposite problem is advice that models can readily produce but developers do not necessarily want: comment this function, add tests, extract a type, or move this logic into a function. A human reviewer applies a local judgment. Is this logic unusually tricky for this codebase? Is someone likely to misunderstand it? Would extraction actually help? Without that judgment, the bot can keep producing defensible cleanliness suggestions almost anywhere. Technical correctness alone does not decide whether a comment belongs in the review.

More context can change the boundary. Repository code, past history, style guides, and explicit rules can help the reviewer determine which advice fits the project. Reimers suggests that the region of comments a model can produce and developers will welcome expands as that context grows. The practical target is the intersection supported by the context available now.

An offline taxonomy provides an initial direction, but it cannot establish that a changing reviewer stays useful. Graphite updated its prompts to request only comments within the model’s capabilities that developers wanted to receive. Reimers reports that reception improved anecdotally. The next problem was maintaining that behavior as the system evolved.

Moving to Claude 4, or choosing Opus instead of Sonnet, raises the question again: does the reviewer still operate in the useful part of the chart? Adding context raises a related opportunity: perhaps it can now produce valuable categories of comments that were previously out of reach. Graphite began by inspecting the distribution of comment types the reviewer actually produced. Reimers presents that distribution as specific to Graphite’s observations. It describes the reviewer’s output, but does not by itself measure whether that output succeeds.

Graphite added emoji upvotes and downvotes to review comments as a feedback channel for capability failures. A spike in downvotes could signal hallucinations or an attempt to push the reviewer beyond what it could reliably assess. The operational response was to narrow its behavior again. That made reactions useful for detecting trouble, while leaving the second axis—whether developers wanted the comments—harder to measure.

Reimers reports a downvote rate below 4% at the time of the talk. He does not specify the denominator, so the figure should not be read as a false-positive or hallucination rate. More fundamentally, a lack of downvotes does not establish that a comment helped someone improve the code. That requires looking at what happens after the comment.

A review comment usually asks someone to change code. Graphite therefore began measuring the percentage of comments followed by the change they described, examining open-source repositories and repositories available through its code review product. This turns usefulness into an observable outcome: did the requested change occur?

Reimers reports that about 50% of human review comments led to changes within the same PR. That supplied a practical target for the bot: reach the level of action associated with human feedback. The observation window matters. A developer may agree with a comment but defer the fix to a follow-up PR; advice may concern future work rather than the current change; and a preference may reasonably remain unimplemented. Healthy review cultures leave room for disagreement, so an unacted-on comment is not automatically a failed review.

Reimers reports that Diamond reached a 52% within-PR action rate as of March. He attributes reaching the human baseline to prompting the reviewer appropriately. These are reported action rates, not bug-detection accuracy; the talk does not provide sample sizes or enough comparison details to establish statistical superiority over human reviewers. The result supports a narrower, useful conclusion: a reviewer constrained to appropriate comments can get developers to make the changes it requests at roughly the observed human rate.

Diamond is the production product Reimers presents for applying these findings. Its underlying approach connects scope to feedback: select comment categories the model can handle and developers welcome, then measure whether those comments produce changes. That is what turns the ability to spot an occasional bug into a code review product people can use.