AI Engineer World's Fair 2025
Don’t get one-shotted: Use AI to test, review, merge, and deploy code — Tomas Reimers, Graphite
Read the talk
When AI writes more code, the outer loop becomes the bottleneck
Getting a feature working is only the first loop. AI-assisted development also demands faster review, testing, merging, and deployment without losing scrutiny.
From a talk by Tomas Reimers
A working feature still has to ship
A developer gets the code working and the feature behaving as intended. Then comes testing, review, merging, and deployment. Faster implementation helps with the first part, but it does not remove the second. Tomas Reimers, co-founder of Graphite, an AI code review company, describes these as software development’s inner loop and outer loop.
| Loop | Work | Immediate objective |
|---|---|---|
| Inner | Code, build, debug | Get the feature working |
| Outer | Test, review, merge, deploy | Move the change through to release |
The distinction matters because producing a change and getting it safely to users are separate workloads. Accelerating one leaves the other to absorb its output.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
More generated code, more code to scrutinize
AI adoption is already changing the inner loop. Reimers points to a GitHub survey showing widespread use among surveyed developers. That evidence concerns enterprise respondents having tried AI tools; it does not establish routine use by nearly every developer worldwide, or that every respondent used them both at work and outside it. The accompanying Copilot statistic also needs its historical scope: GitHub’s February 2023 announcement reported Copilot generating an average of 46% of code for developers using it, not 46% of all code hosted on GitHub.
The forecasts that follow reinforce a directional point: even under Reimers’s more pessimistic framing, AI writes an increasing share of code. Developers consequently produce more changes, and those changes still require review.
Graphite began developing Diamond, its AI code reviewer, about a year before the talk, while its own developers were adopting AI tools. Diamond is the historical product name used here; its review capabilities later moved under Graphite Agent. The concern was not just volume. AI can hallucinate, make mistakes, and introduce security vulnerabilities, so generating code faster does not make scrutiny optional.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The outer loop needs capacity of its own
Tools such as Cursor, Windsurf, Copilot, v0, and Bolt were producing larger volumes of code than Graphite’s developers were accustomed to handling. That increased the work at every subsequent stage: reviewing, testing, merging, and deploying. The outer loop was becoming the bottleneck. Problems once associated with large engineering organizations were beginning to affect smaller companies as well.
The requirements follow the work that now has to scale:
- Pull-request coordination: Prioritize changes, track their progress, and notify people when attention is needed.
- Reviewer assistance: Help reviewers focus and streamline the review process.
- Integration throughput: Optimize CI pipelines and use merge queues to handle the increased volume of changes.
- Deployment tooling: Improve the final stage that gets accepted changes to users.
These are requirements for the wider workflow, not just for an AI reviewer. Deployment is part of that scope, though Reimers does not describe a deployment implementation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Adding AI teammates is only part of the solution
The same technology creating pressure on review could also reduce its manual workload. Reimers’s goal for self-driving code review is to remove painful, repetitive parts of the process, leaving developers more attention for getting products to users and ensuring that features behave as expected.
But AI-generated feedback is imperfect. Graphite’s early idea—add AI teammates to the existing workflow—therefore became only part of the answer. Background agents and automated reviewers can contribute, but the tools around them must also support the increased workload. If developers become orders of magnitude more productive, as Reimers anticipates rather than measures here, the entire toolchain needs to accommodate AI-assisted development, not only the IDE.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Diamond’s place in the review workflow
Diamond is Graphite’s starting point for that broader toolchain. Reimers describes a review platform designed for high signal and low noise, with an understanding of both the codebase and its change history. Its proposed workflow combines several responsibilities:
- Change context: Summarize, prioritize, and review each change.
- Review relevance: Use codebase and change-history understanding to produce useful feedback.
- CI and testing: Integrate with existing infrastructure to summarize errors and correct failures.
The intended scope extends from understanding a pull request to helping resolve problems exposed by its checks.
As Diamond reached larger customers and enterprises, Reimers describes shorter review cycles and more consistent quality as hoped-for benefits that Graphite was beginning to observe. He also presents private and secure code handling, zero setup, one-click suggestions, and customization as product attributes. Some further expansion was not yet public. The product case returns to the broader workflow: AI should improve how changes get reviewed and shipped, as well as how they get written.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Measure whether review comments lead to changes
Reimers closes with reported comment-level results. Here, acceptance means that feedback was integrated into the pull request where the comment appeared. A downvote is explicit feedback on a comment, not a measured code-error rate.
| Measure | Reimers’s reported result |
|---|---|
| Diamond comments downvoted | Less than 4% |
| Human comments integrated into the pull request | Approximately 45–50% |
| Diamond comments integrated into the pull request | Approximately 52% |
The Diamond acceptance figure was new as of March and followed substantial tuning. Reimers does not supply sample sizes, measurement windows, or details establishing comparability between the human and Diamond comment groups. The useful operational distinction is between a reviewer producing feedback and that feedback resulting in a code change: the latter is the behavior Graphite was tuning for.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Further reading
- Diamond's March 2025 launchArticle
Graphite's launch announcement explains Diamond's review features and the company's broader outer-loop product vision.
A 2021 study of insecure Copilot outputs across 89 scenarios designed around high-risk security weaknesses.
A case study describing how Graphite uses developer actions, explicit feedback and evaluation datasets to improve review comments.
Updates since the talk
Current instructions for configuring review rules, exclusions and filters, and monitoring their effectiveness.
Read the complete timestamped transcript
- 0:00
[upbeat music] Awesome.
- 0:15
Uh, hi, everyone. My name is Tomas. I'm one of the co-founders of Graphite. Graphite is an AI code review, uh, company. So to give some context on sort of where we see the industry right now and where we see it going.
- 0:28
Software development currently, and has always had, two loops. The inner loop, which is focused on development, and the outer loop that's focused on review. Developers spend time in the inner loop.
- 0:39
They get their code working. They get the feature the way they want it, and then they go ahead and they move it to the outer loop, where it's tested, reviewed, merged, deployed.
- 0:48
We're seeing the inner loop change right now more than we've ever seen it. More developers are using AI than ever. I think right here we have some statistics from the GitHub Developer Survey.
- 0:56
Nearly every developer surveyed used AI tools both inside and outside of work, and forty-six percent of GitHub is being written by Copi-- uh, code on GitHub is being written by Copilot.
- 1:06
We're seeing more and more code being written by AI. Here we have some statistics around how code has changed over time and how some people predict it will change.
- 1:16
And even if we take a more pessimistic view of that, we still see the way the world's going as just more and more and more code being written by AI.
- 1:26
The inner loop is changing, you know? AI is making more, uh, developers more productive. Developers are now producing higher volumes of code, but that code still needs to be reviewed.
- 1:38
When we first started looking at this, when we first started building, uh, Diamond, our AI code reviewer, about a year ago now, what we found was we read a lot of articles that scared us a lot.
- 1:48
We were seeing within our or-own organization a lot of developers adopting AI tools, but we were also seeing a problem. AI can hallucinate, it can make mistakes, and almost more scarily, it can make security vulnerabilities.
- 2:00
For us, what we saw was that while the inner loop was getting sped up by AI, the outer loop was rapidly becoming the bottleneck. Um, we were seeing tools like Cursor, Windsurf, Copilot, v0, Bull, all of those producing larger volumes of code than we were used to, than we'd ever seen before.
- 2:17
But we were also seeing our developers suddenly have to review higher volumes of code, test higher volumes of code, merge higher volumes of code, and deploy higher volumes of code.
- 2:26
That's what brought us to say, "There has to be a new outer loop here." This-- The way that things are going, this isn't going to work. We're going to break down.
- 2:35
We're watching the problems that used to only ail large companies start to ail all companies, where we were seeing companies deal with higher and higher and higher volumes of code.
- 2:44
The requirements for the new outer loop then look a lot like the problems that larger companies have always had to deal with. You need tools to better prioritize, track, and get notified about pull requests.
- 2:54
You need driver assist features to help reviewers focus and streamline the code review process. You need optimized CI pipelines and merge queues to be able to handle the sheer volume of code changes that are now happening, and you need better deployment tools.
- 3:06
So, um, when we first started looking at this through sort of an AI-first lens, we started to see that, well, the problems are being created by AI, they can also probably be solved by AI.
- 3:19
We can probably start to streamline a lot of these processes which have previously had to been manual, previously were parts of the process that developers did not enjoy, did not want to do.
- 3:27
Um, we wanted to see self-driving code review solutions where we no longer had to do those very manual and painful parts of review, but we could actually start to really focus on what matters most to the developers, making sure that your product can get out to users and that the features work as expected.
- 3:41
Um, we were seeing that AI-generated feedback wasn't perfect, and because of that, we were starting to think that bots weren't enough. I think an early, an early vision of ours was, well, can we solve this by just adding AI teammates, right?
- 3:53
Maybe it's background agents, maybe it's reviewers, maybe it's a whole lot of teammates to the workflow. And while we think that's part of the story, we don't think that's enough.
- 4:00
We think that, as we've built with Diamond, that your entire tool chain has to be AI native, not just your IDE. If you really are going to embrace AI in the age of development, if you're gonna accept the fact that developers are gonna be orders of magnitude more productive than they ever have before, you need tooling that
- 4:15
reflects that. We started by building Diamond, so the winning AI code review platform, with high signal, low noise, has a deep understanding of the code base and change history.
- 4:26
We summarize, prioritize, and review each change, and we integrate with your CI and your testing infrastructure to correct, uh, to summarize errors and correct failures. Um, our hope with it and what we've started to see as we've rolled it out to larger and larger customers and enterprises too, is we reve-- we reduce code review cycles, we enforce
- 4:45
quality and consistency, and we keep your code private and secure. Um, it's high signal, it's zero setup, it's actionable with one-click suggestions, and it's customizable. It's already being used by some of the fastest moving companies in the world.
- 4:57
It's expanding a lot more than we can even say publicly, um, a-and I hope that you all will embrace the idea that AI can change your entire developer workflow, not just your IDE.
- 5:08
Um, by the numbers, we see comments that our AI bot leaves to be downvoted at less than a four percent rate and to be accepted, meaning integrated into the pull request, um, that they were left on at a higher rate than human comments are.
- 5:20
Human comments are integrated about somewhere between forty-five and fifty percent. We're watching our Diamond comments be accepted about fifty-two percent. We've spent a lot of time tuning that. That's, that number is actually new as of March for us.
- 5:30
Um, that's, that's what I have to tell you around Graphite, um, what I have to tell you around Diamond. I hope you give it a shot, and, and thanks for having me. [upbeat music]