How to Build an AI Code Review Agent - Daksh Gupta, Greptile

Um, all right. My name is Daksh. I am one of the co-founders of a company called Greptile, and we build agents that review and test pull requests with the full context of the codebase. And by reviewing and testing these pull requests, we reveal bugs, logical errors, code quality concerns, primarily from AI-generated code. Now, to give you a sense of scale, many thousands of companies use Greptile every day. About one percent of all pull requests on GitHub go through us. And a

variety of many different types of codebases, whether it's the CUDA codebase at NVIDIA or the wallet codebase at Coinbase, and many, many others. And every now and then, we meet a company where we are actually not the right code review or code testing product for them. Their needs are so specialized and so esoteric that it's actually more beneficial for them to build their own. And when that happens, we tend to help them for free. There's two reasons for this. One, we have a lot of specialized knowledge about how to do this really, really well. And second, we get to learn a lot about how we can serve

customers like those in the future. And in this presentation, I wanna tell you what we tell them, which is how to build a really great AI code review agent yourself. Now, there's kind of three things that people like about Greptile, and I'm gonna talk through how to implement those yourself. The first one is codebase context. When customers use Greptile on their pull requests, they find that it really understands their codebase. As a matter of fact, before we were a code validation company, we were a codebase context company. We were teaching language models from

2023 how to understand very large codebases. And as a result of that, we got very good at indexing code. Uh, the second thing that people like about Greptile is that it runs your code, in the sense it spins up your stack in a fully loaded dependency-rich sandbox, and it clicks around and tests things using computer use or through pressure testing API endpoints, and so on. And the third thing is we're actually very efficient allocators of inference. We pick the right model, the right quantity of inference budget for each subtask in the review and validation process. And as a result of that, we're able to get a lot more performance per

dollar than an internal tool could or any competitor could. So there's kind of two reasons that you would want this presentation. Uh, the first one is you want to build your own review agent. And the second reason, which is a bit more subtle, is if you're trying to pick a review or a testing agent for your company, and you want to understand what makes a good one and what makes a great one. All right. So let's dive right in. Now, blank slate. You are part of the platform team of your company, or you're just the, uh,

local, uh, AI psychotic member of your team, and you want to build the review validator of your company. Um, company's writing lots of code. People are vibe coding. Uh, you have, of course, engineers vibe coding, some of them a little too close to the sun. Uh, you have marketing people writing code. And, uh, the new problem is the review bottleneck, the validation bottleneck. And so that's what you want to address. The first thing that you should do is you should get yourself a sandbox. There's many, many companies here that will sell you a sandbox. Uh, you can also just use Firecracker or something

like that. And you'll want to put an agent in that sandbox. Uh, the agent can be Codex or Claude Code. Personally, I recommend going model independent for reasons that I will get into. So you can use something like OpenCode or Pi as your harness. Those are both open source. And, uh, you will need to then connect your GitHub or your GitLab, have a PR webhook send it the contents of a pull request to when a pull request is opened. The agent can have some type of prompt that just says, "Hey, review this and send back some feedback." And then you can wire that back using GitHub's review APIs. Very simple. In fact,

many people think that's actually all it takes to build a review agent. There's a few things that are gonna go wrong when you start using this. But what will happen is you will, in fact, have a pull request go live. The contents will go to an agent, and about five minutes later, you will get some feedback on your pull request. But here's what's going to go wrong in the time that it takes you to build this. The first one is you're gonna miss a lot of bugs. So it turns out that actually bugs are intractably large. They're... In fact, I saw a, a tweet not very long ago that, uh, said that bugs are fractally infinite, so you will never run out of, of

bugs in a particular piece of code. That's the first thing that's gonna go wrong, is you're gonna miss bugs. Of course, that's the one thing that your system needs to do, is to prevent at least very catastrophic outages and incidents. Second thing that's gonna happen is it's gonna miss context, and it's, it's going to misunderstand parts of how your codebase works. It maybe won't understand where you're going with your codebase. It won't understand how a specific part of it works. It maybe will not understand how one of your libraries works. It's possible you're one of those companies that never recovered from the microservices apocal- apocalypse of the 2010s, and so you just have thousands and thousands of repos, and

you have to have context of all of them to be able to operate. So that's the second thing that's gonna go wrong. The third thing is it'll be very expensive. I think people are very surprised by this. So generally, from people we've seen try to do this, they'll find that they're spending between $3 and up to $25 per pull request on reviewing it using coding agents. And these tokens can add up really, really fast. Very quietly, they got very expensive.

The fourth problem they'll find is nitpicky comments. Now, this matters more than you might think. Nitpicky comments are comments that are technically correct but overly pedantic. There's two things that go wrong when you have nitpicky comments. If you are a... As a human reading those comments, they're very annoying. They're like a boy that cries wolf. You'll just see lots and lots of comments, and it'll be kind of annoying for you to just read through them and figure out which ones are just minor stylistic complaints and which ones are actually catastrophic problems. The second thing that will go wrong is they will wreck your codebase. Because typically, the way the review agents work is they'll leave comments on GitHub, and your coding agent picks them up

and resolves them. And coding agents are not discerning. They're going to look at review comments just to address all of them. And you'll end up with extremely painfully defensive code, as we found the coding agents are particularly astute at writing. The next thing that'll go wrong is that every time you run a review agent on the same pull request, you will somehow find new bugs, in the sense that it will keep catching new things in the pull request very annoyingly, and you will... And like I said, bugs will appear to be fractally infinite. So the first thing that you

wanna do, the lowest hanging fruit, is you wanna optimize the prompt. You start it with a very simple prompt, something like, "Review this PR and then output the comments in GitHub's API schema format." Generally, there's two themes within the bugs. Look for specific bug types, and second, skip the nitpicks. And that's what you want to orient your prompt around. You'll want to ask for specific bug types, and you can kind of do some internal data science on existing outages and incidents that have happened in your codebase, and figure out what types of bugs you have the highest propensity to produce. Create a catalog of them,

and then have your prompt look for those specific bug types. The second is you'll need to expressly tell it what a nitpick is and not to look for them. And nitpicks can be anything from asking why there isn't three levels of defense on this particular thing or why error handling is done in this slightly shoddy way. Now, the thing that you'll run into when you change these prompts is when you change a small thing in the prompt, it'll have a weirdly big impact on the number of comments that your agent is writing on your pull request. This is quite scary because it means you don't really know if it's missing a bug or

not. You don't know the counterfactual. The validation agents or really any reviewer or tester can, can tell you about the presence of bugs, but it unfortunately can't tell you about the absence of bugs. That's, that's sort of a, a limitation of the format. And what makes this particularly difficult are bugs are extraordinarily rare and getting rarer by the day. In fact, according to our data science, only about one out of every thousand pull requests has a generally catastrophic bug in it, or I mean one out of every thousand lines of code has a catastrophic bug in it. And so because these bugs are very rare, you might release a version of your

agent that you think is very good, but it secretly misses a catastrophic bug in a pull request. You wouldn't know for a long time because you don't produce those bugs very often. And so that brings us to the second improvement. You will want a dataset of historical PRs. This is relatively easy if you're one of those companies that very nicely catalogs every time something goes wrong in your codebase. If you have that, then you'll have a list of pull requests that have caused outages and incidents in the past. You'll wanna catalog them and then essentially have any version of your review agent run through those and see with what

frequency it can catch those bugs. Now, this will allow you to tune the prompt a little bit more confidently. But at the same time, the interesting thing about bugs is, in a changing world, the topology and ontology of bugs changes over time. And so you'll need to have a real-time updating dataset. And this is tricky because unless you have millions and millions of pull requests, it's actually hard to have reasonably high coverage of bug types. It's true that about 80 or 90% of your bugs will be just repeats of prior mistakes, but increasingly, as coding agents get good, they're going to start making novel mistakes.

They'll fail in new ways that we have not anticipated before. And so this is where you will start to hit limitations. Now, quick side note, this is kind of what we did at Greptile, except we actually have a lot of code from a lot of different companies, and we can test across a very, very large corpus of pull requests and therefore improve our agent very confidently. It's one of the reasons why we're able to eke out little bits of performance on the margins in ways that internal review agents can't. But you can get decently far, especially for your particular company, to do this with your internal data. The next thing you'll wanna do is model inversion.

So for... In case anyone was at, uh, my talk earlier this year in San Francisco at AI Engineer, I talked a lot about the ways in which coding agents fail. And I discovered an interesting pattern through that research, which is that the types of bugs produced by Claude models or Codex models are actually quite different from each other. And they're different in a number of ways. As an example, Claude models are something like three times more likely to produce one-off errors than GPT models. GPT models, on the other hand, are significantly more likely to produce auth bypasses than Claude models. And these are all just

artifacts of the post-training, as far as I understand. Now, to prevent this, you'll find that if you're using the same model to review the code written by the same model, um, you'll actually have relatively fewer bugs caught because they fail in the same ways during review that they do during code generation. And so you'll want to invert the model. If you've used Claude Code to write the code, you'll want to use a GPT model to review it, and so on. Now, doing this is somewhat straightforward for the 80th percentile and quite tricky for the last 20. So

if you're using coding agents, you'll notice they leave some type of fingerprint to tell you what coding agent was used for a particular pull request. For instance, Codex will name branches in a codex/branch name format, or Claude is going to write a "Co-authored by Claude" in the PR description and commit message. And about 80% of the time, this will be true. In the final 20, when you happen to manually create a branch name or there was some other reason that a different coding agent was perhaps used, it's significantly harder to figure out which model was used during generation. And you have to develop some type of heuristic to figure out what are the signatures of models being used. Often, you might have an internal

catalog where people report what coding agents they're using these days or have some type of telemetry that's coming out of your dev machines and, and your engineers' computers. We found that there's a 10% higher catch rate if you flip models. And this might sound very small, but you have to know that 10% is, in terms of incidents and outages, is actually quite a lot. That means that there's a 10% higher chance that something will go wrong if you're not using model inversion, which can be quite catastrophic. And so we highly recommend that you use some type of model inversion in your process.

The third improvement that you can make is codebase context. Now, this is something that I care a lot about. Before we were doing code validation, we were doing codebase context. Uh, we were trying to figure out better ways to do RAG over very large codebases. And even today, people consider grep to have effectively solved all of codebase context. I actually think that's not true for a number of reasons. One, grep is extremely imprecise. And second, it's very expensive. To give you some sense of scale, single-digit percentage points of GDP growth last year were inference. A very large percentage of inference was coding agents. And if you actually look at a coding

agent trace, the vast majority of tokens used by coding agents are all the same: grep, file read, cache write for the file contents. That means a non-trivial percent of GDP growth is grep, file read, cache write. That's very bad. And this is because grep is very imprecise. It's reading files in very large quantities. And I don't know, there's perhaps some adverse incentives for companies paid by the token to figure out ways to burn more tokens. But even so, you'll find that you'll need to use slightly more surgical methods of figuring out where you should be reading. And the way we do this is we produce an

internal knowledge base where we essentially check across the entire codebase all the different parts, catalog them, and then index saying, "Okay, well, this is authentication. Here's how it works. Here are the flows. Here's a diagram of the flows. Here's every file that's involved." One example of where codebase context matters is, for instance, when you're trying to catch an auth bypass It's actually very difficult to figure out if an auth bypass has occurred because you need to have a pretty concrete understanding of whether you're in an authenticated environment when you're in that part of the codebase or not. And that's very tricky. And so you'll find that you will need some type of, uh, of sort of codebase context to be able to

do this. Now, I've been very centered around the concept of bugs, but code can be very, very bad even if it's not buggy. For one, it can be terrible quality, but second, it can just not fulfill the expressed intent of the author. Recently, I saw a post that said that people were increasingly using Slack as their IDE. Now, I think that's a little funny, but at the same time, there's not that much ex- expressed intent in a Slack message. You're only saying maybe a sentence or two, if even, asking an agent to go and fix something, which means the,

there's a lot of ambiguity in your instruction. As a result of this, you might find that the coding agent has either misinterpreted your instructions or just it was too ambiguous and it didn't know exactly what to do. And so your correct code can actually still be quite wrong. Now, one of the ways in which we fight this is by giving agents increasingly more context. And we find that the two types of contexts that matter most are docs and tickets and, of course, the codebase from your codebase indexer or knowledge base. You can get remarkably far if you give it access to your Jira and Linear and then your Confluence or Notion or

whatever it is that you store your docs. You'll find that there's a lot of tribal knowledge that is subconsciously helping you produce better code. And when you have agents do the same thing, it feels like they just don't know what's happening, um, because you over-- you underestimate how much depth of knowledge you yourself have.

Now, the way that Greptile approaches this problem is we actually disintermediate all of these context sources with a living wiki. This wiki is essentially a sort of world model of how your company works. It's everything that it knows about your codebase, everything it knows about your application through production telemetry, through logs, everything it knows about your roadmap from your documentation and your Linear tickets and so on and so forth. And it uses this wiki to supply context, both making the agent more context-aware and making its code searches significantly more

precise because it will never read a file that it didn't need to read.

The last thing that you should do... All right, this might be four out of five. The next thing that you should do is autonomous testing. One of the things that we think about a lot is that, for instance, for UI changes, you might be able to read through a bunch of change files for React code and conclude that there's nothing wrong with the code. But in fact, when you render it, it might be, for instance, hidden on an iPhone. Now, this is one of the more technically difficult problems to solve. The way that we approach this at Greptile is we essentially build these dependency-rich

sandboxes, which are completely configurable virtual machines. And we have customers, we work with them to essentially spin up their stack in these sandboxes, and then Greptile's agent uses computer use to test them end-to-end. If you haven't been paying attention, computer use has gotten very good, and it's, like, far better in these new generation of GPT models and Opus models than it was in the previous. For the first time, I think end-to-end QA is n-nearly solved. The thing that you need to do, though, is you've to document all the flows through your application. And second, you need to be able to have a sufficiently simple build

script so that the agent can spin up your stack in your environments. We have a team of forward-deployed engineers that help companies do this. We found that it's one of the trickier parts, unless, of course, you have very good dev boxes already, in which case, congratulations on working at Stripe.

So... Oh, I-- this is the diagram I meant to show you. But yes, you will need a dev box. You will need PR branch and dependencies where you're running the app, and you'll need some type of flow map that says, "This is the auth code, and then this is the sign-up flow," and so on. And using this, your agent can now test your code end-to-end and use vision models to say, "Oh, like, this button technically appears, but it actually is hidden on iPhones," or, "It's not in the first fold on, uh, the iPhone Duo," et cetera. Now, if you do all of this, you'll find that the costs are still quite high. It can cost you anywhere from three bucks to fifteen to twenty to do a single pull request review.

And the thing you'll want to solve for now is to use cheaper models for some things. Now, to me, frontier models are still, generally speaking, more capable than open source models. This has been true for some time, even though the rubber band is tightening. So our share today is about twenty percent frontier models for the most critical tasks, and then the long tail of tokens, which is exploratory, file reads, and so on, we use a variety of smaller models. And so if you do this, you can actually reduce the cost of running your review agent quite drastically. So this is sort of

a illustrative example of what we do, which is we'll use a small model for finding context. We'll use a very lightweight computer use model for testing user flows, and then we use a frontier model for things like analyzing bugs, applying the review process, and for submission. In practice, we have actually 20 different subtasks, and each of them has a very specific set of models that we use for each of them. And we have a small research team that is constantly iterating exactly what models to use for what task. You can probably get seventy to eighty percent of the way there using something of this kind, where you just swap out very simple tasks with very simple

models. Now, this is another example of it really helps to have a very large dataset. If you don't have a good dataset, you might be able to ha- trigger happily replace a lot of different sub-agents with smaller, faster models, and you won't know that it's, this agent is actually now quite a bit worse at catching bugs, and you'll only find out when it misses something that then causes an outage or incident.

And you'll also find that you can kind of combat this problem of overactive file reads using a cheap model that essentially looks through all of your context pieces and figures out which ones actually should be included. In theory, you should be able to use a decision model for this, something like Jev or the Decisions API, to do effective re-ranking on all of the different things and deciding what should be and shouldn't be loaded into frontier model cache, which is, in some ways, your most expensive and most scarce resource. If you do all of these things, you might be able to actually get quite far in building your own review agent. And if you do,

please let me know. My email is daksh@greptile.com. I'd love to help you. And, uh, if you choose not to build it, then you can also just use our product, um, which is free in comparison. But, uh, thank you so much.