AI Engineer World's Fair 2026
How to Kill the Code Review
Read the talk
How to Kill the Code Review Without Losing Team Alignment
Replacing line-by-line review requires more than an AI reviewer: preserve implementation decisions, turn recurring feedback into checks, and review evidence against intent.
From a talk by Ankit Jain
Before you start: Familiarity with pull requests, automated tests, and AI coding assistants will help you follow the workflow.
What would make line-by-line review unnecessary?
What evidence would let a team merge a change without reading every line? Ankit Jain, Aviator’s co-founder, approached that question in How to Kill the Code Review, proposing five layers of trust around generated code. Building an AI code verification platform has led him to revisit that proposal: the challenge is not only establishing that code works, but preserving what teammates learn and decide through review.
The immediate pressure is growing code volume. Jain describes a bottleneck moving from writing code to reviewing it, citing an 861% increase in churn, rising incidents per pull request, and longer review times. Churn needs a precise reading here: the closely matching Faros analysis defines it as quarterly deleted lines divided by added lines. An increase in that ratio is not an equivalent increase in useful output.
Jain describes review waiting time as four times its previous level and says over 30% of changes merge without review. The matching Faros figures use different denominators: median time in review increased 441.5%, and 31.3% more PRs merged without review. Those are relative changes between organizations’ lowest and highest AI-adoption periods, not the share of all PRs lacking review; time in review is also not necessarily time waiting for a first reviewer. The findings are observational correlations.
Now consider the everyday GitHub workflow: two or three AI reviewers leave comments, a coding agent or developer receives them, and another round of changes resolves the feedback. The human arrives at the end, assumes the agents have found most problems, skims the diff, and merges. If nobody meaningfully reads the exchange, why organize it around a human-facing review interface? Adding automated reviewers has preserved the ceremony without necessarily preserving human judgment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The part of review that must survive
Today’s review interface is not an immutable requirement of software engineering. Jain points to Google’s Mondrian in 2006 and early Windows development to illustrate how practices have changed. That history should be read narrowly as a discussion of modern tool-mediated review: formal inspections go back at least to Fagan’s work in 1976, as documented in Modern Code Review: A Case Study at Google. The specific Mondrian date and Windows comparison are Jain’s historical examples.
Review performs two different jobs:
| Function | What the team needs |
|---|---|
| Semantic accuracy | Find bugs, enforce conventions, identify security issues |
| Alignment | Share knowledge, mentor, discuss architecture, onboard teammates |
The second job was missing from Jain’s original trust model. A reviewer explaining a design choice or helping a junior engineer understand a service boundary is doing more than checking correctness.
This distinction matters most for teams collaborating on a shared system. A solo project or a fully unattended development operation has different coordination needs. For a team, alignment must survive even as correctness checks become more automated. The design question is whether shared understanding can come from something more useful than reading a diff line by line.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A specification cannot contain decisions nobody has made yet
Spec-driven development offers an appealing sequence: write the specification, give it to an agent, generate the implementation, then verify it. Jain compares that sequence to the familiar waterfall model associated with 1970. The criticism applies specifically to a frozen, one-pass workflow; Royce’s original paper already described feedback between phases and warned about the simple sequential approach.
The missing feedback becomes visible during an ordinary Claude Code, Codex, or Cursor session. The developer interacts with the agent because the specification leaves questions unanswered. Implementation then exposes additional problems. If the specification is treated as finished, those discoveries never make it back into the document that supposedly defines the work. An LLM also makes implementation choices rather than producing a deterministic expansion of the specification. Spec-driven development remains useful, but its most valuable contribution is intent, not an assumption that all decisions can be completed up front.
Intent is distributed across several artifacts:
| Artifact | What it contributes |
|---|---|
| Jira ticket | The goal |
| Product requirements document | The detailed plan |
| Prompts and agent conversation | Decisions made during implementation |
The conversation is especially valuable because it records the developer’s answers as ambiguities emerge. Creating a pull request and discarding that conversation throws away part of the explanation for why the code exists in its current form. Preserve those decisions with the change.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn repeated comments into reusable checks
Preserving intent does not eliminate bugs. LLMs make mistakes, and AI reviewers can miss them. Jain still expects some manual review; the opportunity is to stop spending that human attention on the same findings repeatedly. His AI slop registry, also called a register, collects recurring review issues and turns them into guardrails that can be checked automatically.
Each repeated finding becomes an input to the verification system rather than another comment somebody must remember to write. As the registry grows, it incorporates the team’s review experience. Jain describes this as learning or training on top of a standard LLM, but does not specify model-weight updates. The concrete mechanism is codifying recurring feedback so the system can identify it again.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make intent and guardrails produce a test plan
Alignment and semantic accuracy meet in a single verification loop:
- Capture the developer’s responses from the coding session.
- Turn those decisions into acceptance criteria.
- Combine the criteria with the maintained AI slop register to create a test plan.
- Start a preview environment and verify the plan end to end.
- Review the intended capability alongside the observed behavior.
The acceptance criteria describe what this change should accomplish. The registry contributes constraints learned across changes. Together, they give verification a target beyond whether the implementation looks plausible.
The review surface becomes intent plus evidence. Did the implemented capability match what the developer meant to build? Did its behavior satisfy the acceptance criteria? Architectural disagreements still belong in the review; they now sit beside explicit requirements and observations rather than being inferred entirely from the diff.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Generate the tests, review the plan
In Aviator’s walkthrough, even a simple task involves clarification questions and user feedback. Those exchanges contain engineering decisions worth retaining: they explain the implementation to teammates and give junior engineers examples of how to reason through a change. An LLM can extract those decisions into acceptance criteria and help create the test plan, reducing the effort of translating a conversation into something verifiable.
Criteria plus invariants then drive checks in a preview environment. Jain proposes generating tests in real time for new features, with the aim of removing much of the recurring test-writing and maintenance burden. Human governance shifts toward reviewing whether the plan asks the right questions. The walkthrough slide makes that surface concrete: a checklist sits beside pass/fail results, including a red failure for Threshold typed as Money. A failed check is visible evidence to investigate, not a successful outcome to infer from plausible code.
This resembles behavior-driven development more than simply moving traditional test-driven development into an agent. An English-language test plan lets product managers and designers participate in deciding what correct behavior means. Where criteria admit deterministic checks, the verification system can then establish whether each one was met. Human agreement about the requirement and machine checking of the result play different roles.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Verify a payment form through observable behavior
Not every part of verification can be deterministic. Jain’s rule is “deterministic where it can be, but LLM where you must.” Use precise checks where possible and agent judgment where the task requires it, without treating the whole system as perfect.
His example is a newly introduced payment form. A test plan describes the behavioral change; an AI agent browses the application, fills out the form, and captures screenshots. Those screenshots, together with database snapshots, provide evidence for deciding whether the acceptance criteria were met. The browser view shows what the user encountered, while the stored state helps establish what the application actually did.
For a concrete implementation of that evidence check, suppose the captured requirement says a successful payment must create a paid record for order order-42, for 2,500 cents in USD. The following TypeScript checks that proposed criterion against before-and-after snapshots; screenshot evidence remains a separate part of the review:
typescript
type Payment = {
id: string;
orderId: string;
amountCents: number;
currency: string;
status: string;
};
function verifyPayment(
before: readonly Payment[],
after: readonly Payment[],
): boolean {
const previousIds = new Set(before.map(payment => payment.id));
const added = after.filter(payment => !previousIds.has(payment.id));
const orderPayments = added.filter(
payment => payment.orderId === "order-42",
);
return orderPayments.length === 1 && orderPayments.every(payment =>
payment.amountCents === 2500 &&
payment.currency === "USD" &&
payment.status === "paid"
);
}
The agent’s navigation can be flexible while this particular check remains exact. Screenshots and snapshot checks give the reviewer inspectable evidence; agent-based screenshot assessment can still require judgment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Derive the checks from decisions, not just the implementation
The session should retain what was requested, what was attempted, and what was rejected. That origin matters. If an agent writes code and then derives the test plan only from that code, it can reproduce its own mistaken assumptions in the checks. Jain connects this warning to a point he recalls Dex making the previous day: tests that merely describe the implementation can miss the gap between implementation and intent. Session-derived criteria provide a separate statement of what was wanted.
The reviewer can then discuss data models, service interactions, and architectural choices at the level where collaboration is most valuable. Those decisions remain open to challenge, and the collected verification evidence shows what happened when the proposed behavior was exercised. Moving beyond line-by-line review does not remove technical discussion; it gives that discussion a different set of artifacts.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build the register before retiring the ritual
Jain’s starting assignment is concrete: mine the last 1,000 review comments for repeatable issues. Turn the recurring best practices into entries in the AI slop register. Each merged PR can then contribute another reusable check, reducing the need to write the same feedback again while preserving the collaborative part of review.
That investment has an initial cost. Jain describes a J-curve: building the registry takes time and creates pain before the accumulated checks pay off. The adoption slide explicitly cautions against killing review on day one. His invitation to try Aviator’s approach sits within that gradual transition: improve tooling for semantic accuracy while retaining the alignment that makes review valuable to a team.
At the time of the talk, Aviator Verify was being piloted with a few companies and seeking early design partners. Its proposed combination is the same one the workflow requires: captured intent for alignment, and a growing register of reusable checks for semantic accuracy. The goal is to make a merge decision rest on shared decisions and verified behavior, with human attention still available for the questions the checks do not settle.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Jain's original five-layer proposal for replacing line-by-line review with intent, constraints, and verification.
Official product overview covering intent capture, acceptance criteria, verification evidence, and access requests.
Further reading
First-party explanation of code churn and review metrics from telemetry across 22,000 developers and 4,000 teams.
Research on Google's review practices, with historical context distinguishing formal inspections from modern code review.
Royce's August 1970 paper on development stages, feedback, and the risks of discovering problems only during final testing.
Read the complete timestamped transcript
- 0:00
[upbeat music] Okay. Ooh, hello. Hey, everyone.
- 0:16
Thanks for joining in. Today, uh, we will be talking about how to kill the code reviews, everyone's favorite topic. I'm Ankit, co-founder of Aviator. At Aviator, we are building AI code verification platform.
- 0:29
So we'll bring in some of the ideas and concepts that we talk about in the, uh, that we build in our product. Uh, but first, let's dive into a little bit.
- 0:40
So a few months ago, I wrote a post on Latent.Space about, uh, how to kill code review, creating a framework, a five-layers trust model. So this model was focused around how do we actually, layer by layer, build trust into the code that can then be merged without needing line by line review.
- 1:03
And I got some things right, and I got some things wrong. So this talk will be about just really diving a bit more, more into it. I'm not gonna talk about specific layers, but we will talk about some of the concepts that emerge from this session.
- 1:17
So let's just talk about the problem. We are looking today at the volume of code is increasing every day, and we are struggling to keep up. So when we think about, like, how long will it take us to actually stop reading code line by line,
- 1:35
and the reality is we have already stopped reviewing it. There is eight hundred and sixty-one percent code churn. That means we are, like, producing more code. The incidence to PR ratio is increasing.
- 1:46
That means even if you're doing reviews today, they're not effective. So the medium time of review is increasing. We have just increased the bottleneck to-- from the coding, now kind of like coding is solved, to now reviewing, where everything just gets stuck there.
- 2:03
You're spending four x the time that you were spending before just waiting for the reviews. And today, over thirty percent of changes are actually getting merged without a review at all.
- 2:14
So let's just think about it a little bit. Think about AI reviews. So everyone is probably using some form of AI reviews today. When AI write the code and AI review the code, why are we doing it in a UI, right?
- 2:28
Like, we open a GitHub, there's kind of like, you know, maybe two or three AI coding agents who are, like, doing the reviews. You know, you-- the review goes back-- passed to the user, uh, or the agent, and it gets resolved, and then kind of like you're doing back and forth with the agent.
- 2:42
Where is human here in the loop? Like, you're eventually just looking at, okay, you know, if AI has reviewed it, most of the things probably we found. Let's just kind of like, you know, do a skimming of it and merge it.
- 2:53
So when AI reviews and nobody reads, we have configured the wrong thing.
- 3:00
So let's just take a step back. Code reviews are not very old. They are maybe fifteen to twenty years ago. In two thousand and six was when Google launched Mondrian internally, and they made formal code review as a thing.
- 3:13
If you think about our Windows back in the days, the first versions were actually built without reviews. But if we look carefully, code review is not just about code reviews.
- 3:24
You know, obviously, we are looking at it like catching bugs, understanding kind of conventions, identifying security issues. But code reviews is also about alignment. And this was one piece which was missing from my five layers model that I talked about a few months ago.
- 3:40
So a big part of code reviews is knowledge sharing, mentorship, architectural feedback, onboarding, being able to collaborate. This is, again, if you're doing, um, white coding, you're working on a solo project, this is not a talk for you.
- 3:56
If you are working in teams, which I believe most of you folks are, collaborating in teams, you're not likely using completely dot factories, orchestrators, where nobody looks at the code.
- 4:05
You're actually collaborating in teams. You need to do knowledge sharing, which is the alignment part. And that is the part which is the most important aspect of reviews. So for semantic accuracy, we can build better tooling, but alignment must survive.
- 4:21
So let's just kind of like dive into alignment. What does it mean? Like, in today's world, can we actually think of better model than aligning just based on reading line by line code?
- 4:34
So most folks have probably heard about spec-driven development by now. So spec-driven development is like, okay, we write a spec. You know, um, it covers all the details. We pass it to an agent.
- 4:45
It generates the code, and then we verify. So what's wrong here? If you look back in nineteen seventy, this is what waterfall model was. You know, you have requirements, you have specification, you implement and then you verify.
- 4:57
But there's no feedback loop. You know? The spec is written before we identified everything else, right? That's why today everyone still wants to use your coding sessions, whether it's Cloud Code, Codex, Cursor, whatever you're using, you want to interact with the agents.
- 5:13
And the reason you're interacting with agents is because there were certain things which are not clear in the spec, and we still need to capture that. And second is, as you implement, you identify more issues, and you never go back and update the spec because, like, if you're doing a spec-driven development, it's already done.
- 5:29
Once the spec is done, you expect, like, you know, the code will come deterministically. But guess what? LLM is not deterministic. It's gonna make decisions itself. So that's why spec-driven development is a great, uh, methodology, but it falls short in day-to-day software development.
- 5:45
But there are some interesting aspects of this which we should carry forward. The most important part is the intent.
- 5:53
And intent doesn't only live in the spec.
- 5:56
Intent live in your Jira ticket. That's the goal, right? Like, it's kind of like where you express what we want to do. It lives in your PRDs. It's kind of like details, like it's a plan.
- 6:07
But most importantly, it lives in your prompts today. This is where the real decisions are being made. You start with like, okay, this is a Jira ticket I'm gonna look at, but you're going back and forth with the agent, and this is where all the user decisions are being made.
- 6:21
But what we do today is we create a change, we cr-create a pull request, and then we throw away the prompts. And this is one of the things that we need to change.
- 6:32
Let's just first talk also about semantic accuracy because like, you know, you're saying, "Hey, Ankit, okay, I understand the alignment part, but like there are still bugs in the code.
- 6:39
Who's gonna look at that?" LLMs are also not great at this, and we already talked about how AI agents may-- reviewers may not also be perfect.
- 6:49
So this is where I introduce you to a concept of AI slop registry. So think about this. We are reviewing-- If you're reviewing code today manually, and I expect everyone should be doing some degree of this,
- 7:02
we are essentially possibly identifying the same issues over and over again. Can we actually capture these concepts and codify them so that we don't have to always create those review feedback one by one?
- 7:18
You actually also have all of those things automatically identified. The beauty of this is if you do it a few times, you now build sort of like a system which actually learns over time.
- 7:29
So think of this as sort of like you're not-- you know, you're doing more training on top of the standard LLM that you have actually extracted, uh, like built on top of.
- 7:38
So AI slop registers now can create better results because it's trained, it's learning from the review experience that you as humans are providing.
- 7:52
Every referring comment is now a guardrail that you don't have to review again.
- 7:58
Okay. So let's kind of like try to put both of these alignment and the semantic accuracy together.
- 8:05
It is two halves of the same problem. We are trying to id-- understand what are the core mechanics of review, how do we actually break it down into two parts, which is the alignment and the semantic accuracy, and bring them together into a single loop.
- 8:19
So first thing is you go take your session and you capture the, uh, user responses, and that essentially forms your acceptance criteria. The acceptance criteria then tied with your AI slop register that you are now con-constantly maintaining finally creates a test plan.
- 8:38
And this is a test plan which then gets verified. This is part of the system that we are building is the verification system, where it spins up a preview, takes your test plan, and make sure it actually works end-to-end.
- 8:51
Even if the code looks right, does it actually work? So this verification part's become interesting. And now the kicker is this is now your review surface. You're not reviewing code line by line, but rather you're looking at the evidence of what was the intent, did the user actually implement the capability that was defined in their intent, and
- 9:15
did the behavior actually meet the requirements that we had in the accept-- as acceptance criteria. So you're still having the architectural decisions, you're still having these arguments, but the review surface changes.
- 9:30
So just giving a walkthrough of like how we have built our system. The session becomes a criteria, so all these decisions that you are doing here with the agent, you're asking, uh, you know, you're providing this feedback.
- 9:43
Like even if there is a simple task, many times you're going back and forth. Agent will talk to ask questions. These are the decisions that we need to capture.
- 9:52
This is the intent. This is what makes your review, makes your collaboration more valuable. This is how you teach your junior engineers on how to improve over time. These are the decisions which make a software engineer valuable today.
- 10:11
We convert those into an acceptance criteria. This is where you can also leverage LLM to do so. So you capture these user int-- uh, decisions and make sure you can actually create a test plan based on this.
- 10:22
I know test plan creation is always painful, which is where I would always recommend people to use LLM for this purpose.
- 10:30
And finally, the criteria plus invariant is what makes the test plan, uh, and we build the verification systems to actually capture the test plan, run your previews, and be able to test based on this test plan.
- 10:44
So even imagine if you're building a new feature, you don't have to maintain tests at all. This is creating test in real time, and this is where you can leverage the power of LLM because the test plan maintenance and creation can be very painful.
- 11:00
But the value of human in the loop here is the governance and the review part, and the part where you're reviewing the test plan and not the code, right?
- 11:14
I know it's been like over twenty years we came up with test-driven development. This, in some ways, is closer to behavior dev-de-driven development, where the test plan is now something which even you can share with your product managers, your designers.
- 11:28
Everyone can participate because these things are now in English. At the same time, we have now deterministic verification which actually verifies whether the particular test criteria has been met or not.
- 11:44
Let's move on. So this is where I would say the system is not supposed to be perfect. It's deterministic where it can be, but LLM where you must. Not everything can be
- 12:02
deterministic. Not every system can be built in a way which is hundred percent built on deterministic, uh, systems. This is where you use LLM as a fallback. Let me give you an example.
- 12:17
Like, if you're making a change in your web application, what you can do is it creates a test plan of what the behavior changes. Let's say you introduce a new payment, uh, form.
- 12:28
So it creates a new payment form. The verification here is, does the payment system changes? Uh, an AI agent can go and browse through your application to fill out a form and capture screenshots as evidence, and then take those screenshots as well as your database snapshots to identify whether the criteria was met.
- 12:52
So the screenshot testing or, like, sampling can then still be done by agents, but at the same time, you're creating more solid evidence which now a reviewer can look at and build more confidence that this actually works.
- 13:08
So now reviewers are reviewing the intent, not the diff. You're reviewing the intent decisions, uh, what we said to build out, what we tried and rejected, and capture all of these things from the sessions.
- 13:22
Remember, capturing it from the sessions is the key. If we try to build it from the code, you end up in the same situation that we were talking about, I think Dex was talking about yesterday, which is if your code is built by the same agent which is actually building a test plan, it's not going to build
- 13:40
a test plan which will actually catch issues. So that's why it's important to actually use the session information to build out a test plan. You can discuss architectural decisions, so how you're creating the data models, how these services interact with each other.
- 13:54
So you're moved one level above. So you're essentially, instead of, like, reviewing line by line, you're actually having discussions on architecture, which are very critical for any kind of collaboration.
- 14:05
And then you look at the evidence, everything that was collected from the verification.
- 14:12
So here's a homework for everyone. Go home and mine your last one thousand review comments and build out a AI slop register for the things which are repeatable. So a vast majority of the comments that you're providing in your code review are something that we repeat over and over again.
- 14:34
This compounds with every merged PR. Every time you capture something as a register, you don't have to capture that comment again. And this is where you can actually codify some of the best practices of doing code, maintain semantic accuracy.
- 14:51
At the same time, do not lose the collaboration part of the review.
- 14:58
It does follow a J curve, so pain is real. Uh, you will have to spend some time to actually make it pay off because initially creating a registry can take some time, and this is where I would recommend you folks can come and try out our product.
- 15:13
So code review is not just about code review, it is about, uh, really getting the alignment, and where we can build better tools is creating the semantic accuracy and defining your AI slop register.
- 15:29
So if you remember one thing from today, remember code review is not just about code review, it is about getting the alignment.
- 15:38
And yes, we are, uh, piloting our new product called Verify. Uh, please, uh, join and, uh, be our early design partners. We are working with a few companies to, uh, pilot out a new verification system.
- 15:52
This combines both the alignment side of things as well as building tools and capabilities for detecting semantic accuracy using the AI slop register.
- 16:04
Thank you, everyone. Thanks for joining. [audience applauding] [outro jingle]