# How to Kill the Code Review

Ankit Jain, Aviator

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/YgEv7IQzGdM-kill-code-review) · [Video](https://www.youtube.com/watch?v=YgEv7IQzGdM)

## How to Kill the Code Review Without Losing Team Alignment

Replacing line-by-line review requires more than an AI reviewer: preserve implementation decisions, turn recurring feedback into checks, and review evidence against intent.

Familiarity with pull requests, automated tests, and AI coding assistants will help you follow the workflow.

### What would make line-by-line review unnecessary?

[0:16](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=16s)

What evidence would let a team merge a change without reading every line? Ankit Jain, Aviator’s co-founder, approached that question in [How to Kill the Code Review](https://www.latent.space/p/reviews-dead), proposing five layers of trust around generated code. Building an AI code verification platform has led him to revisit that proposal: the challenge is not only establishing that code works, but preserving what teammates learn and decide through review.

The immediate pressure is growing code volume. Jain describes a bottleneck moving from writing code to reviewing it, citing an 861% increase in churn, rising incidents per pull request, and longer review times. Churn needs a precise reading here: the closely matching [Faros analysis](https://www.faros.ai/blog/tokenmaxxing) defines it as quarterly deleted lines divided by added lines. An increase in that ratio is not an equivalent increase in useful output.

Jain describes review waiting time as four times its previous level and says over 30% of changes merge without review. The matching [Faros figures](https://www.faros.ai/blog/why-cheaper-ai-models-can-cost-more-hidden-model-tax) use different denominators: median time in review increased 441.5%, and 31.3% more PRs merged without review. Those are relative changes between organizations’ lowest and highest AI-adoption periods, not the share of all PRs lacking review; time in review is also not necessarily time waiting for a first reviewer. The findings are observational correlations.

![Slide with four cards showing +861% code churn, +243% incidents-to-PR ratio, +441% median time in review, and +31.3% of PRs merging with no review.](https://ai.engineer/api/talk-essay-media/c396d69f7ac163b60c7eeb0ad4953ea1a2c72456bfe6c483e544be24c0ca493a)

“We've already stopped reviewing it”: four indicators of code-review strain.

Now consider the everyday GitHub workflow: two or three AI reviewers leave comments, a coding agent or developer receives them, and another round of changes resolves the feedback. The human arrives at the end, assumes the agents have found most problems, skims the diff, and merges. If nobody meaningfully reads the exchange, why organize it around a human-facing review interface? Adding automated reviewers has preserved the ceremony without necessarily preserving human judgment.

### The part of review that must survive

[3:00](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=180s)

Today’s review interface is not an immutable requirement of software engineering. Jain points to Google’s Mondrian in 2006 and early Windows development to illustrate how practices have changed. That history should be read narrowly as a discussion of modern tool-mediated review: formal inspections go back at least to Fagan’s work in 1976, as documented in [Modern Code Review: A Case Study at Google](https://storage.googleapis.com/gweb-research2023-media/pubtools/4476.pdf). The specific Mondrian date and Windows comparison are Jain’s historical examples.

Review performs two different jobs:

| Function | What the team needs |
|---|---|
| Semantic accuracy | Find bugs, enforce conventions, identify security issues |
| Alignment | Share knowledge, mentor, discuss architecture, onboard teammates |

The second job was missing from Jain’s original trust model. A reviewer explaining a design choice or helping a junior engineer understand a service boundary is doing more than checking correctness.

![Two panels contrast semantic accuracy tasks with alignment activities: knowledge sharing, mentorship, architecture feedback, onboarding, shared ownership, and API design.](https://ai.engineer/api/talk-essay-media/3a2f0c3b4bbde55914dbd285fb8f5ea94a7a9dc607526ff6b0c824c20c9c39ff)

Code review serves both semantic accuracy and team alignment.

This distinction matters most for teams collaborating on a shared system. A solo project or a fully unattended development operation has different coordination needs. For a team, **alignment must survive even as correctness checks become more automated**. The design question is whether shared understanding can come from something more useful than reading a diff line by line.

### A specification cannot contain decisions nobody has made yet

[4:34](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=274s)

Spec-driven development offers an appealing sequence: write the specification, give it to an agent, generate the implementation, then verify it. Jain compares that sequence to the familiar waterfall model associated with 1970. The criticism applies specifically to a frozen, one-pass workflow; [Royce’s original paper](https://cse.msu.edu/~cse435/Homework/HW3/royce1970.pdf) already described feedback between phases and warned about the simple sequential approach.

The missing feedback becomes visible during an ordinary Claude Code, Codex, or Cursor session. The developer interacts with the agent because the specification leaves questions unanswered. Implementation then exposes additional problems. If the specification is treated as finished, those discoveries never make it back into the document that supposedly defines the work. An LLM also makes implementation choices rather than producing a deterministic expansion of the specification. Spec-driven development remains useful, but its most valuable contribution is **intent**, not an assumption that all decisions can be completed up front.

Intent is distributed across several artifacts:

| Artifact | What it contributes |
|---|---|
| Jira ticket | The goal |
| Product requirements document | The detailed plan |
| Prompts and agent conversation | Decisions made during implementation |

The conversation is especially valuable because it records the developer’s answers as ambiguities emerge. Creating a pull request and discarding that conversation throws away part of the explanation for why the code exists in its current form. Preserve those decisions with the change.

### Turn repeated comments into reusable checks

[6:32](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=392s)

Preserving intent does not eliminate bugs. LLMs make mistakes, and AI reviewers can miss them. Jain still expects some manual review; the opportunity is to stop spending that human attention on the same findings repeatedly. His **AI slop registry**, also called a register, collects recurring review issues and turns them into guardrails that can be checked automatically.

Each repeated finding becomes an input to the verification system rather than another comment somebody must remember to write. As the registry grows, it incorporates the team’s review experience. Jain describes this as learning or training on top of a standard LLM, but does not specify model-weight updates. The concrete mechanism is codifying recurring feedback so the system can identify it again.

### Make intent and guardrails produce a test plan

[7:58](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=478s)

Alignment and semantic accuracy meet in a single verification loop:

1. Capture the developer’s responses from the coding session.
2. Turn those decisions into acceptance criteria.
3. Combine the criteria with the maintained AI slop register to create a test plan.
4. Start a preview environment and verify the plan end to end.
5. Review the intended capability alongside the observed behavior.

The acceptance criteria describe what this change should accomplish. The registry contributes constraints learned across changes. Together, they give verification a target beyond whether the implementation looks plausible.

**The review surface becomes intent plus evidence.** Did the implemented capability match what the developer meant to build? Did its behavior satisfy the acceptance criteria? Architectural disagreements still belong in the review; they now sit beside explicit requirements and observations rather than being inferred entirely from the diff.

### Generate the tests, review the plan

[9:30](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=570s)

In Aviator’s walkthrough, even a simple task involves clarification questions and user feedback. Those exchanges contain engineering decisions worth retaining: they explain the implementation to teammates and give junior engineers examples of how to reason through a change. An LLM can extract those decisions into acceptance criteria and help create the test plan, reducing the effort of translating a conversation into something verifiable.

Criteria plus invariants then drive checks in a preview environment. Jain proposes generating tests in real time for new features, with the aim of removing much of the recurring test-writing and maintenance burden. Human governance shifts toward reviewing whether the plan asks the right questions. The walkthrough slide makes that surface concrete: a checklist sits beside pass/fail results, including a red failure for `Threshold typed as Money`. A failed check is visible evidence to investigate, not a successful outcome to infer from plausible code.

![Slide titled “Criteria + invariants → a test plan that runs,” with a test-plan checklist beside results showing green passes and a red failure for “Threshold typed as Money.”](https://ai.engineer/api/talk-essay-media/3b8cbe3be8c5b309fc8ef2a92701318dedb556d9d6da0932433d7364157481c4)

Criteria and invariants become a test plan with visible verification results.

This resembles behavior-driven development more than simply moving traditional test-driven development into an agent. An English-language test plan lets product managers and designers participate in deciding what correct behavior means. Where criteria admit deterministic checks, the verification system can then establish whether each one was met. Human agreement about the requirement and machine checking of the result play different roles.

### Verify a payment form through observable behavior

[11:44](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=704s)

Not every part of verification can be deterministic. Jain’s rule is “deterministic where it can be, but LLM where you must.” Use precise checks where possible and agent judgment where the task requires it, without treating the whole system as perfect.

His example is a newly introduced payment form. A test plan describes the behavioral change; an AI agent browses the application, fills out the form, and captures screenshots. Those screenshots, together with database snapshots, provide evidence for deciding whether the acceptance criteria were met. The browser view shows what the user encountered, while the stored state helps establish what the application actually did.

For a concrete implementation of that evidence check, suppose the captured requirement says a successful payment must create a `paid` record for order `order-42`, for 2,500 cents in USD. The following TypeScript checks that proposed criterion against before-and-after snapshots; screenshot evidence remains a separate part of the review:

```typescript
type Payment = {
  id: string;
  orderId: string;
  amountCents: number;
  currency: string;
  status: string;
};

function verifyPayment(
  before: readonly Payment[],
  after: readonly Payment[],
): boolean {
  const previousIds = new Set(before.map(payment => payment.id));
  const added = after.filter(payment => !previousIds.has(payment.id));
  const orderPayments = added.filter(
    payment => payment.orderId === "order-42",
  );

  return orderPayments.length === 1 && orderPayments.every(payment =>
    payment.amountCents === 2500 &&
    payment.currency === "USD" &&
    payment.status === "paid"
  );
}
```

The agent’s navigation can be flexible while this particular check remains exact. Screenshots and snapshot checks give the reviewer inspectable evidence; agent-based screenshot assessment can still require judgment.

### Derive the checks from decisions, not just the implementation

[13:08](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=788s)

The session should retain what was requested, what was attempted, and what was rejected. That origin matters. If an agent writes code and then derives the test plan only from that code, it can reproduce its own mistaken assumptions in the checks. Jain connects this warning to a point he recalls Dex making the previous day: tests that merely describe the implementation can miss the gap between implementation and intent. Session-derived criteria provide a separate statement of what was wanted.

The reviewer can then discuss data models, service interactions, and architectural choices at the level where collaboration is most valuable. Those decisions remain open to challenge, and the collected verification evidence shows what happened when the proposed behavior was exercised. Moving beyond line-by-line review does not remove technical discussion; it gives that discussion a different set of artifacts.

### Build the register before retiring the ritual

[14:12](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=852s)

Jain’s starting assignment is concrete: mine the last 1,000 review comments for repeatable issues. Turn the recurring best practices into entries in the AI slop register. Each merged PR can then contribute another reusable check, reducing the need to write the same feedback again while preserving the collaborative part of review.

That investment has an initial cost. Jain describes a J-curve: building the registry takes time and creates pain before the accumulated checks pay off. The adoption slide explicitly cautions against killing review on day one. His invitation to try Aviator’s approach sits within that gradual transition: improve tooling for semantic accuracy while retaining the alignment that makes review valuable to a team.

![Slide titled “Build the register. Don't kill review on day one.” Text recommends mining the last 1000 review comments, beside a curve dipping below baseline before rising toward payoff.](https://ai.engineer/api/talk-essay-media/35d3a29cb888dcda185f5c9e45981c87466d32be05947e04481644c773bce756)

Build an AI slop register from recurring review comments, allowing for an initial dip before payoff.

At the time of the talk, [Aviator Verify](https://www.aviator.co/verify) was being piloted with a few companies and seeking early design partners. Its proposed combination is the same one the workflow requires: captured intent for alignment, and a growing register of reusable checks for semantic accuracy. The goal is to make a merge decision rest on shared decisions and verified behavior, with human attention still available for the questions the checks do not settle.

## Resources

- [How to Kill the Code Review](https://www.latent.space/p/reviews-dead) — Jain's original five-layer proposal for replacing line-by-line review with intent, constraints, and verification.
- [Aviator Verify](https://www.aviator.co/verify) — Official product overview covering intent capture, acceptance criteria, verification evidence, and access requests.
- [Faros: AI activity versus engineering outcomes](https://www.faros.ai/blog/tokenmaxxing) — First-party explanation of code churn and review metrics from telemetry across 22,000 developers and 4,000 teams.
- [Modern Code Review: A Case Study at Google](https://storage.googleapis.com/gweb-research2023-media/pubtools/4476.pdf) — Research on Google's review practices, with historical context distinguishing formal inspections from modern code review.
- [Managing the Development of Large Software Systems](https://cse.msu.edu/~cse435/Homework/HW3/royce1970.pdf) — Royce's August 1970 paper on development stages, feedback, and the risks of discovering problems only during final testing.

## Slides

### How to Kill the Code Review

Ankit Jain introduces his talk on code review, drawing on ideas from Aviator’s AI code verification platform.

[0:19](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=19s) [Title slide for Ankit Jain's talk, How to Kill the Code Review.](https://assets.ai.engineer/sites/aiecode2025/sha256/fbd494881f2cdbdfaa631d561567fb5a288050c68543daf0c8007dc5c62694a3.webp)

How to Kill the Code Review

The math broke. The gate broke. The theater's over.

Ankit Jain | aviator

Interface text:
NOTES
1 / 19

### The 5-layers of trust

Ankit Jain proposes five layers of checks to build enough trust in code to merge it without line-by-line review. The diagram passes specs and agents through each layer, from comparing options to adversarial verification.

[0:38](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=38s) [A five-layer diagram shows multiple options, deterministic guardrails, acceptance criteria, permission systems and adversarial verification.](https://assets.ai.engineer/sites/aiecode2025/sha256/ce08bee5cf37f29f90af22aba486850951449d3e1624ca1a547ac23aef31a7c9.webp)

The 5-layers of trust

How to Kill the Code Review
Code must not be written by humans -> Code must not be reviewed by humans

Specs + Agents
Compare multiple options
Deterministic Guardrails
Acceptance Criteria
Permission Systems
Adversarial Verification

Building trust through layers

latent.space/p/reviews-dead

Interface text:
NOTES
2 / 19

### The problem

Growing code volume is making it harder for teams to keep up, raising the question of how long they can continue reading code line by line.

[1:19](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=79s) [Part 01 introduces the state of code review today.](https://assets.ai.engineer/sites/aiecode2025/sha256/31d5feefd2130b0ae37d278b740a3bc247715b181ac73690d889f0797dd39f30.webp)

The problem

PART 01
State of code review today.

01

Interface text:
NOTES
3 / 19

### The review theater

An agent and AI reviewer iterate without a human in the loop; a human then skims and merges. The slide questions why this exchange needs a UI.

[2:14](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=134s) [A workflow shows an agent writing code, AI reviewing and revising it, a human skimming, and a merge.](https://assets.ai.engineer/sites/aiecode2025/sha256/488622a8c5bd2e314cb8d418b81626cc8170e25d8538139a584fb6443639e495.webp)

THE REVIEW THEATER

AI writes the code. AI reviews the code. Why is there a UI?

Agent writes ↔ AI reviews & revises → Human skims → Merge
↻ iterates ×N — no human in the loop

When AI reviews AI in a UI nobody reads, you’ve configured the wrong thing.

Interface text:
NOTES
5 / 19

### Code review is not about code review

Better tools can improve semantic accuracy; teams still need review for alignment through knowledge sharing, mentorship, and architecture feedback.

[2:57](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=177s) [The semantic accuracy panel lists catching bugs, conventions, security, regressions, naming and edge cases.](https://assets.ai.engineer/sites/aiecode2025/sha256/1c3bb6db0e78bf7202ac1292584f6d4aeb5d3c645ff4e9cade149016190fa32c.webp)

THE PIVOT
Code review is not about code review.

SEMANTIC ACCURACY
Catch bugs
Conventions
Security
Spot regressions
Naming
Edge cases

Interface text:
NOTES
6 / 19

[4:12](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=252s) [Code review separates semantic accuracy from alignment, including knowledge sharing and mentorship.](https://assets.ai.engineer/sites/aiecode2025/sha256/7e24984bb3cd2410ede4f7bf487b1b050c6f37eb2f2bbc37e443f8dc1e4d3f88.webp)

THE PIVOT
Code review is not about code review.

SEMANTIC ACCURACY
Catch bugs
Conventions
Security
Spot regressions
Naming
Edge cases

ALIGNMENT
Knowledge sharing
Mentorship
Architecture feedback
Onboarding
Shared ownership
API design

Semantic accuracy needs better tools. But alignment must survive!

Interface text:
NOTES
6 / 19

### Spec-driven development is waterfall in new clothes

The speaker compares spec-driven development to waterfall: both proceed from a plan to implementation and verification. He argues that decisions uncovered during implementation rarely make it back into the spec.

[5:19](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=319s) [Waterfall and spec-driven development are compared, followed by points about missing feedback and drifting specs.](https://assets.ai.engineer/sites/aiecode2025/sha256/afbd7a45cfabd7d2e515c945cecfcfeda3a0c5fa33942f35edd9d2dfca208cf9.webp)

THE POPULAR ANSWER
Spec-driven development is waterfall in new clothes.

WATERFALL · 1970
Requirements → Design → Implementation → Verification

≡

SPEC-DRIVEN · 2026
Spec → Architecture → Generation → Verification

• No feedback loop. The spec is written before the work has taught you anything.
• Specs drift. Implementation surfaces decisions the spec never anticipated — and they never make it back.

Interface text:
NOTES
8 / 19

### Intent doesn't only live in a spec

Jira tickets capture the goal and design docs the plan. The speaker argues that prompts capture decisions made while working with an agent, yet are discarded after the pull request.

[6:06](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=366s) [Cards describe intent in Jira tickets, design documents, and prompts used during coding sessions.](https://assets.ai.engineer/sites/aiecode2025/sha256/488cee34d7be62c5606800098c61f3fbd7391d5d0ee713a116692f7bc9f19e35.webp)

A BETTER ANSWER · INTENT-DRIVEN

Intent doesn't only live in a spec.

JIRA ticket
The goal. Coarse, written before anyone touched the code.

PRDs / design docs
The plan. Already drifting by the first commit.

Your prompts
Where the real decisions get made — live in the session, then thrown away when it ends.

Interface text:
NOTES
9 / 19

### Semantic accuracy

The talk turns to bugs that can remain in code even when intent is aligned, and notes that AI reviewers are imperfect.

[6:31](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=391s) [Section title: Semantic accuracy. Make the machine prove the code.](https://assets.ai.engineer/sites/aiecode2025/sha256/8239b702e7daff792915bae4050568abd458306c6deb5453e67fe7767591c9fe.webp)

PART 03 · JOB TWO
Semantic accuracy
Make the machine prove the code.

03

Interface text:
NOTES
10 / 19

### The AI Slop Register

A recurring request to use Money becomes a rule rejecting float fields for amounts, prices and totals. The slide proposes filling a review queue weekly from merged-PR comments, with approval before rules go live.

[7:19](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=439s) [A register of recurring review rules, a codified comment example, and an explanation of the review queue.](https://assets.ai.engineer/sites/aiecode2025/sha256/0e91a76e281e870df70c24b960ec504dceb6e12bce4037fa5f85d3d64cfbef1d.webp)

CODIFY THE COMMENT
The AI Slop Register.

Active 6    Review 3
Use the Money type for currency amounts — TYPES
Use the structured logger, not print — OBSERVABILITY
No direct writes to user_* tables — DATA-ACCESS
Error paths increment a metrics counter — OBSERVABILITY
No hardcoded secrets — SECURITY

CODIFY THE COMMENT
# said on the last 100 PRs
"use Money for amounts, not float"
            ↓ codify once
invariant("no-float-currency"):
  reject  FloatField(/amount|price|total/)
  message "Use Money, not float"

IT BUILDS ITSELF
The Review tab fills weekly from your merged-PR comments.
You approve before they go live.

Every recurring comment is a guardrail you haven't written yet.

Interface text:
app.aviator.co/verify/invariants
aviator  Verify · Invariants
NOTES
11 / 19

### The system

This section brings alignment and semantic accuracy together in a single review loop.

[7:55](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=475s) [Part 04: The system. Putting both jobs together.](https://assets.ai.engineer/sites/aiecode2025/sha256/e71f35dbf8bdbbd821c336b09594d80b746c0df7f2bfb41d41e1b2a84ea5ae44.webp)

PART 04
The system
Putting both jobs together.

04

Interface text:
NOTES
12 / 19

### Two halves, one loop

Session decisions become acceptance criteria. Those criteria and standing invariants shape a test plan; results from the preview environment give the reviewer evidence to judge intent, architecture and behavior.

[9:19](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=559s) [A four-stage flow from session and acceptance criteria through a test plan and results to reviewer judgment.](https://assets.ai.engineer/sites/aiecode2025/sha256/0573d0b52f5ee3a0098847bc9a100ec5f58ccdc0af683e9817bc75eb9be96a04.webp)

HOW IT COMES TOGETHER
Two halves, one loop.

1 · CAPTURE
Session
Your prompts
↓
MCP
Acceptance criteria
reviewer signs off

2 · BUILD
AI Slop Register
standing invariants
↓
Test plan
criteria + invariants

3 · RUN
Preview env
runs scenarios
↓
Results
pass / fail + evidence

4 · REVIEW
Reviewer
intent · architecture · verdicts
not a diff

Interface text:
NOTES
13 / 19

### The session becomes the criteria

A rate-limit session records decisions about the existing RateLimiter and authentication. Those decisions appear as acceptance criteria for a reviewer to approve before the code is judged.

[10:09](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=609s) [A coding session’s decisions become acceptance criteria for a rate-limit change.](https://assets.ai.engineer/sites/aiecode2025/sha256/19336917bd1f0e7d40f801acc672dd1e3cc5e49b80b8960b45ba7a6066de6c0d.webp)

WALKTHROUGH · CAPTURE
The session becomes the criteria.

> Add a rate limit to the exports endpoint.
● Drafting a token-bucket limiter...
  └ Update(exports/service.py) · +34
> Use the existing RateLimiter util.  DECISION
● Refactored. Added an auth check.
> Skip the auth — middleware handles /api/v2.  DECISION
> Use the Money type, not float.  DECISION

Acceptance Criteria  4
1  Returns 429 above 100 req/min  GENERATED
2  Uses the existing RateLimiter util  SESSION
3  No new auth on /api/v2  SESSION
4  Threshold typed as Money  MANUAL

Reviewer signs off on intent — before any code is judged
Agree

Interface text:
Claude Code
~/exports-service
app.aviator.co/r/1287/verify
aviator  Verify · r/1287
NOTES
14 / 19

### Criteria and invariants form a test plan

The example combines a runtime rate-limit check with code scans. The verification run reports four passes and one failure: the threshold was declared as float instead of Money.

[10:35](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=635s) [A test plan appears beside verification results showing four passes and one failure.](https://assets.ai.engineer/sites/aiecode2025/sha256/98e6600d7214909fd1d58ecf233659eaeace06918f7bd75e74225ef3a5d9857f.webp)

WALKTHROUGH · VERIFY
Criteria + invariants → a test plan that runs.

Test plan
Fire 101 req/min → expect 429  RUNTIME
Calls into RateLimiter util  CODE SCAN
No auth decorator on /api/v2  CODE SCAN
threshold field type = Money (invariant)  CODE SCAN
No hardcoded secrets (invariant)  CODE SCAN

PR #1287 · a3f9c2 · preview env · 2:31 PM
passed: 4  failed: 1  total: 5
✓ Returns 429 above 100 req/min
screenshot · console
✓ Uses RateLimiter util
× Threshold typed as Money
declared float · exports/service.py:142

Interface text:
app.aviator.co/r/1287/verify/run-8f2/plan
aviator  Verify · Test plan
app.aviator.co/r/1287/verify/run-8f2
aviator  Verify · Run 8f2
NOTES
15 / 19

### Deterministic where you can, LLM where you must

Reproducible checks handle clear criteria; an LLM helps with fuzzy judgment inside the harness. The speaker describes an agent using screenshots and database snapshots to assess a payment-form change.

[11:45](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=705s) [Two panels compare reproducible deterministic checks with LLM judgment for fuzzy behavior.](https://assets.ai.engineer/sites/aiecode2025/sha256/198eabc0d8d1de805b28fe83869e59c7c27218227c44861ebd98e58a5823df09.webp)

THE BALANCE

Deterministic where you can. LLM where you must.

Deterministic check
Scan · assertion · status code · screenshots. Reproducible — same code, same result, every time.

Runtime + LLM judge
Screenshots · fuzzy behavior. For the criteria that are genuinely judgment calls.

The model is a tool inside the harness — never the judge of record.

Interface text:
NOTES
16 / 19

### Reviewers review intent

Reviewers move above line-by-line diffs to examine intent and decisions, discuss architectural trade-offs, and inspect verification evidence for whether the behavior occurred.

[13:57](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=837s) [Three panels describe reviewing intent and decisions, architecture, and verification evidence.](https://assets.ai.engineer/sites/aiecode2025/sha256/2f95f41498fb2623a6e3e561ce0e6daf0df55362d95e52c25dc97a44bbc8d385.webp)

WHAT REVIEW BECOMES

Reviewers review intent. Not diffs.

INTENT & DECISIONS
• What we set out to build
• What we tried and rejected
• Captured from the session

ARCHITECTURE
• The shape of the change
• Trade-offs worth a human
• The conversation that stays

VERIFICATION RESULTS
• Verdicts + evidence
• Deterministic, reproducible
• Did the behavior happen?

The reviewer is more important than ever. They're just not reading diffs anymore.

Interface text:
NOTES
17 / 19

### Build the register

Mine the last 1,000 review comments for repeatable patterns, then add invariants with each merged PR so later reviews catch more. The chart sketches a month 1 dip before payoff.

[14:11](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=851s) [Homework slide recommends mining review comments for repeatable patterns and shows an initial dip followed by payoff.](https://assets.ai.engineer/sites/aiecode2025/sha256/6a116faabc9afa992402ca68e8928d6c7566fd61104b5bf9ca7ca2996af7ff3d.webp)

HOMEWORK

Build the register. Don't kill review on day one.

Mine your last 1000 review comments and build an AI slop register for the repeatable patterns.

It compounds: every merged PR → proposes new invariants → the next PR catches more → reviews shrink. The feedback loop tightens itself.

Chart labels: payoff; baseline; month 1 dip; time →

Pain is real. Payoff is real too.

Interface text:
NOTES
18 / 19

### Code review is not about code review

Review aligns the diff with intent captured from the session. An AI slop register codifies repeated comments into checks that run on every PR.

[15:11](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=911s) [Closing slide connects review with intent and recommends codifying repeated comments as checks for every pull request.](https://assets.ai.engineer/sites/aiecode2025/sha256/325dcd41b0e7daf9d18d5fe7293802ac4f49e0c77f2afe2b1200cb32843bd4b7.webp)

IF YOU REMEMBER ONE THING

Code review is not about code review.

ALIGNMENT
diff → intent
Review what we meant — captured from the session, not the diff.

SEMANTIC ACCURACY
the AI Slop Register
Every repeated comment, codified into a check that runs on every PR.

aviator.co/verify
[redacted] · [redacted]

Interface text:
NOTES
19 / 19

## Full transcript

[0:00](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=0s) [upbeat music] Okay. Ooh, hello. Hey, everyone.

[0:16](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=16s) Thanks for joining in. Today, uh, we will be talking about how to kill the code reviews, everyone's favorite topic. I'm Ankit, co-founder of Aviator. At Aviator, we are building AI code verification platform.

[0:29](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=29s) So we'll bring in some of the ideas and concepts that we talk about in the, uh, that we build in our product. Uh, but first, let's dive into a little bit.

[0:40](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=40s) So a few months ago, I wrote a post on Latent.Space about, uh, how to kill code review, creating a framework, a five-layers trust model. So this model was focused around how do we actually, layer by layer, build trust into the code that can then be merged without needing line by line review.

[1:03](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=63s) And I got some things right, and I got some things wrong. So this talk will be about just really diving a bit more, more into it. I'm not gonna talk about specific layers, but we will talk about some of the concepts that emerge from this session.

[1:17](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=77s) So let's just talk about the problem. We are looking today at the volume of code is increasing every day, and we are struggling to keep up. So when we think about, like, how long will it take us to actually stop reading code line by line,

[1:35](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=95s) and the reality is we have already stopped reviewing it. There is eight hundred and sixty-one percent code churn. That means we are, like, producing more code. The incidents to PR ratio is increasing.

[1:46](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=106s) That means even if you're doing reviews today, they're not effective. So the median time of review is increasing. We have just increased the bottleneck to-- from the coding, now kind of like coding is solved, to now reviewing, where everything just gets stuck there.

[2:03](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=123s) You're spending four x the time that you were spending before just waiting for the reviews. And today, over thirty percent of changes are actually getting merged without a review at all.

[2:14](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=134s) So let's just think about it a little bit. Think about AI reviews. So everyone is probably using some form of AI reviews today. When AI write the code and AI review the code, why are we doing it in a UI, right?

[2:28](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=148s) Like, we open a GitHub, there's kind of like, you know, maybe two or three AI coding agents who are, like, doing the reviews. You know, you-- the review goes back-- passed to the user, uh, or the agent, and it gets resolved, and then kind of like you're doing back and forth with the agent.

[2:42](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=162s) Where is human here in the loop? Like, you're eventually just looking at, okay, you know, if AI has reviewed it, most of the things probably we found. Let's just kind of like, you know, do a skimming of it and merge it.

[2:53](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=173s) So when AI reviews and nobody reads, we have configured the wrong thing.

[3:00](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=180s) So let's just take a step back. Code reviews are not very old. They are maybe fifteen to twenty years ago. In two thousand and six was when Google launched Mondrian internally, and they made formal code review as a thing.

[3:13](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=193s) If you think about our Windows back in the days, the first versions were actually built without reviews. But if we look carefully, code review is not just about code reviews.

[3:24](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=204s) You know, obviously, we are looking at it like catching bugs, understanding kind of conventions, identifying security issues. But code reviews is also about alignment. And this was one piece which was missing from my five layers model that I talked about a few months ago.

[3:40](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=220s) So a big part of code reviews is knowledge sharing, mentorship, architectural feedback, onboarding, being able to collaborate. This is, again, if you're doing, um, vibe coding, you're working on a solo project, this is not a talk for you.

[3:56](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=236s) If you are working in teams, which I believe most of you folks are, collaborating in teams, you're not likely using completely dark factories, orchestrators, where nobody looks at the code.

[4:05](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=245s) You're actually collaborating in teams. You need to do knowledge sharing, which is the alignment part. And that is the part which is the most important aspect of reviews. So for semantic accuracy, we can build better tooling, but alignment must survive.

[4:21](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=261s) So let's just kind of like dive into alignment. What does it mean? Like, in today's world, can we actually think of better model than aligning just based on reading line by line code?

[4:34](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=274s) So most folks have probably heard about spec-driven development by now. So spec-driven development is like, okay, we write a spec. You know, um, it covers all the details. We pass it to an agent.

[4:45](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=285s) It generates the code, and then we verify. So what's wrong here? If you look back in nineteen seventy, this is what waterfall model was. You know, you have requirements, you have specification, you implement and then you verify.

[4:57](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=297s) But there's no feedback loop. You know? The spec is written before we identified everything else, right? That's why today everyone still wants to use your coding sessions, whether it's Claude Code, Codex, Cursor, whatever you're using, you want to interact with the agents.

[5:13](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=313s) And the reason you're interacting with agents is because there were certain things which are not clear in the spec, and we still need to capture that. And second is, as you implement, you identify more issues, and you never go back and update the spec because, like, if you're doing a spec-driven development, it's already done.

[5:29](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=329s) Once the spec is done, you expect, like, you know, the code will come deterministically. But guess what? LLM is not deterministic. It's gonna make decisions itself. So that's why spec-driven development is a great, uh, methodology, but it falls short in day-to-day software development.

[5:45](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=345s) But there are some interesting aspects of this which we should carry forward. The most important part is the intent.

[5:53](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=353s) And intent doesn't only live in the spec.

[5:56](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=356s) Intent live in your Jira ticket. That's the goal, right? Like, it's kind of like where you express what we want to do. It lives in your PRDs. It's kind of like details, like it's a plan.

[6:07](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=367s) But most importantly, it lives in your prompts today. This is where the real decisions are being made. You start with like, okay, this is a Jira ticket I'm gonna look at, but you're going back and forth with the agent, and this is where all the user decisions are being made.

[6:21](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=381s) But what we do today is we create a change, we cr-create a pull request, and then we throw away the prompts. And this is one of the things that we need to change.

[6:32](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=392s) Let's just first talk also about semantic accuracy because like, you know, you're saying, "Hey, Ankit, okay, I understand the alignment part, but like there are still bugs in the code.

[6:39](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=399s) Who's gonna look at that?" LLMs are also not great at this, and we already talked about how AI agents may-- reviewers may not also be perfect.

[6:49](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=409s) So this is where I introduce you to a concept of AI slop registry. So think about this. We are reviewing-- If you're reviewing code today manually, and I expect everyone should be doing some degree of this,

[7:02](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=422s) we are essentially possibly identifying the same issues over and over again. Can we actually capture these concepts and codify them so that we don't have to always create those review feedback one by one?

[7:18](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=438s) You actually also have all of those things automatically identified. The beauty of this is if you do it a few times, you now build sort of like a system which actually learns over time.

[7:29](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=449s) So think of this as sort of like you're not-- you know, you're doing more training on top of the standard LLM that you have actually extracted, uh, like built on top of.

[7:38](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=458s) So AI slop registers now can create better results because it's trained, it's learning from the review experience that you as humans are providing.

[7:52](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=472s) Every recurring comment is now a guardrail that you don't have to review again.

[7:58](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=478s) Okay. So let's kind of like try to put both of these alignment and the semantic accuracy together.

[8:05](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=485s) It is two halves of the same problem. We are trying to id-- understand what are the core mechanics of review, how do we actually break it down into two parts, which is the alignment and the semantic accuracy, and bring them together into a single loop.

[8:19](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=499s) So first thing is you go take your session and you capture the, uh, user responses, and that essentially forms your acceptance criteria. The acceptance criteria then tied with your AI slop register that you are now con-constantly maintaining finally creates a test plan.

[8:38](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=518s) And this is a test plan which then gets verified. This is part of the system that we are building is the verification system, where it spins up a preview, takes your test plan, and make sure it actually works end-to-end.

[8:51](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=531s) Even if the code looks right, does it actually work? So this verification part's become interesting. And now the kicker is this is now your review surface. You're not reviewing code line by line, but rather you're looking at the evidence of what was the intent, did the user actually implement the capability that was defined in their intent, and

[9:15](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=555s) did the behavior actually meet the requirements that we had in the accept-- as acceptance criteria. So you're still having the architectural decisions, you're still having these arguments, but the review surface changes.

[9:30](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=570s) So just giving a walkthrough of like how we have built our system. The session becomes a criteria, so all these decisions that you are doing here with the agent, you're asking, uh, you know, you're providing this feedback.

[9:43](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=583s) Like even if there is a simple task, many times you're going back and forth. Agent will talk to ask questions. These are the decisions that we need to capture.

[9:52](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=592s) This is the intent. This is what makes your review, makes your collaboration more valuable. This is how you teach your junior engineers on how to improve over time. These are the decisions which make a software engineer valuable today.

[10:11](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=611s) We convert those into an acceptance criteria. This is where you can also leverage LLM to do so. So you capture these user int-- uh, decisions and make sure you can actually create a test plan based on this.

[10:22](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=622s) I know test plan creation is always painful, which is where I would always recommend people to use LLM for this purpose.

[10:30](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=630s) And finally, the criteria plus invariant is what makes the test plan, uh, and we build the verification systems to actually capture the test plan, run your previews, and be able to test based on this test plan.

[10:44](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=644s) So even imagine if you're building a new feature, you don't have to maintain tests at all. This is creating test in real time, and this is where you can leverage the power of LLM because the test plan maintenance and creation can be very painful.

[11:00](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=660s) But the value of human in the loop here is the governance and the review part, and the part where you're reviewing the test plan and not the code, right?

[11:14](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=674s) I know it's been like over twenty years we came up with test-driven development. This, in some ways, is closer to behavior dev-de-driven development, where the test plan is now something which even you can share with your product managers, your designers.

[11:28](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=688s) Everyone can participate because these things are now in English. At the same time, we have now deterministic verification which actually verifies whether the particular test criteria has been met or not.

[11:44](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=704s) Let's move on. So this is where I would say the system is not supposed to be perfect. It's deterministic where it can be, but LLM where you must. Not everything can be

[12:02](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=722s) deterministic. Not every system can be built in a way which is hundred percent built on deterministic, uh, systems. This is where you use LLM as a fallback. Let me give you an example.

[12:17](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=737s) Like, if you're making a change in your web application, what you can do is it creates a test plan of what the behavior changes. Let's say you introduce a new payment, uh, form.

[12:28](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=748s) So it creates a new payment form. The verification here is, does the payment system changes? Uh, an AI agent can go and browse through your application to fill out a form and capture screenshots as evidence, and then take those screenshots as well as your database snapshots to identify whether the criteria was met.

[12:52](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=772s) So the screenshot testing or, like, sampling can then still be done by agents, but at the same time, you're creating more solid evidence which now a reviewer can look at and build more confidence that this actually works.

[13:08](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=788s) So now reviewers are reviewing the intent, not the diff. You're reviewing the intent decisions, uh, what we said to build out, what we tried and rejected, and capture all of these things from the sessions.

[13:22](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=802s) Remember, capturing it from the sessions is the key. If we try to build it from the code, you end up in the same situation that we were talking about, I think Dex was talking about yesterday, which is if your code is built by the same agent which is actually building a test plan, it's not going to build

[13:40](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=820s) a test plan which will actually catch issues. So that's why it's important to actually use the session information to build out a test plan. You can discuss architectural decisions, so how you're creating the data models, how these services interact with each other.

[13:54](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=834s) So you're moved one level above. So you're essentially, instead of, like, reviewing line by line, you're actually having discussions on architecture, which are very critical for any kind of collaboration.

[14:05](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=845s) And then you look at the evidence, everything that was collected from the verification.

[14:12](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=852s) So here's a homework for everyone. Go home and mine your last one thousand review comments and build out a AI slop register for the things which are repeatable. So a vast majority of the comments that you're providing in your code review are something that we repeat over and over again.

[14:34](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=874s) This compounds with every merged PR. Every time you capture something as a register, you don't have to capture that comment again. And this is where you can actually codify some of the best practices of doing code, maintain semantic accuracy.

[14:51](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=891s) At the same time, do not lose the collaboration part of the review.

[14:58](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=898s) It does follow a J curve, so pain is real. Uh, you will have to spend some time to actually make it pay off because initially creating a registry can take some time, and this is where I would recommend you folks can come and try out our product.

[15:13](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=913s) So code review is not just about code review, it is about, uh, really getting the alignment, and where we can build better tools is creating the semantic accuracy and defining your AI slop register.

[15:29](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=929s) So if you remember one thing from today, remember code review is not just about code review, it is about getting the alignment.

[15:38](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=938s) And yes, we are, uh, piloting our new product called Verify. Uh, please, uh, join and, uh, be our early design partners. We are working with a few companies to, uh, pilot out a new verification system.

[15:52](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=952s) This combines both the alignment side of things as well as building tools and capabilities for detecting semantic accuracy using the AI slop register.

[16:04](https://www.youtube.com/watch?v=YgEv7IQzGdM&t=964s) Thank you, everyone. Thanks for joining. [audience applauding] [outro jingle]
