The Death of the Code Review: What the Data Actually Says — Laurie Voss, Arize AI

Code can arrive much faster than a team can decide whether to trust it. Laurie Voss, Head of Developer Relations at Arize AI and co-founder of npm Inc., opens with a study tracking more than 100,000 GitHub developers and matching their activity against telemetry showing when they began using AI. Developers who enabled autonomous agents wrote 741% more code, while shipped software increased by only 30%. In Voss’s account, the study identifies review as the bottleneck between those outcomes.

Generating a change and accepting responsibility for it consume different resources. Agents increase the amount of code entering the pipeline; they do not automatically increase the capacity of the people checking it. Establishing trust remains expensive, particularly when a change touches sensitive code or can damage systems beyond the immediate task.

The migration examples make that imbalance concrete. Voss cites Stripe and Anthropic launch materials reporting a 50-million-line Ruby migration in one day, against an estimate of more than two months of team work. Bun reported moving more than a million lines from Zig to Rust in six days. Generating changes at that scale creates an urgent question: what would justify accepting them?

The obvious response is to assign developers more review work, perhaps making it their entire job. But sustained inspection has limits. Voss cites a Cisco study conducted over ten months, covering 2,500 reviews and 3.2 million lines of code. Reviewers became less effective at finding defects beyond roughly 400 lines in one sitting, with effectiveness dropping sharply above 450 lines per hour. These figures describe that study’s conditions rather than a universal limit for every codebase.

Follow one 10,000-line agent PR through that constraint. At 450 lines per hour, reading it requires more than 22 hours; Voss estimates three or four working days for a real review. That is one change from one agent. Running several agents increases incoming work without increasing a reviewer’s attention, while turning developers into full-time reviewers adds the burnout problem. “Review harder” does not close the throughput gap.

An alternative changes the unit of human work. Peter Steinberger’s proposal is to design the loops that prompt coding agents, rather than continually prompting them yourself. Voss connects this with Andrej Karpathy’s argument for removing humans from the execution loop. The engineering question becomes whether the loop can inspect enough of the system to make a trustworthy decision.

OpenAI’s internal-product experiment takes loop design seriously. Starting from an empty repository, agents produced about a million lines of code and approximately 1,500 merged PRs over five months, with three engineers. Even the agent-review scaffolding was agent-written. Human PR review was optional, and most review work happened between agents. The product’s purpose and source were not disclosed in the account Voss describes, limiting what other teams can conclude about where the approach transfers.

The bet is that loop design can substitute for direct inspection. Its effectiveness depends on visibility: a check can judge only what it can observe. Tests supply an inexpensive proxy for quality, but passing the tests establishes less than a maintainer’s decision to merge.

METR tested that gap by hiring four active maintainers from projects represented in SWE-bench to inspect PRs that had already passed its grader. Maintainers considered only about half mergeable. Rejections included code-quality problems and changes that broke behavior outside the test suite. The agents could not revise their work after feedback, so this measures first-pass mergeability rather than the eventual outcome of an iterative contribution. For the question of removing humans entirely, however, requiring maintainer feedback would restore the checkpoint being tested.

Cognition’s FrontierCode is introduced around the maintainer’s actual question: “Would you merge this?” Voss describes more than 20 maintainers creating 150 tasks from their repositories, each representing more than 40 hours of expert work. The evaluation broadens the decision beyond the requested behavior:

• Behavioral correctness: Does the change do what the task requires?
• Regression safety: Does it preserve other behavior?
• Scope discipline: Does it stay within the intended task?
• Test quality: Are its checks useful?
• Maintainability: Is the resulting code suitable for the repository’s future?

The reported contrast is striking: 88% on SWE-bench Pro versus 29% on the hardest slice of a maintainer-oriented evaluation, with another model scoring under 6% on that slice. The benchmark attribution is unresolved because this passage alternates between FrontierCode and Frontier Bench; the model identities are also uncertain. These figures therefore support the reported contrast between test performance and mergeability, without establishing a verified model-by-benchmark comparison. They also describe different benchmark conditions, rather than a controlled test of one isolated review skill.

A mergeability benchmark would do more than rank models. Voss draws on Sarah Guo’s explanation of why coding models improved quickly: compilers and test suites provide cheap verification. A training process can repeatedly generate candidates, check them and use the results to improve subsequent behavior. Making maintainability, scope discipline or regression safety similarly checkable would give model trainers additional targets.

That makes writing the rubric consequential. Behavior rewarded by today’s evaluator can become behavior produced by tomorrow’s model. Voss points to CriticGPT, trained by OpenAI in 2024 to find bugs in model-written code, as prior work: humans assisted by the model outperformed either humans or the model alone. Better review assistance can improve the feedback used to train coding systems.

Completeness remains the hard part. A machine-checkable rubric can capture useful parts of human review without capturing every reason a maintainer would reject a change. Voss treats a comprehensive mergeability standard as future work, then turns to the narrower review systems already operating in production.

Automated review is already a substantial production workload. Voss reports 60 million reviews by GitHub Copilot’s reviewer, accounting for more than one in five reviews on GitHub. That establishes widespread use; useful individual findings still require careful engineering.

Cursor’s architecture exposes several mechanisms for improving those findings:

• Repeated inspection: Its first reviewer ran eight passes over each diff and shuffled reviewer order because order affected the results.
• Agreement filtering: Multiple passes can discard false positives. Voss cites a Peking University study reporting up to a 44% improvement in review quality from keeping findings on which passes agreed.
• Directed investigation: The rebuilt reviewer reasons over the diff, calls tools and chooses where to dig.
• Default suspicion: The model needed instructions to assume something might be wrong, rather than accept plausible-looking code at first glance.

False positives matter because every reported issue spends a developer’s attention. Repeatedly flagging correct code teaches people to ignore the tool. Default suspicion addresses a separate failure: declining to investigate because the change looks reasonable. A useful reviewer must search actively while keeping its eventual reports selective.

Review is also joining repair. Cursor’s reviewer can spawn a fix agent from a finding, write a patch and return a diff for human approval. Running the code to demonstrate that the reported bug is real is described as a next step Cursor wants to take. Repair reduces the work between finding and resolution, but approving the patch remains a human decision.

Where does the human remain when review produces a repair? The flow below follows a diff through investigation, a finding and a proposed patch. The checkpoint has moved: the developer receives a repair to approve, while the reviewer and fix agent perform the intervening work.

Other vendors address different parts of the problem:

• CodeRabbit: Voss reports more than 13 million requests reviewed, illustrating the scale of dedicated review services.
• Greptile: A repository-wide graph helps the reviewer follow a change into distant code.
• Graphite: Accepted and rejected suggestions form an evaluation set for improving the reviewer.

Human acceptance supplies the common feedback signal. Cursor calls its measure the resolution rate and reports raising it from 52% to over 70%. These companies tune their harnesses against developers’ judgments at scale. Acceptance tells them whether people take a suggestion; it does not independently prove the resulting software correct. Humans still review the reviews.

Nicholas Carlini’s C compiler experiment separates being “in the loop” from being “on the loop.” Sixteen agents built a compiler in Rust over about 2,000 sessions, and it could compile the Linux kernel. Humans did not approve each code change, but a human built the test harness and feedback systems that judged the work. Removing per-change approval still left human judgment embedded in the machinery.

Bun’s migration develops the earlier throughput example into a review problem. Agents ported about a million lines from Zig to Rust in six days. The existing test suite served as the gate, and 99.8% of it passed. When the agents deleted the old Zig code in one giant PR, another robot flagged the deletion as “AI slop.” The mismatch is funny, but also useful: judging whether a diff looks plausible is different from checking whether a migration has completed.

Closer inspection changed what the passing tests meant. Voss reports 13,044 unsafe blocks in the Rust port, compared with roughly 74 in a comparable human-written codebase. In her explanation, these blocks place responsibility for memory-safety assumptions on the author. Their count does not prove that many defects exist; it identifies safety obligations that successful public-interface tests do not establish.

The sequence matters. Agents changed the implementation language. Existing tests checked much of the observable behavior, and those tests largely passed. Inspection then revealed thousands of internal safety assertions outside what the suite was designed to assess. Behavioral compatibility and unresolved safety obligations can coexist. The gate did real work, but it could not answer every question raised by the new implementation.

Returning to OpenAI’s experiment reveals the infrastructure behind optional human review. Codex reviews its changes, calls additional agents to review those reviews and continues until every agent reviewer is satisfied. The system can boot at every change, letting an agent run it, inspect the UI and check whether a bug was fixed. Access to the logging stack adds information that a diff alone cannot supply.

What can this review loop see? The diagram separates the proposed change from the running application and its logs, then shows additional agents inspecting the review. Agreement supplies the stopping condition. Execution and logging broaden the evidence available before the reviewers reach it.

Accumulated low-quality code required another mechanism. For a period, humans spent every Friday cleaning up “AI slop.” That recurring manual work did not scale, so agents were trained to identify and remove it. The response was to improve the inspection and cleanup system rather than ask the same people to try harder.

Experience can still force a reversal. Voss recounts Dexter Horthy spending six months advocating shipping without reading the code, then publicly retracting that advice after large parts of the system had to be ripped out and replaced. His plea, “Please, please read the code,” gives an operational counterexample to assuming that successful generation makes inspection unnecessary.

Some reasons to preserve code live outside its tests. Voss uses a module with three external users and an obscure cron job depending on that module as examples. A change can satisfy the visible task while deleting something another system still needs. Human checkpoints therefore tend to survive where correctness is expensive to establish, consequences are large or someone must accept responsibility for the result.

Moving human judgment into a review system creates another review task: checking that system itself. Voss cites Anthropic’s automated security reviewer README warning that the action is not hardened against prompt injection and should review only trusted PRs. A reviewer reading attacker-controlled material can be influenced by that material while deciding whether it is safe.

In a study Voss describes, vulnerable code presented with an innocent-looking commit message fooled an autonomous review agent in 88% of attempts, compared with 35% for human reviewers receiving the same attempts. These rates belong to that attack setup, rather than automated review generally. Within it, the framing influenced acceptance despite the underlying vulnerability.

Confident presentation compounds the problem because coding agents can supply reassuring explanations alongside faulty code. The review system has to evaluate the change without inheriting its author’s confidence. Evaluating reviewers is itself unsettled: Voss notes benchmark scaffolds leaking answers and disagreement over how to measure review quality. The grader needs scrutiny alongside the code it grades.

Once pre-merge review is automated, production behavior becomes the final check against reality. Voss calls it the “last reviewer standing.” The useful object of inspection is the trajectory: what the system actually did, step by step, while interacting with the real world. A passing pre-release test cannot substitute for observing those consequences after deployment.

Code review becomes an engineered system with several things to inspect: benchmarks, classifiers, rubrics, test suites and evaluations. Humans move from reading every line to designing and supervising those mechanisms. Each layer still needs someone to decide whether it deserves trust.

The closing instruction—“stop reviewing PRs”—is deliberately provocative after the failures just described. Voss’s practical recommendation is to spend human judgment building a reliable review harness: codify what good means, supply company context and domain knowledge, and give agents rules and evaluations they can work against. The earlier exceptions still matter. Expensive correctness checks, large consequences and accountability justify human intervention.

The proposed payoff is greater shipping capacity from judgment applied higher in the system, rather than repeated over every diff. This is a direction for teams to pursue, not a measured speedup guaranteed by the examples. The goal is to explain, with evidence, why the software you shipped deserves trust.