ReviewDebt: a practical framework for scoring every pull request — Sachin Gupta, eBay

What happens when a team produces more code while giving each change less human attention? Sachin Gupta opens with that gap, citing GitHub’s 2025 Octoverse figures: commits increased approximately 25% year over year, while comments on commits decreased 27%. He uses commit comments as a proxy for review activity. That proxy needs a boundary: GitHub’s separate issue/PR-comment measure was essentially flat, so fewer commit comments alone cannot establish a decline in overall review attention.

The review bottleneck appears more directly in Faros’s 2026 engineering telemetry. Faros reports median time in review increasing 441.5%, or about 5.4 times the baseline, and 31.3% more PRs merging without review. Its methodological explanation describes comparisons between each organization’s lowest and highest AI-adoption periods across two years, rather than a simple year-over-year comparison. These are correlations; the unreviewed-merge figure is a relative increase, not the share of all PRs merged without review.

The concern is compatible with coding agents making engineers faster. ReviewDebt is the gap between producing a change and establishing that humans can responsibly trust it. Gupta’s framework makes that gap discussable through five signal families, three scanner walkthroughs, and a scan of public repositories.

Delivery dashboards can tell an encouraging story while leaving this gap invisible. Gupta describes roughly 16% more PRs per developer in the Faros cohort of 22,000 developers and more than 4,000 teams; Faros’s public summary labels the corresponding 16.2% figure as an increase in PR merge rate. DX’s PR-size analysis reports a median increase from 44 to 72 lines between July 2025 and June 2026, which Gupta describes as about 63%. That size analysis is separate from DX’s 16-month productivity study, covering more than 400 organizations, in which AI tool usage increased 65% while median PR throughput rose just under 8%. Gupta also describes open-to-merge cycle time as modestly lower.

Those measurements describe production, but their interpretation depends on how the work changed. Splitting one PR into seven increases PR count without necessarily increasing delivered value. Larger PRs can represent bloat. A shorter merge cycle can result from reviewers ceasing to push back. The speed of production does not measure the speed of trust.

The missing costs show up in recognizable engineering behavior:

• Reviewer fatigue: engineers carry more review load, with less attention available for each change.
• Late-night merges: a PR waits three or four days, then receives a thumbs-up at 11 p.m. before a Friday deadline.
• Test theater: tests assert what the implementation does, including bugs, rather than what it should do.
• Architectural drift: the same problem acquires three different solutions in three different files.
• Incident lag: a failure appears weeks or months after merge, without anyone connecting it to the original AI-authored change.

A throughput dashboard can improve while all five conditions worsen.

More precisely, ReviewDebt is the accumulating gap between code an agent has produced and code humans have reviewed, trusted, and understood. Its interest is paid in human attention. Three feedback loops make the debt harder to repay as it grows.

First, insufficiently reviewed code becomes material for the next change. Fine-tuning, retrieval-augmented grounding, and in-context suggestions all allow yesterday’s implementation to shape tomorrow’s PR. A questionable pattern can therefore reproduce through the repository. Second, when most of a PR is generated, reviewers can narrow their attention to syntax and obvious bugs. Architectural decisions then move from review time to never. Third, leadership resets its expectations around the new throughput without adding proportional review capacity. The slack needed to revisit old decisions disappears. Each loop is survivable on its own; together, they can sustain continued accumulation.

The scoring contract is deliberately deterministic: five signal families contain ten checks computable from a pull request and its repository. An LLM judge would introduce another moving part. Changing the model could change the score for the same PR, making the result harder to defend in an engineering review. A useful score should instead lead back to identifiable properties of the change. The families are diff size and coupling, test evidence, directory and ownership spread, AI-authorship indicators, and evidence and rationale.

Diff size and coupling starts with net lines changed, files touched, and whether the changes cluster within one module or spread across many. Gupta describes a recurring failure mode in which an agent fixes the same symptom at multiple call sites instead of moving the fix to its root cause. That distinction matters more than line count alone: a reviewer must hold the relationships between the changed files in mind. As coupling grows, review effort can rise more steeply than the size of the diff.

Test evidence gap uses a simple per-PR ratio:

Gupta describes AI-authored changes arriving with minimal tests or no test files. Yet generating more tests does not automatically close the gap: a test can faithfully encode an implementation’s current behavior while preserving its bugs. The ratio measures test presence, not whether the assertions express the intended behavior. Human review still has to establish the latter.

Directory and ownership spread counts the distinct code-owner teams whose files appear in the diff. A change contained within one team’s territory may fit one reviewer’s mental model. A cross-team change requires several people to reconstruct different contexts and coordinate their approvals. The typing time saved by an agent can be consumed by the cost of assembling that attention.

AI-authorship indicators route attention; they are not proof of authorship or grounds for blaming an engineer. Gupta describes three detection modes:

• Co-author footers: commit attribution naming an assistant such as Copilot, described as the strongest signal.
• Branch patterns: prefixes associated with Codex, Copilot, or Cursor.
• Disclosure phrases: wording such as generated by or assisted by in PR bodies and commit messages.

In an initial check of 524 PRs across three anonymized public repositories, A, B, and C, Gupta reports a 0% result for a disclosure signal in repository C despite his understanding that coding agents were used there. He suggests markers might have been suppressed, but the result only establishes missing detectable disclosure—not the absence of AI assistance or the reason for missing markers.

Evidence and rationale gaps ask whether the PR explains why the change is necessary, rather than merely naming what changed. The contrast is concrete:

The stronger description gives a reviewer a claim to evaluate and evidence to follow. Without that context, the reviewer must reconstruct the author’s diagnosis before assessing the implementation.

Gupta calls missing rationale the fastest destroyer of reviewability in regression fixtures and hand-scored examples. He also distinguishes those demonstrations from the open-source sample, where conventional formatting was common and this signal fired rarely. A dramatic fixture shows what a check detects; it does not establish how often the problem occurs.

The combined score runs from 0 to 100. Its weights are defaults to calibrate against a team’s actual review experience, not universal constants. Gupta recommends scoring the last 200 merged PRs and checking whether the highest scores correspond to changes reviewers found burdensome. The presentation specifies the signal families and action bands, but not the complete ten-check formulas, default weights, or conversion to reviewer minutes.

These bands turn a number into a next step. A high score should explain what information or restructuring would make review tractable.

The action policy can be represented independently of the underlying scorer:

type ReviewAction =
  | "low-burden"
  | "standard-review"
  | "request-evidence"
  | "split-or-request-context";

function reviewAction(score: number): ReviewAction {
  if (!Number.isFinite(score) || score < 0 || score > 100) {
    throw new RangeError("ReviewDebt score must be between 0 and 100");
  }
  if (score < 25) return "low-burden";
  if (score < 50) return "standard-review";
  if (score < 75) return "request-evidence";
  return "split-or-request-context";
}

Keeping the action mapping separate makes the operational decision explicit without pretending that the score itself supplies all the evidence a reviewer needs.

The demonstration proceeds through prepared CLI reports rather than a live scanner run. The clean PR scores 0/100, with an estimated six minutes of review effort and no checks fired. Its Why, Reviewer Focus, and Author Next Actions sections are empty. That absence is intentional: the scanner should generate advice only when it has a specific concern, leaving healthy changes with a minimal report.

The next example, described as public-repository material used in the scanner’s regression suite, scores 60/100 in the Needs Evidence band, with an estimated 86 minutes of review effort. Here the report has three useful jobs: explain which checks fired, tell the reviewer where to focus, and tell the author what to change or supply next. The score establishes urgency; the structured advice makes it actionable.

The report labels soft AI-authorship indicators as “information only, not a definitive claim and not a penalty on its own.” Nevertheless, the AI-indicator check contributes five of the 60 points; the remaining 55 come from diff size, claim mismatch, and missing tests. The intended distinction is conditional amplification rather than a standalone penalty for using an agent. The exact conditional rule is not specified, but most of this example’s burden comes from properties that would also make a human-authored PR difficult to review.

The third PR is AI-authored and well-shaped. Gupta describes the same agent doing similar work, now with tests added, green CI, and the risky path called out in the description. It scores 7/100, in the low-burden band, and displays 14 minutes of review effort. That figure is presented as scanner output, not an established measurement of elapsed human review. The AI indicator still fires at low severity and contributes two points, while the report retains its information-only qualification. AI assistance and low review burden can coexist when the change carries the evidence needed to evaluate it.

The broader scan covers 524 PRs across three public repositories. Gupta initially introduces it as a 90-day view. The central observation is that review burden can rise while detected AI-authorship indicators remain relatively steady. He reports 186 estimated senior-reviewer hours for one repository over 27 days, versus 43 hours for another, describing equal windows with different PR volumes. Those figures are effort estimates from the framework, not time-tracking records.

Gupta reports that AI indicators fired on 5–20% of PRs weekly across the three repositories, without disproportionate concentration in the high-burden bands. Only four of the 524 PRs reached Needs Evidence or High, and all four involved structural work: large migrations, SDK rewrites, or multi-team refactors. Marker prevalence describes detected disclosure, not the true proportion of AI-authored code. Within this scan, complexity distinguishes the hardest individual reviews, while volume determines how much review work accumulates.

The aggregate figures need to be read with their reporting limits. Gupta gives a total of 228 estimated senior-reviewer hours across the three repositories, now specifying observation windows of 27–90 days. That total does not reconcile with the earlier 186- and 43-hour figures, which already sum to 229 hours before a third repository is included. The presentation does not resolve the discrepancy, so the aggregate cannot support a precise staffing calculation. Separately, he reports a sustained merge rate of nine PRs per day for the high-velocity repository.

The most extreme individual PR has an estimated review effort of 5,036 minutes—about 84 hours—while scoring 73 in Needs Evidence. This separates two outputs that should not be conflated: a bounded score used to route review and an effort estimate used to describe workload. A score below the highest band can still accompany a very large review task. Gupta’s operational concern is the accumulation of that work as PR volume increases, not a claim that every AI-assisted PR is unusually difficult.

The immediate interventions follow the signal families and require no new tooling:

• One logical change per PR: optimize for a coherent review task, rather than smallness in the abstract.
• Tests ship with the change: even when an agent writes both implementation and tests, the human author confirms that the assertions capture intended behavior.
• One owner territory: split cross-cutting work into team-specific PRs where possible, keeping each review within one context and mental model.
• The human author writes the why: Gupta treats the PR description as the moment the author commits to understanding what is being shipped.
• The same review standard applies to everyone: AI-authored and human-authored changes need the same evidence. Gupta describes the authorship amplifier as activating when other signals are weak; strengthening the other four families addresses the underlying burden.

These practices reduce the work a reviewer must reconstruct before making a responsible decision.

A rollout can fit into the existing PR workflow:

1. Backfill: score the last 200 merged PRs and inspect the highest-scoring changes.
2. Set a justification threshold: Gupta suggests 50 as a starting point; an author supplies a comment for a PR at or above that score.
3. Surface the result: post a score comment on every PR for visibility, without making the scanner a merge blocker.
4. Aggregate weekly by team: watch the slope of the debt line, not just individual scores.
5. Discuss the result: bring it into retrospectives and roadmap reviews so review capacity becomes part of planning.

The justification threshold creates a conversation about evidence. It does not have to become an automatic rejection rule.

That conversation can connect delivery gains to their human cost: an AI rollout added X% throughput, while Y review-debt points accumulated over Z weeks; at the current slope, the associated workload is approximately N senior-engineer hours. The hours component requires a locally credible effort estimate. Its purpose is to make staffing and prioritization discussable alongside production gains.

Gupta forecasts a shift from adoption in 2026 toward governance in 2027. The questions then become concrete: can the team trust the code it ships, who is accountable when an AI-authored change causes an incident, and where is the audit trail? ReviewDebt is intended to connect those questions to everyday review decisions while retaining the productivity gains of coding agents.

Before building the tooling, take 20 PRs from the previous week and score them manually using the five signal families. That small exercise tests whether the framework identifies the review problems the team actually recognizes. Then plot the results weekly: Gupta emphasizes the slope more than the current level. Bring throughput, debt, and the direction of change into the next review or sprint-planning discussion.

The measurement only matters if it changes review behavior. An approve-with-comment merge is not a substitute for resolving a concern. Saying the AI did the boring part, expecting QA to catch the problem, or pointing to smaller PRs does not establish that someone understands the change. Gupta closes by asking engineers to stop reflexive LGTM approvals and apply the habits that make a PR reviewable. The responsibility remains with the people shipping the code.