AI Writes Code Faster Than Your Team Can Review It - Andrew Norrish, Sourcegraph

For a couple of years, Andrew Norrish could count on one question in conversations about AI adoption. In his words, “How many heads can this save us?” At Sourcegraph’s most recent customer advisory board, the question never came up. By then, customers had experience with what happened after they bought coding tools, encouraged developers to use them heavily, and expected releases to accelerate in proportion.

The tools could produce more code. Getting that code into production still required someone to decide whether it belonged there. In the pattern Norrish heard across customers, developers faced both a growing queue of pull requests and larger changes inside each one. The time needed to understand a change started competing with the pressure to clear the queue.

Consider what a reviewer needs to establish about an AI-written change. It may work in the service where the agent wrote it, yet break a caller elsewhere in the company. It may also violate an internal standard or, at a financial institution, a regulatory requirement. Someone who knows the surrounding systems has to recognize which questions to ask. Norrish described changes reaching production with little or no human inspection.

When one of those changes causes an incident, the site reliability engineer (SRE) inherits the questions the reviewer never answered. There may be no runbook for the failure and no engineer who remembers why the code works the way it does. Customers told Norrish that these incidents were taking longer to resolve and that some kept recurring.

In a large organization with years of legacy code, even cloning the entire codebase locally may be impractical. A coding agent working from a partial checkout has to make guesses about the rest. The responder investigating its work can run into the same limit from the opposite direction. An error surfaces in one service, its cause lives in another, and engineers end up scanning commits by hand to connect the two.

Finding the responsible commit solves only part of the problem. A fix discovered in one repository may apply to hundreds or thousands of others. If each team has to find and patch its own copy, some services remain exposed after the original incident is closed. Hiring more SREs adds people to the rotation, but customers were already seeing incidents accumulate faster than they could hire experienced responders.

Norrish cited an industry forecast that AI would write about two-thirds of new code by 2027. The nearer-term comparisons concerned work already landing in review. Some organizations estimated that they were merging almost twice as many pull requests with agentic coding, that those requests were more than twice the size of human-written ones, and that developers were spending almost twice as much time on review.

Developers also reported that generated code was often harder to review. Almost every developer in the findings Norrish described expressed some distrust of AI output, yet almost half said they did not always verify it. He cited reports of more incidents per pull request, alongside rollbacks, hotfixes, and customer-impacting deployment problems. A finding he attributed to Anthropic adds a complication for responders trying to consult the author of a change. As developers relied more heavily on AI to write code, their comprehension of that code declined.

Norrish sees a better fit for agents in work with a defined scope and enough context to apply the organization’s rules. Vulnerability remediation, security scanning, and migrations all offer ways to specify what an agent should look for and which changes it can make. Platform and reliability teams already run semi-automated incident playbooks and monitor known vulnerabilities. They also see firsthand which failures in generated code escape review.

Sourcegraph’s starting point is a code graph indexing the organization’s codebase. Search and code navigation sit above that index. Sourcegraph developed these capabilities over more than a decade for engineers working across large codebases. Exposing them to agents gives an agent a way to look up code beyond its immediate checkout. That matters as much to the agent diagnosing a failure as it does to the one writing the change.

For investigation, Sourcegraph’s deterministic search can query repositories, branches, and commits together. An engineer who knows the pattern to look for can search for it directly. When the starting point is a question about what broke, Deep Search can investigate and return an answer with citations back to the code. Norrish said this tooling can bring root-cause investigations down from days to minutes.

Once engineers recognize a risky pattern, they can set a monitor to alert when someone commits code that matches it. That gives teams a way to catch a known compliance issue even as pull request volume grows. For a pattern already in production, the same search can establish which repositories need attention. Sourcegraph’s bulk-change tooling then lets a team apply the repair across those repositories without waiting for each service owner to rediscover it.

Sourcegraph’s own security team had used HackerOne since before ChatGPT. As report volume climbed, a costly step was deciding which submissions deserved an engineer’s attention. The team connected HackerOne webhooks to agentic code search so an incoming report could be investigated against Sourcegraph’s code. The workflow can propose a remediation, flag a report for deeper security review, or judge that it needs no further attention.

At a customer with more than 5,000 developers, downtime or a significant spike in errors starts an internal root-cause analysis agent. It takes in logs and traces, then establishes which branch or tag was deployed. Production, Norrish pointed out, rarely runs main. With the deployed revision identified, the agent searches Git history for the commit and author associated with the fault. It can revert the change and trigger a redeployment, or hand the investigation to the SRE team for triage.

Another customer maintains a code review agent used by thousands of its developers. The agent follows specifications particular to a massive monorepo, and its evaluation suite spans hundreds of historical branches. Repeatedly running grep across the monorepo made that history expensive to search. The team needed fast, affordable lookups across branches and wanted to limit the variability introduced by other agentic tools. It chose Sourcegraph’s read-only Code Finder agent for retrieval. The evaluation suite governs the release decision. Passing it triggers the team’s push and publish processes.