Does AI Actually Boost Developer Productivity? (100k Devs Study) - Yegor Denisov-Blanch, Stanford

The pressure to adopt AI coding tools often begins with sweeping expectations that developers can be replaced or engineering output can immediately surge. Yegor Denisov-Blanch offers a narrower conclusion: AI can increase developer productivity, but it can also reduce it. The meaningful question is not whether AI generates code, but when its output translates into useful software engineering work. 0:15

The research draws on a dataset spanning more than 100,000 software engineers, more than 600 companies, dozens of millions of commits, and billions of lines of code. It combines historical Git data with comparisons across enterprises, midsized companies, and startups, allowing researchers to examine changes over time and differences between organizations. 1:35

A particularly important feature is that most observations come from private repositories. Denisov-Blanch argues that public repositories can misrepresent a developer’s work because contributions may happen occasionally or outside a primary job. Private repositories, while not a complete account of every engineering activity, provide a more self-contained view of the work performed by teams and organizations. 1:35

Counting commits, pull requests, completed tasks, or time between commits can confuse activity with accomplishment. Tasks differ in size, and AI can inflate apparent throughput by generating code that subsequently requires bug fixes. In that situation, the team records additional work without obtaining a corresponding increase in useful functionality. 4:05

Controlled experiments can also exaggerate the relevance of their results when they ask developers to build isolated projects from scratch. AI performs especially well on Greenfield tasks and boilerplate, but established software development often involves an existing codebase, dependencies, and accumulated context. Results from clean-slate exercises therefore do not automatically transfer to Brownfield engineering. 5:07

Self-reported productivity presents another measurement problem. In an experiment involving 43 developers, participants misjudged their productivity by roughly 30 percentile points, and only about one in three placed themselves within the correct quartile. Surveys remain useful for understanding morale and subjective experience, but Denisov-Blanch argues that they should not substitute for measuring productivity or the effects of AI on engineering output. 6:04

The proposed measurement method begins with an idealized expert review: independent engineers assess code changes across dimensions such as quality, maintainability, output, and how long equivalent work might take. Denisov-Blanch says expert judgments agree with one another and predict real-world outcomes, but assembling these panels repeatedly is too slow, expensive, and difficult to scale. The research group therefore built a model intended to automate that evaluation while correlating with expert assessments. 7:18

The model connects to Git and analyzes the source-code changes in individual commits rather than treating commit counts or lines of code as productivity. Commit authors, unique SHAs, and timestamps make it possible to organize those assessments across teams and time. Under this approach, engineering productivity is defined around the functionality delivered over a period, rather than the raw volume of code produced. 8:24

A pilot involving approximately 120 developers illustrates why this distinction matters. The analysis separates added functionality, removed functionality, refactoring, and rework. Rework changes recently written code and is characterized as wasteful, whereas refactoring alters existing code but is not necessarily wasteful. Following AI adoption, rework increased alongside apparent output, leaving an estimated 15–20% productivity improvement rather than the larger gain suggested by coding activity alone. 8:24

Across the broader picture presented in the talk, AI-assisted coding can initially appear to increase productivity by roughly 30–40%. Once teams spend time correcting bugs and revisiting problematic generated code, the average net improvement falls to approximately 15–20%. These numbers describe an aggregate pattern, not a universal guarantee: the amount of useful progress depends on the type of work and the downstream correction burden. 10:24

The strongest gains appear when task complexity is low and the project is Greenfield. In the distributions Denisov-Blanch presents, simple work generally benefits more than complex work, and Greenfield development benefits more than Brownfield development. Some high-complexity tasks show negative productivity effects. He explicitly notes that the underlying causes of those decreases are not yet clear and that these results concern enterprise settings rather than personal projects. 10:24

A simplified matrix, based on data from 136 teams across 27 companies, estimates 30–40% gains for low-complexity Greenfield tasks, 10–15% for high-complexity Greenfield tasks, 15–20% for low-complexity Brownfield tasks, and 0–10% for high-complexity Brownfield tasks. These figures are presented as orientative guidelines, not precise predictions for every team or codebase. Their practical value is in showing that project maturity and task difficulty interact rather than operating independently. 12:32

Programming-language popularity introduces another boundary. For widely used languages such as Python, Java, JavaScript, and TypeScript, the talk reports gains around 20% for low-complexity work and 10–15% for high-complexity work. For less popular languages including COBOL, Haskell, and Elixir, assistance is less useful even on simpler tasks, and complex work can become slower. Denisov-Blanch adds that developers may abandon a tool when its suggestions are useful too inconsistently. 13:40

Denisov-Blanch also presents a more tentative relationship between codebase size and AI-assisted productivity. He describes this portion as illustrative and less empirically established than the preceding findings, while arguing that gains decline as projects grow. The proposed explanations are limited effective context, a worsening signal-to-noise ratio, and the additional dependencies and domain-specific logic accumulated in larger systems. 14:54

A large advertised context window does not necessarily mean a model can use every supplied token effectively. Citing work referred to as Nolima, Denisov-Blanch describes declining coding-task performance as context length increases from 1,000 to 32,000 tokens. His example is Gemini 1.5 Pro, which he describes as having a 2 million-token context window but declining from roughly 90% to 50% performance by 32,000 tokens in the comparison he presents. 15:56

The practical conclusion is selective adoption rather than either blanket enthusiasm or blanket rejection. AI can improve developer productivity in many cases, but its effects vary with task complexity, codebase maturity, language popularity, codebase size, and context length. Teams should evaluate useful delivered functionality and correction costs together, especially when applying tools to complex work inside established systems. 9:33