How to Quantify AI ROI in Software Engineering (Stanford Study / 120k Devs)

Companies spend millions on AI tools for software engineering. How can they tell whether those tools improve engineering outcomes? License purchases establish access, but answering the investment question requires a way to evaluate the work engineers deliver. Yegor Denisov-Blanch introduces research conducted over two years that uses historical Git data to examine changes over time and differences across companies.

The measurement starts with a code commit. Multiple independent panels, each containing 10–15 experts, evaluate its implementation time, maintainability and complexity. Denisov-Blanch reports that millions of these evaluations supplied labels for a machine learning model trained to reproduce panel judgments. This makes the assessment scalable; when an output is disputed, another expert panel can provide a comparison. The model is therefore an estimate of expert evaluation, not a direct observation of business value.

That measurement approach supports four successive questions: what drives productivity gains, how teams actually use AI, how usage connects to return on investment, and what a detailed enterprise case study reveals.

The first comparison matches 46 teams using AI with 46 similar teams not using AI and measures net productivity gains quarterly. Denisov-Blanch reports a median gain of about 10% for this cohort as of July in the presentation year. The chart’s shaded region contains the middle 50% of observations; its dark blue line shows the median. This is a result for the matched cohort, not a claim about every engineer in the broader research program.

The distribution matters as much as its center: the gap between stronger and weaker performers is widening. Denisov-Blanch extends that pattern into an explicitly illustrative, unscientific projection in which successful early adopters compound their gains while struggling teams fall further behind. He expects eventual convergence, rather than unlimited divergence. The practical question for a leader is where their organization currently sits—and whether its adoption needs correction. Without measuring outcomes, that position remains invisible.

Token consumption is an obvious candidate explanation for the differences between teams. The next chart plots productivity increase against tokens used per engineer per month, with a logarithmic horizontal axis. Denisov-Blanch reports a loose linear correlation of roughly 0.20 between token usage and productivity gains. He also identifies a directional dip around 10 million tokens per engineer per month, where teams appear to do worse than somewhat lighter users.

The quality of AI use may matter more than its volume. The observed dip does not establish a token budget to target or avoid; it shows why increasing consumption alone is a poor strategy for increasing productivity.

The next potential explanation is the environment in which engineers work. The experimental Environment cleanliness index combines tests, types, documentation, modularity and code quality into a score from 0 to 1. Against productivity lift relative to teams not using AI, Denisov-Blanch reports R² = 0.40 for the relationship with environment cleanliness. That association motivates investment in codebase hygiene as part of AI adoption.

An illustrative task-share diagram explains the mechanism. At any given level of cleanliness, a vertical slice represents the mix of tasks in a sprint:

As cleanliness improves, the diagram shows more work becoming suitable for AI assistance. It expresses a relationship between the codebase and task suitability, rather than a measured forecast of task automation.

That environment is not fixed. Unchecked AI use can accelerate codebase entropy—the accumulation of technical debt—and move the codebase toward lower cleanliness. Human maintenance must push in the other direction to preserve the conditions that make AI useful. Generating more code while neglecting its structure can erode the very productivity gains the tool initially offered.

Task selection creates another feedback loop. When engineers use AI for unsuitable work, they receive outputs that are rejected or require heavy rewriting. Repeated failures weaken trust, engineers stop using the tools, and potential gains shrink further. Knowing when to use AI—and when to do the work without it—is therefore part of effective adoption.

To move beyond consumption, the research introduces an AI engineering practices benchmark. It scans repositories for artifacts or fingerprints that indicate how teams use AI. Each pattern is quantified as a percentage of active engineering work, and the analysis repeats monthly using Git history. Denisov-Blanch describes the detection as directional and still evolving.

The distinction between the last two levels is scope: completing a bounded task autonomously is different from orchestrating the full workflow. At the time of the presentation, Denisov-Blanch describes the benchmark as a forthcoming open source tool available through signup on the SWEPR research portal.

Applying the benchmark inside one company reveals why access is insufficient. Two business units had the same tools, licenses and spending, but their adoption patterns differed substantially. The benchmark indicated AI use in almost 40% of the first business unit’s work, while the second unit lagged.

A company-wide purchasing decision did not produce a uniform way of working. Leaders need to understand both whether engineers use AI and how they incorporate it into their work; an aggregate access or usage figure can conceal the difference between business units.

Ideally, AI ROI would be measured directly through business outcomes: revenue, net revenue retention or another business KPI would improve after engineers received AI tools. The attribution problem is that sales execution, macroeconomic conditions and product strategy also influence those outcomes. Too much happens between tool adoption and revenue to cleanly isolate the engineering contribution. Measuring engineering outcomes offers a closer signal, provided the analysis goes beyond AI usage itself.

Using engineering outcomes as a proxy for value rests on several conditions:

• Product direction: The product function must channel additional engineering capacity into valuable work. More capacity directed at the wrong problem does not solve the business problem.
• A meaningful bottleneck: Engineering must constrain the organization’s ability to create value; increasing capacity matters less if the limiting factor sits elsewhere.
• Balanced measurement: Guard against Goodhart’s law with multiple metrics and a culture that does not weaponize them. Optimizing a target in isolation can undermine the outcome it was meant to represent.

These conditions define when an engineering improvement can plausibly translate into business value.

The alternative is not necessarily a perfect causal measurement. AI adoption is changing quickly, and Denisov-Blanch favors useful proxy metrics over having no measurement while waiting for flawless ones. Imperfect measures can still help an organization identify problems and improve its decisions.

The measurement design has two parts: establish AI exposure, then measure engineering outcomes. For enterprise exposure, the presentation distinguishes two approaches:

Access-based analysis is simpler, but noisy: receiving a license does not establish that someone used it. Denisov-Blanch calls usage-based telemetry the gold standard because it gives a more direct view of who uses AI and where.

Telemetry granularity also depends on the vendor and reporting surface. The presentation contrasts aggregated GitHub Copilot data with more granular Cursor data. Treat that as a description of the reporting being discussed, not a permanent vendor boundary: Copilot also introduced user-level reports under specific enterprise access and policy conditions. The practical requirement is to establish which granularity is available for the analysis.

Historical Git data makes the outcome side retrospective. An organization that has already adopted AI can examine engineering work before and after adoption instead of starting from zero and waiting months for a new experiment. The usefulness of that history still depends on pairing it with an appropriate account of access or actual usage.

For engineering outcomes, the proposed framework separates a primary metric from guardrail metrics. The primary measure is engineering output estimated by the model that reproduces expert-panel assessments. It is not lines of code, pull request counts or DORA metrics.

The guardrails cover three groups:

• Rework and refactoring: How much effort goes into changing existing work.
• Quality, technical debt and risk: Whether delivery preserves the health of the codebase.
• People and DevOps: Useful indicators of the working and delivery environment, rather than direct productivity measures.

Increase engineering output while maintaining healthy guardrails. Maximizing every indicator is neither necessary nor sensible: a useful operational measure can stop tracking productivity when pushed beyond a healthy range.

The enterprise case study begins with a 350-person team reporting to a vice president. The team adopted AI in May of the presentation year; the analysis compares four months before adoption with four months after. Denisov-Blanch reports a 14% increase in pull requests over those windows. Viewed alone, that looks like an improvement. But it leaves reviewer burden and code quality unanswered.

The next measure is code quality, defined here as maintainability on a scale from 0 to 10. Before AI adoption, quality was relatively stable and consistent. After adoption, it declined and became more erratic. The additional pull requests therefore did not arrive with an unchanged quality profile.

The engineering-output view then sums the work delivered each month and separates it into rework, refactoring, additions and removals. Rework changes recent code; refactoring changes older code. The chart also provides benchmarks against similar companies in the same industry. Denisov-Blanch reports that rework rose to 2.5 times its previous level while effective output, a productivity proxy, did not meaningfully increase. The displayed slide labels these comparisons as 2.6× rework and +1% effective output; the spoken and displayed rework figures differ slightly, but both show substantially more rework with little output gain.

The recap puts the measures together:

More pull requests alone cannot settle whether the team became more productive. Here, the broader measurement shows deteriorating quality and more corrective work without a meaningful increase in effective output.

For this team, AI ROI might be negative; the presentation does not establish a definitive monetary return. Counting pull requests alone could have produced a very different story: label the increase a productivity gain, translate it into millions of dollars of engineering capacity, and conclude that it easily covers license costs. That calculation would assign economic value to an improvement the broader outcome measures did not show.

The response is to diagnose the adoption, not abandon AI. The company can use the quality, rework and output evidence to investigate what is going wrong and improve how engineers work with the tools. The value of measurement is that it makes correction possible instead of allowing a favorable activity metric to stand in for success.

The closing invitation provides a route to that analysis: organizations can participate in Stanford’s research to access company-specific insights, including some through live portal dashboards. Denisov-Blanch particularly invites companies using Cursor Enterprise to support research on granular AI usage in software engineering. Organizations can register through the research portal.