How to Quantify AI ROI in Software Engineering (Stanford Study / 120k Devs)

Talk transcript

1. 0:00

   [upbeat electronic music] So companies spend millions on AI

2. 0:24

   tools for software engineering, but do we actually know how well these tools work in the enterprise, or are these tools just all hype? To answer this, and for the past two years, we've been researching the impact of AI on software engineering productivity, and our research is time-series because we look at Git historical data, meaning we can go

3. 0:45

   back in time, and it's also cross-sectional because we cut across companies. And the way we use to measure most of the, of the impact is by a machine learning model that replicates a panel of human experts.

4. 0:59

   The way this works is that imagine you have a software engineer who writes a code commit, and this code commit would be evaluated by multiple panels of, of ten, uh, fifteen independent experts who would evaluate that code commit across implementation time, maintainability, and complexity, and then produce an output evaluation.

5. 1:21

   So we took the labels of these panels across, you know, millions of, of kinda evaluations, and then trained a model to replicate this panel of experts, meaning that we can deploy this at scale, and if there's ever any doubts around the model's output, you can always kinda assemble your own panel and see that it correlates pretty well

6. 1:38

   with reality. Today, we'll talk about four things. We'll start off with looking at some of the things that are driving AI productivity gains in software. Then we'll look at a AI practices benchmark that we developed.

7. 1:53

   We'll then look at how we propose to measure AI return on investment in software engineering. And lastly, we'll finish things off with a case study.

8. 2:04

   So here, we took forty-six teams that were using AI, and we matched them with forty-six similar teams that were not using AI. And we measured their net productivity gains from AI quarterly.

9. 2:17

   And the shaded area is the middle fifty percent of the data, and the dark blue line is the median, which as of July of this year, stands at about ten percent for this cohort.

10. 2:28

    I'd like to direct your attention to the fact that the discrepancy between the top performers and the bottom ones is increasing. There's a widening gap. And so if we very unscientifically and very illustratively project this forward, we might get something like this, right?

11. 2:44

    Where, uh, you can have these top performers being part of this, the rich-gets-richer effect, where the, these successful early AI adopters might compound their gains while these strugglers could fall further behind.

12. 2:56

    At some point, this is gonna converge, and this is very directional. But my point here is that if you're a leader in a company, you definitely need to know in which cohort you are right now so that you can course correct.

13. 3:07

    And without measuring the impact of AI on your engineers, you're not gonna be able to do this.

14. 3:14

    So we started investigating, what are some of the factors that drive these top teams to perform better? And the first thing we looked at is AI usage, or basically token spent.

15. 3:24

    In this graph, you have the same kinda, um, on the vertical axis, the productivity increase, and then on the horizontal one, you have the token usage per engineer per month on a logarithmic scale.

16. 3:35

    And what you can see is that the correlation is quite loose, point twenty or so linearly, and there is a bit of a death valley effect around the ten million, uh, token mark, whereby com-- teams that were using that amount of tokens seem to be doing worse than teams that were using a bit less tokens.

17. 3:51

    It's very directional, but interesting nevertheless. The conclusion here might be that AI usage quality matters more than AI usage volume.

18. 4:02

    We dug deeper, and we said, "Well, does the environment in which the engineers work impact the productivity from AI?" And we came up with an Environment cleanliness index, index.

19. 4:13

    It's quite experimental. It's a composite score that looks at tests, looks at, uh, types, at documentation, and at modularity, and at code quality. And that index is on the bottom axis here from zero to one.

20. 4:25

    And then on the vertical axis, once again, you have the kind of productivity lift relative to teams not using AI. And so what you can see is that there is a point forty R-squared, meaning a pretty decent correlation around environment cleanliness and gains from, uh, AI, or productivity gains from using AI.

21. 4:44

    And so the takeaway here is to invest in codebase hygiene to unlock these AI productivity gains.

22. 4:51

    We dug deeper to illustrate this concept, and here we have on this graph on the vertical axis, the percentage of tasks that might, uh, be able to be completed by AI based on three colors.

23. 5:04

    And so green means that AI can do most of the work for that task in that sprint, yellow means that AI can help someone, and red, uh, means that AI is not very useful.

24. 5:13

    And this is quite illustrative, but it, it conveys the point. And so then any codebase at any point in time sits on a vertical line across this graphic, and what you can see is that clean code amplifies AI gains.

25. 5:27

    Secondly is that you need to manage your codebase entropy, right? Your codebase tech debt. Because if you just use AI unchecked, this is gonna accelerate this entropy, which is gonna push and degrade your cleanliness to the left kinda, right?

26. 5:41

    And then you as, as a human need to push on the other side to kinda improve or maintain that cleanliness to keep reaping the benefits from AI.

27. 5:50

    Thirdly is that it's important that engineers need to know when to use AI and when not to use AI. And what happens when they don't is this kinda line on the left whereby you have AI out- AI outputs that are rejected or need heavy rewriting.

28. 6:05

    Which then leads to engineers losing trust in AI, saying, "Okay, this just doesn't work. I'm not gonna use it," which then further collapses your AI gains.

29. 6:17

    Now, we said, can we find out whether we can look not only at usage but at how are these companies and these engineers using AI? And we came up with an AI engineering practices benchmark.

30. 6:31

    The way this works is that we can scan your codebase and detect these AI fingerprints or artifacts, basically traces of how your team is using AI. It's quite directional at this point, but evolving.

31. 6:43

    And we can quantify this based on the percentage of your active engineering work that uses each AI pattern, and then we kinda repeat this monthly using Git history.

32. 6:53

    And the way this works is more or less you have kinda a few levels, and level zero might be how humans are just not using AI and write all of the code.

33. 7:02

    Level one is kind of like personal use, where engineers are not sharing prompts across the team or not versioning them. Level two is team use, whereby teams are, are sharing these kind of prompts and rules.

34. 7:14

    And then level three is even more sophisticated. It's where AI autonomously does specific tasks, maybe not the entire workflow. And level four is, you know, agentic orchestration, which is where AI just runs the entire process.

35. 7:27

    And so this is gonna be an open source tool which you can leverage if you sign up on, uh, on the SWEPR research portal.

36. 7:35

    We applied this benchmark to one of the companies in our research dataset, and we saw this. This company had two business units with equal access to AI tools, right?

37. 7:46

    Same licenses, same spend, same tools, same everything. But the adoption rate and the usage rate was very different by business unit. On the left, the first business unit, uh, you can...

38. 7:57

    as you can see in the area in the blue, seemed to be using AI a lot more for almost forty percent of their work. Whereas on the, on the, uh, right, the second business unit seemed to struggle behind a bit more.

39. 8:09

    And so the takeaway here is that access to AI and even AI usage doesn't mean or doesn't guarantee that, that AI is gonna be used in the same way across a company.

40. 8:22

    As a leader, you w- really wanna be understanding not just whether they're using, but also how your engineers are using AI.

41. 8:31

    Great. Now let's dive into how do we actually measure AI return on investment in software engineering.

42. 8:40

    Oh, uh, there we go. Okay. So here, ideally, we would be measuring this based on business outcomes, right? I give my AI engineer... uh, my, uh, engineer's AI, and then I make more money, more revenue, net revenue, retention, whatever business KPI you wanna track.

43. 8:56

    The problem is that there's too much noise between the treatment, right, giving AI, and the result, which is the business outcome. And on top of this, there's confounding variables such as your sales execution, the macroenvironment, your product strategy.

44. 9:10

    And therefore, although that would be ideal, unfortunately, uh, I think we need to find alternative paths, and the most logical one is to simply look at the engineering outcomes because there is a clear signal, right?

45. 9:22

    But here we need to go beyond measuring AI usage into measuring engineering outcomes. There's a few caveats, and this topic is quite heavily discussed, and so I wanna mention some of them.

46. 9:34

    The first one is that this is assuming that our product function can properly direct that increased capacity into something that generates value. And if they aren't directing that, then it's a product problem, which although sits quite close to engineering, it's slightly different, right?

47. 9:50

    The second caveat is that this assumes that engineering is a meaningful bottleneck for value, which frankly it typically is, and that you can guard against Goodhart's law by using a balanced set of metrics and also by having a good company culture that doesn't weaponize these metrics.

48. 10:06

    And thirdly is that AI is still very new, and measuring proxy metrics is still better than not measuring. There's gonna be winners and losers in this AI race, and progress is better than perfection here.

49. 10:19

    And so metrics don't need to be flawless to be useful, is what I want to illustrate.

50. 10:28

    So then, um, here we have, uh, two parts which you need to do to get the ROI from AI, right? You kinda need to measure usage, and then you need to measure engineering outcomes.

51. 10:40

    And so let's start with usage. There's really two buckets for enterprises. There's kinda more in a research environment, but to make it simple, there's access-based and there's usage-based. Access-based is basically looking at when did people get access to the tool.

52. 10:56

    And here we have... you can kinda do a pilot group, give that group AI, and then compare it to a similar group without AI, or you can measure the same team across time.

53. 11:05

    The problem is that access-based is noisy, and the gold standard is really usage-based, which, uh, uses telemetry from APIs from these coding assistants, right, to, uh, give you the right data to know who's using AI and, and where.

54. 11:21

    And the caveat here is that the vendor API is different. Unfortunately, tools like GitHub Copilot aggregate the data, and other tools like Cursor give you more granular data.

55. 11:31

    The big takeaway is that you can measure impact of, um, retroactively by using Git history. And so you don't need to set up an experiment now and wait six months.

56. 11:41

    You can actually, if you've already adopted AI, you can go back in time and, and, and do this. It's quite easy.

57. 11:48

    Now, we've seen usage. Let's look in-into how do we actually measure engineering outcomes. What are some of the metrics we propose?

58. 12:00

    Here we have, um, our framework which we propose, which is using a primary metric and a guardrail metric. And so here, um, the primary metric is engineering output. It's not lines of code, it's not PR counts, and it's not DORA, and it's basically based on this machine learning model that replicates the panel of experts, right?

59. 12:17

    And the second set of metrics are the guardrail ones, which you wanna maintain at a healthy level, but you don't wanna maximize. It doesn't make sense to maximize them, truly.

60. 12:27

    And so then there's three categories within the guardrail ones, rework and refactoring, quality, technical debt, and risk, and then people and DevOps. The third bucket, it's important to highlight that these are not productivity metrics.

61. 12:38

    They're useful, but you cannot just kinda use them, like maximize them to maximize developer productivity. They kinda fall off at some point. And so the goal here might be to keep your guardrail metrics healthy while increasing the primary metric to whatever degree possible.

62. 12:54

    Now, let's dive into a case study. Here we worked with

63. 13:01

    a company that, a large enterprise. We took a team of th- uh, three hundred and fifty people under a vi- a vice president, and we measured pull requests. The reason we did this is to illustrate that you cannot measure pull requests to understand whether AI is helping you.

64. 13:16

    And so here, this team adopted, um, AI in May of this year, and we measured the four months before, four months after. We saw a fourteen percent increase. Great.

65. 13:24

    That's fantastic. But what about reviewer burden? What about code quality? So we measured code quality, and here what we saw is, um, I mean, firstly, actually, code quality, think of it as maintainability, scale from zero to ten, and, uh, there's kinda these bands.

66. 13:41

    Uh, it uses our, our methodology. You can read it online. But basically, what you see is that in the pre-AI period, their code quality was quite stable and consistent, and once they adopted AI, two things happened: code quality decreased, and then code quality became more erratic.

67. 13:59

    Next, we took a look at our metric, which is engineering output. It's not lines of code. And here for every month, you see the sigma, the sum of the output delivered for that month, broken down into four buckets, rework and refactoring.

68. 14:13

    So rework is when you're changing or editing code that was... it's still kinda fresh, so it's recent. Refactoring is when you're changing code that's a bit older. And, uh, what's, uh, then, like added and removed, it's pretty self-explanatory.

69. 14:27

    And then also you can see these kinda benchmarks, so we can benchmark this company against similar companies in their industry. And here, AI usage had two effects. Firstly is that rework went up by two point five times, which is really bad, and effective output, which is kind of like a proxy for productivity or so, didn't really change.

70. 14:46

    And so then what's the conclusion here? Let's do a recap. So we saw that PRs went up by fourteen percent, but this is inconclusive because more PRs doesn't mean better.

71. 14:55

    We saw that code quality decreased by nine percent, which is problematic. We saw that effective output didn't increase meaningfully, and then we saw that rework increased by a lot.

72. 15:06

    And so then the question here is, what is the ROI of this AI adoption, right? It might be negative. And what I wanna point out here is that had this company not measured this more thoroughly and simply measured PR counts, they would have thought, "Hey, we're doing great.

73. 15:20

    We increased our productivity by fourteen percent. Let's run the numbers. That's how many million? Lots of millions of dollars. And does this offset the AI licenses?" Sure thing it does, right?

74. 15:30

    The other thing is that I don't think this company should abandon AI. They should simply use this data to understand what they're doing wrong, how can they improve, because AI is here to stay.

75. 15:38

    It's a tool that's gonna transform how engineers are, are working, right? And you can't just, um, kind of like abandon it or so.

76. 15:46

    Great. So this concludes our insights for today. If you've enjoyed this, uh, talk and you would like similar insights for your company, I invite you to participate in our research.

77. 15:56

    Everything you've seen today can, uh, be accessed through kind of participating in our research, some of them through live dashboards in our research portal. And especially I'd like to invite companies that have access to Cursor Enterprise to participate because we have a high need for this so we can publish papers around the granularity of using AI, um,

78. 16:15

    in software engineering. You can sign up at softwareengineeringproductivity.stanford.edu. Thank you so much. [upbeat music]