What Data from 20 Million Pull Requests Reveal About AI Transformation

Hi, my name is Nicholas Arcolano, and I'm the head of research at Jellyfish. Today I'd like to talk to you about AI transformation, specifically what real world data can tell us about what's actually happening in the wild.

Now, a lot of AI native companies are being founded right now, and there are many more existing companies that are trying to transform themselves into being AI native. I've talked to many folks from these companies, and they all have the same big questions.

Number one, what does good adoption of AI coding tools and agents actually look like? Uh, number two, what productivity gains should I be expecting as we transform our team and the tools that we use?

Uh, three, what are the side effects of this transformation? And perhaps most importantly, if AI transformation isn't delivering as advertised, what's going on and what can you do about it?

Now, at Jellyfish, we believe the best way to get answers is with data. So in the next fifteen, twenty minutes or so, I'm gonna give you some data-backed insights from studies we've done to help you tackle these big questions.

Okay. Before we jump in, though, uh, let's take a minute to talk about the data behind the rest of the stuff in this talk. Uh, now at Jellyfish, we provide analytics and insights for software engineering leaders, and to do this, we combine information from multiple sources, including usage and interactions with AI coding tools like Copilot, Cursor, Claude

Code, uh, interactions, uh, with autonomous coding agents, things like Devin and Codex, as well as PR review bots. We also combine this with data from source control platforms like GitHub, so we can understand things about the actual code base where the work is happening.

We also pull in data from task management platforms, uh, things like Linear or Jira, and that tells you about what the actual goal of, uh, the work being done is.

So for the rest of this talk, we're gonna be looking at findings from a data set with data like this, uh, across our customers. It comprises about twenty million pull requests.

Uh, these were written and merged by about two hundred thousand, uh, developers from around a thousand companies. We've been collecting this data for more than a year, so today we'll be looking at results that span from June 2024 to the present.

Okay? So let's dig in. Question one, what does good adoption look like?

Well, let's start with lines of code. I don't think this is a great metric, but it's one we all hear about in the media a bunch, so it's worth talking about.

Here's data from a cohort of companies we've been tracking since June of last year. The purple bar represents the fraction of those companies that are generating fifty percent or more of their code with AI.

So if you look at that purple bar, you can see that starting last summer, only about two percent of these companies were generating fifty percent or more of their code with AI.

But you can see this has been steadily growing, and as of last month, among these same companies, now nearly half are generating fifty or more of their code with AI.

Now, I think a more useful thing to look at actually is developer adoption, because this gets at the actual behavior change that you wanna see in your team. It's also the thing I've seen that correlates most directly with good productivity outcomes, and we're gonna talk about this a lot more later.

Uh, but first, we define an AI adoption rate for developers by computing the fraction of time that they use AI tools when they code. So a hundred percent for a developer, that means you're using AI tools every time you code.

A company's adoption rate for the whole company, that's just the average of the adoption rates for all their individuals. So a hundred percent for a company means that every developer is using AI every time they code.

So what you see here, this is a plot of the twenty-fifth, fiftieth, and seventy-fifth percentile of company adoption rates, uh, by week for the developers and companies that we've been tracking.

And if you look at the AI adoption rates as of last summer, you can see the median adoption rate was around twenty-two percent. So, uh, median company developers are using AI twenty-two percent of the time that they code.

It's grown steadily since then. Today, we're seeing median adoption rates close to ninety percent. Now, if you're like me and you're using multiple tools constantly in parallel, both synchronous and asynchronous modes, uh, you're, you're at a hundred percent.

It might seem crazy to you that not everyone else is at a hundred percent. However, the reality is that for many teams, there are still real technical, organizational, and cultural barriers to adopting these tools more completely.

So that brings me to my final point on adoption. You might ask, what about autonomous coding agents? Now, the results I've just shown you, those are overwhelmingly from interactive coding tools, things like Copilot, Cursor, Claude Code.

Now, we know that these tools all have interactive agentic modes, but what about your, your kind of true, fully autonomous agents, like your Devins or your Codexes? Uh, maybe you're using agents like these or something else to good effect, or maybe you haven't really gotten going with autonomous agents yet.

It's fine, you know, wherever you are on your journey. But i- if it feels like you're slow-going getting off the ground with autonomous agents, I'm here to tell you you're not alone.

So in our data set, we only see about forty-four percent of companies have done anything with autonomous agents at all in the past three months. The vast majority of that work is what you'd consider, um, a trialing and experimentation type stuff, like not full-scale production.

And ultimately, it all amounts to less than, uh, point two percent of the millions of PRs that were merged over that timeframe. Uh, so, you know, still very early days.

All right, let's move on. Now I'd like to talk about productivity. So even though autonomous agents aren't yet delivering at scale, we're still seeing big gains from adoption of interactive coding agents.

So let's talk about what we're seeing. First, though, what do we mean by productivity? This can be a very loaded term, kinda squishy, overloaded. There's many ways to attack it.

Uh, a good place to start, though, is plain old PR throughput. How many pull requests does the average engineer merge per week? Not the most exotic metric. But it's proven, it's widely accepted.

Uh, do note that the absolute level of PR throughput is something that varies, right? It depends on things like how you like to scope work. It actually also depends on your architecture, and put a pin in that because we're gonna talk about that more later.

Uh, however, measuring the change in PR throughput, especially if you keep all these other things constant, measuring that for your team is a good way to, uh, track productivity gains.

Another good one, cycle time. Uh, you know, lots of different ways to define that one, but basically the latency or lead time to code getting deployed. For our purposes, we'll take each PR, and we'll measure the timeframe from the first commit in the PR until it was merged.

Okay, so here's what we're seeing for changes in PR throughput, and let me explain this chart. Uh, every data point here is a snapshot of a given company on a given week.

The X-axis is the company's AI adoption rate that we discussed earlier. The Y-axis is the company's average PRs per engineer that week. So you can see here a clear correlation between AI adoption and PR throughput.

The average trend here is about a 2X change as you go from zero to full adoption. So on average, a company should expect to double their PR throughput if they go from not using AI at all, which not really anybody's doing anymore, to 100%, uh, adoption of AI coding tools.

Now, we also see some gains in cycle time, so more work is happening and it's happening faster. This is similar to the previous chart, but now on the Y-axis, we're looking at median cycle time for PRs merged each week instead of PR throughput.

Uh, this is a cool chart. As an aside, I like the cycle time distribution because you can see these two clear bands, uh, horizontally. So that lower horizontal cluster, that corresponds to tasks that take less than a day, and then you see sort of a valley, and then there's a band in the middle for tasks that take

about two days. Then there's a long tail of stuff going up the Y-axis that takes much longer. I've truncated it here because, as we all know, some things can take, uh, quite a while [laughs] to, to get merged.

Um, but, you know, what's exciting here is the average trend is a 24% decrease in cycle times as you go from 0% to 100% adoption of AI coding tools.

So big picture, this is good news for productivity gains, and maybe you're seeing these things in your own organization. But, uh, what about the side effects? We all know there's no free lunch, so what other things change as you go through an AI transformation?

Well, one thing we've observed is that PRs are getting bigger. So here's a plot like the previous ones I've showed, except now the Y-axis is PR size. So on average, teams that have fully adopted AI coding tools are pushing PRs that are 18% larger in terms of net lines of code added.

Now, that size change is due much more, uh... You know, when I say net, it's due more to additions than deletions. So that means that the combined change is primarily coming from net new code, not necessarily just, uh, you know, fully rewritten or heavily reworked code.

Uh, another kind of interesting detail is that the average number of files touched is about the same, so this change is more about code that's, uh... It's more thorough or maybe just more verbose, but it's not the case that AI is touching more files and changing code in more different places in, in the code bases.

It's largely happening within the same files. Well, now, if teams are pushing more PRs and writing and merging them faster, and the PRs are getting bigger, then you might be wondering about quality.

So are we seeing effects on quality as we use more AI and push code faster? Well, right now the answer is not really. We're not really seeing any big effects.

We've looked at bug tickets created, and we looked at rates of PR reverts, code that had to be rolled back, and we haven't found any statistically significant relationship with the rate of AI adoption.

Uh, interestingly, we have found increases in the rates of bugs resolved. Uh, when you dig into the data, you find this is because, um, teams are disproportionately using AI to tackle bug tickets in their backlog.

So you see a lot more, uh, bug tickets being, um, uh, addressed by AI, but not necessarily being caused by AI. Uh, this makes sense. You know, bugs are often well-scoped verifiable tasks that AI coding tools can be set up well to succeed at, and we're, we're seeing, uh, a lot of people having success throwing AI at,

at those kinds of tasks. Uh, but basically there's, there's no smoking gun on quality yet, though, you know, we're gonna keep digging in here, especially as usage of, of asynchronous agents grows.

All right, last question. What if what you're seeing at your org doesn't align with the kind of results we've been talking about here so far? You know, what if you're listening to this and it is just not your reality?

Well, I think I've made it clear, uh, so far that the most important thing to focus on first is adoption. You're not gonna see gains until you get folks using these tools at scale.

I think that's common sense. But maybe you are seeing high adoption, and you're still not seeing the kind of productivity gains that all your friends on LinkedIn are crowing about.

So what's going on? Well, we've looked at a lot of things here, and there's plenty more to investigate, but I'd like to share one that's particularly interesting, and, uh, that's code architecture.

By code architecture, uh, what I mean is how are the code for your products and services organized across your repositories? So, uh, think about code being organized into monorepos versus polyrepos, and that arrangement of, of your code, it could be indicative of monolithic services versus microservices.

It could be the difference between a centralized versus a more federated product strategy. Uh, and the way that we actually measure this, you know, one key metric for understanding it is active repos per engineer.

This is actually a pretty straightforward one. It's just how many distinct repos a typical engineer, uh, pushes code to in a, in a given week. One really cool thing about this metric is that it's scale independent.

So it turns out that, you know, by computing this per engineer, normalizing by the number of engineers, you remove any correlation with the size of, of the company, uh, with the size of the team.

So in other words, this metric, it tells you something about the shape of the code that your engineers have to work with on a daily and weekly basis, and it tells you that regardless of how big your company is.

So, you know, this metric that, that I'm introducing here, this is what the distribution of that metric looks like. Uh, here's a probability distribution across the companies in our study.

The more centralized architectures you can see on the left, uh, and then there's a long tail of highly distributed architectures to the right. And then more balanced architectures, you know, balanced and lightly distributed line between these two extremes.

So we've, we've got these four regimes as you increase, um, the active repos per engineer. So, you know, here's where it gets really interesting. So remember those 2x gains in PR throughput that I showed you before?

Here's a flashback. Remember this? Uh, well, if we take this plot, you, you know, take all these data points, all these different companies, and you segment on, um, this active repos per engineer,

we've got, you know, four different regimes that we can do this analysis in. So we've got centralized, balanced, distributed, and highly distributed. And if we perform that same analysis, we see big differences.

So looking at that top row, you can see centralized and balanced code architectures, uh, they trend more like 4x, not like 2x, so they're doing much better than the average.

And the distributor architecture there, uh, in the, the lower left-hand corner in the teal, that, that looks more like that global 2x trend that we see when you look at all the data.

What's really interesting is this highly distributed case. There's essentially no correlation here between AI adoption and PR throughput. Um, and actually, the, the weak trend that does exist is actually slightly negative.

So w- what's, what's going on here? Like, why are teams with highly distributed architectures struggling? They don't seem to be getting real gains, at least not on average from AI.

Well, a big part of what you're seeing here is really the problem of context. So most of today's tools are really set up, uh, best to work with one repo at a time.

You know, we've used these. Uh, you know, you, you pick a repo and, and you dive in. And combining context across repos, it's often challenging. It's challenging, uh, for humans as well as for coding tools and for agents.

Uh, moreover, the relationships between these repos and the systems and products they relate to, they're often not even written down very clearly. They might be largely locked in the heads of senior engineers, but definitely not accessible often to coding tools and agents.

So it's gonna take some time for, for teams to invest in the context engineering that's needed here. It's an interesting challenge, and, uh, especially, you know, in light of the fact that, uh, a lot of folks are saying, uh, you may have heard this too, that microservices are the right way to go for AI native development.

So I could see a world certainly where we solve these context challenges, we adopt autonomous agents at scale, they're set up for success and this whole thing flips, and this highly distributed category becomes the most productive way to do things.

But right now, this is what we're seeing out in the world. Um, as an aside, another thing you may notice here is that all of these distributions, they, you know, as you go from the most centralized to most distributed, uh, these, uh, you know, this, um, PRs per engineer, uh, shifts upward.

Uh, you know, what's happening is the absolute number of repos, uh, increases as architectures get more distributed. Basically in a highly distributed architecture, it just takes more PRs overall to get things done due to things like migrations, cross-repo coordination.

And I bring this up because this is one of the many reasons why counting PRs in the absolute sense isn't, isn't a great metric. You really need to be tracking change in PR throughput to understand productivity, uh, because these things vary due to, to factors like architecture choices.

Okay. So that's it. Uh, to recap, you know, probably not news to anyone, uh, watching this, but AI coding tools are being used in a big way. Autonomous agents though, not so much.

It's still, uh, still early days. Uh, we're seeing big productivity gains with mo- more code being shipped and faster. Even if all you're using is interactive AI coding tools like Copilot, Cursor, and Claude Code, you feel like maybe, you know, you're not, uh, as up on agentic, you know, fully autonomous agentic coding as you ought to be.

2x change in PR throughput, uh, should be your, your expectation. You should, you should be seeing that or more. Um, but also you should expect bigger PRs. Uh, but maybe we can all ease up on some extreme quality anxiety.

Like we wanna keep an eye on that, but we're just not seeing big issues there, at least not yet. And finally, there are a lot of reasons why your mileage may vary and we're gonna continue looking at this.

But one place you can start is to think about your code architecture, how it might be holding you back, what you can do, um, you know, to, to compensate for some of the context limitations you have, and ultimately try to unlock some of those, uh, the sweet AI productivity gains.

So that's it. That's all I've got. I'm Nicholas Arcolano, head of research at Jellyfish. Thank you so much for listening.