Better AI Models Won't Fix Your Engineering Team — Nicholas Arcolano, Jellyfish

I'm, uh, I'm Nicholas Arcolano. I'm the head of AI and research at Jellyfish. It's great to be back here at AI Engineer. And, uh, today, I wanna talk about why, if you're not getting the outcomes that you want with AI, uh, chances are it's actually not the model, the, the problem is you. Um, since we're here, uh, you know, we're doing this, this finance theme thing. We're in the leadership track, we're gonna start with a golf analogy. Seems appropriate. Uh, is anybody here a golfer? Any golfers? Anybody? A few?

Okay. Um, so yeah, I... If you don't golf, bear with me. I apologize. If, if you do golf, I also apologize. It's a very painful sport. Um, I love golf. Uh, I'm a bad golfer. Um, it's part of the culture to say you're a bad golfer, but I, I genuinely am a bad golfer. And when you're out there, you just wanna make the pain go away, and one of the things you think of is like, "Oh, could... Should I just buy better clubs, right? That would make this easier." Uh, and then you get an instructor, and he's like, "No," like, "you're the

problem. You don't need better clubs," right? Uh, it's about the fundamentals. But, you know, I'm a, I'm a data guy, so I'm like, "I don't, I don't know. Is that really true?" Uh, if you look at the data, like, clubs matter. Clubs have gotten better. So what you're looking at here, this is the average driving distance for the PGA Tour, uh, from 1980 to the present. In 1980, I was shocked. I actually didn't know this. The average was two hundred and fifty-seven yards. Like, I've hit the ball that far, and as I mentioned, I'm a terrible golfer. Uh,

so it's gone up about twenty percent since 1980, and, you know, one bit of evidence that the clubs matter is, uh, this solid core ball era thing in the middle. By, uh, 2000, a bunch of technol- technological changes happened, and there was, like, a fifteen-yard bump, right, at that time. So, like, people didn't magically get better over that three-year span. Like, that was all technology-driven. And this, uh, chart reminded me a lot of what we're seeing with the model frontier, right? We're

seeing the steady advance of models getting better and better. This is a chart from Artificial Analysis. I assume everyone has seen one or many of these and checks them every time a new model comes out, right? So the models have gotten better and better. Um, I think that's uncontroversial. We can do things that we just could not do a year ago, right? So the models matter a lot, right? Yes? Well, I... So I don't know. The-- So this is interesting, right? So I work, um, I

work at this company called Jellyfish. Uh, we, uh, join AI coding tool and agent telemetry with engineering workflow data, engineering outcomes. Uh, we have the largest data set of its kind, hundreds of thousands of developers, about thirteen hundred companies, and we're tracking sort of all the things they're doing with AI, uh, in the development space. And one thing that I noticed in our data is, um, this is data from around the time when, uh, SpaceX announced its acquisition of Cursor, and we

noticed, uh, a huge bump in Grok usage, uh, and the biggest model change that we've ever seen, far faster than just normal engineers sort of changing their preferences and trying to seek out the best model for the job. And when we dug into it, what happened was Cursor was doing a lot of things with the UI and sort of promoting Grok up in the, uh, up in the UI, changing how Auto mode worked. And so, um, you know, it, it shot up very fast. And what was

exciting about this as being a, a data scientist is this is what we call, uh, kind of a natural experiment. There was this external shock, and a natural question to ask would be, "Well, what, what happened?" Right? Did output change, for instance? So what you're looking at here on this chart, this is change in, uh, merged PRs, uh, per engineer. And so over that time span I showed you, every dot here is an engineer, and the X-axis is how much

of their, uh, model share went from not using Grok to using Grok on a scale from zero to a hundred percent. The Y-axis is the relative change in their PR throughput relative to their colleagues at the same company. And the story here is a non-story. There's the, the cloud sort of centered around zero. There's no statistical correlation. Output didn't change. Sort of shunting everyone to Grok didn't really matter. So I left this experiment confused and wanting to know how

much does the model matter, and if it doesn't matter much, what are the other things that do actually matter? So that's what we're gonna spend the rest of the talk talking about. Uh, so here's what we're gonna do. We're gonna continue to focus on this output metric of merged PRs per engineer per week. We're gonna use a, a curated data set here. This is a subset of our data where we've got all the pieces we need to do this experiment. About fifty thousand engineers at four hundred and forty-seven companies.

Data from this year for Cursor and Claude Code users. And some of the stuff that matters, so you've got the tool. We're gonna put the harness and the model together because they're very intertwined. Select your clubs, okay? There's how well you use it. By that I mean things like, um, you know, skill use, Cursor rules, the, the harness customization, the tricks of the trade that we're all learning as AI engineers to make the way that we use these tools better. The golf equivalent would be, like, this is knowing how to get out of a bunker, which is not, like, a natural thing. It's a

thing you have to learn. There's how much you use it. So separate from these things, just raw amount of token spend. So golf, time on the course, right? Uh, and then there's factors like where you work, your environment, so your company and your team. This is the course itself, something you don't have control over as a golfer. And then there's you, the individual, all the things that you carry with yourself as an engineer that- Holding all these other factors constant that make you still distinct.

So this is the map for the rest of our talk. You put all this stuff together, you do a bunch of big statistical analyses, the details of which I would love to get into here, but would bore everyone except for statisticians like me. But this is the map. You've got about, uh, less than one percent the-- in terms of how your-- how much your PR throughput changes is controlled, explained by the tool. How well you use it, the skills part, that's about three percent. Uh, how

much you use it, that's like ten X of the tool. That's nine percent. Where you work is twelve percent. The biggest piece is you, the individual, and we'll go through each of these and explain a little bit about why this looks like this.

Um, on the outcome measure of PR merged, is there any kind of discerning between PRs that merge within five minutes versus like, you know, you have development-

N-no, there's no quality metric here, but quality is a thing we look at a lot, and I'm happy to talk more about that later. But good question.

Uh, so let's do the tool first. So we'll kind of speed through this. This is the least interesting, as you might guess, uh, um, from the title of my talk. But what happens when engineers switch tools? We saw what happened when they were kind of encouraged, let's say, to switch tools. But engineers, uh, switch the models they use all the time. They switch harnesses. So you can do things like look at change in the weekly merged PRs. You can compare an engineer to

themselves and look at what happens when they go all in on Opus versus Sonnet, switch to ChatGPT. Um, what happens if they switch from Cursor to Claude Code, which we have a lot of data around people who've made that switch. What happens when, uh, you switch from working totally in an autocomplete mode to an agentic mode? The story's kind of the same. Statistically, it's kind of a nothing burger. There's not a lot here in terms of explaining any meaningful change in, um, the amount of PR throughput that you get.

What about how well you use it? This is three percent, so a much bigger piece of the pie. Let's talk about skill use. This one's interesting because skills-- This should be bigger. Skills, actually, we find matter a lot. One of the reasons it doesn't matter more is, and I don't-- this might surprise some folks, people don't use skills that much like in the real world outside of, like, this, this wonderful land of AI engineering conferences that, that we're inhabiting. Uh,

this chart shows, um, you know, skill use engineer weeks over time. So proportion of engineering weeks that have, you know, low sk- zero skill use, a low level, a medium level, and a high level of skill use. And as of September, you know, almost half of the weeks here shown had essentially no skill use, uh, and only twelve percent had-- are using skills at a level that I think we would consider as, like, what you should be doing in terms of modern AI engineering.

And the reason that this is so important is because skill use, when it's prevalent, moves output way more than model choice. So I'm showing here again those results I flashed from earlier about changing model family as, as kind of a reference. But on the bottom here is level of skill use and the amount of effect it has in terms of changing the level of PR throughput, holding these other factors constant. And you can see you get, um, you know, when you get up to

thirty plus runs, uh, of using skills, uh, per engineer per week, that's a thirty percent bump just from that factor alone, holding other things constant. So it's a big effect.

All right, let's talk about how much you use it. This is consumption. This is tokens. This is active use days. Uh, this one, I could do a whole talk about this. In fact, I did a whole talk about this at the AI Engineer World's Fair, so I encourage you, if you find this interesting, to, uh, to take a look at that. Um, but I'll give you the one-slide version. So the data you're looking at here is from that talk. So e-- this is, um, about five hundred thousand developer weeks of behavior.

Each dot is a two percentile bin of, uh, folks that use the same level of consumption in that week, and it's their average token spend and their merged PRs per developer per week. And the basic story is more tokens, more throughput, and there's diminishing returns. So there's sort of a saturating curve. I likened it to the rocket equation in that talk. It's like rocket fuel. And what you get is, if you look at the top ten percent versus the median, uh, you can basically double

your output between the median and the top ten percent. Costs you about ten X the tokens. But more tokens mean more output.

Now we get to the bigger pieces of the pie. These are interesting. So there's the factor of, you know, where you work, your company, your team. Actually, if you tease out the factors, about a third of it's the team, two-thirds is the company. You know, your, your code base, your review process, um, your culture. So once you sort of fix the amount of consumption, you know, do these other things still matter?

One way to look at this is, you know, if you fix these factors and just look at the variability, you look at the spread of companies relative to the average company holding these other things fixed. Um, this is what it looks like. And one way to interpret this is, I just showed you that token curve. So you could spend ten X the tokens to move from the median up to the top ten percentile and get double the output. There's also a two X spread between the top ten percent company and the bottom ten percent company in this distribution holding your

consumption fixed. So in other words, your culture, your team, your company, there's a lot of variation just in those things, even for fixed AI spend. This matters a lot, and it's interesting for this group in the, in the leadership track here because this is what leaders can control. Like, this is our, our job is to control these things, even if maybe our CFO tells us we can't have any more tokens than we have, um, which hopefully they don't. So what about the biggest piece?

You, the individual, right? This is, this is three-quarters of this statistical model, so it matters a lot. Um, one way of thinking about this is this question that a lot of people have asked is, you know, does AI mean that you need different engineers? Do we need different people, right? Is it resorting the rankings? Is it changing what it means to be a good engineer? So, uh, you saw that distribution I just showed you with companies. If you do the same sort

of distribution for engineers, it's even wider. So there's even more variability, and that's why it's such a big factor in this, this statistical map we've, we've laid out for ourselves to explain the different levers available to us. Uh, it's even wider. It's, you know, three point five X spread between the bottom ten percent and the top ten percent, uh, instead of two X as for companies. And there's a lot to unpack here. This would be a whole another talk to unpack all the things that make different, uh, AI engineers, uh, better than

others. One thing that we can answer a little bit in the few minutes I've got left is, is AI changing who the most, uh, productive engineers are? So one way to look at this, here's a small study ranking ten thousand engineers, and we have long, uh, you know, longitudinal data, a long timeline. Jellyfish has been around ten years, so we have, you know, tons of history about how engineers behave. And if you look at, you know, how, um, top engineers rank, uh, you know, quarter over quarter,

six months over six months, uh, we can look at, you know, how much do your top engineers cycle in terms of productivity here, um, PR throughput we're still talking about, how much, uh, they cycle out of the top quartile. And if you compare that, uh, so about forty-four percent cycle out of the top quartile, uh, every half year. Uh, if you look at before AI versus after, it actually tightens a little. People get a little less mobile. It's forty-one percent. So, and we've done other studies like this. We're not

seeing, um, you know, a big resorting of the world order. Your, your best engineers before AI, by and large, are still your best engineers. So the, the sort of narrative that AI is leveling the, the playing field, uh, there's-- craft still matters for a lot, and that's why, part of why this factor is as big as it is.

So, you know, going back to this, this picture that I've, I've been showing you, when you add it all up, you know, what I want you to take away is, is these things on the left, you know, the tool especially, are, like, really exciting to talk about. It's the cool new tech. It takes up kind of all the air in the room, all the air on, on Twitter, and, like, that's exciting, and I love that stuff. Um, but we don't have a lot of control over that. I don't work at a foundation lab. I can't make the tools better. Uh, and as costs tighten, we have, um, less and less freedom maybe to buy the

tools that we want. But as leaders, the things we can control are on the other side of the, uh, the equation. It's, it's the people, it's the organization. And I think what's, you know, going back to our golf analogy, what's really exciting about that is unlike golf, you can change the course. You can change your environment. So you can, like, fill in water hazards or do things that they yell at you if you try to do on a golf course. But, um, those are things as leaders that are available to

us. So in closing, um, you know, if you're not getting the outcomes that you want, I would encourage you to think, you know, maybe it's not the model, you know, maybe it's you.

So

thank you. I appreciate that. That's all I've got. If you're interested, we publish monthly benchmarks with all of this data and more. Go to Jellyfish AI Engineering Trends. Uh, a lot of our research is published on our Substack. Uh, thank you.