1
00:00:03,201 --> 00:00:07,461
Hi, I'm Steve Jamison, AI Research and Engineering Leader at Jellyfish.

2
00:00:08,021 --> 00:00:11,061
I'm here to show you why tokens are the cheap part.

3
00:00:11,481 --> 00:00:14,621
The research that I'll present is drawn from the Jellyfish platform,

4
00:00:14,621 --> 00:00:20,321
the industry's largest set of engineering workflow data joined to AI telemetry data.

5
00:00:21,761 --> 00:00:24,401
It's no secret that AI costs are climbing.

6
00:00:24,821 --> 00:00:27,221
They're up 60x from this time last year.

7
00:00:27,461 --> 00:00:29,781
The use cases are wide and varied.

8
00:00:30,101 --> 00:00:32,461
The models and harnesses are getting much better.

9
00:00:32,781 --> 00:00:34,461
The implications are exciting.

10
00:00:34,841 --> 00:00:38,781
But discussions around budget and spend are not as much fun.

11
00:00:39,101 --> 00:00:42,861
These conversations involve your peers and senior leadership and the board.

12
00:00:42,861 --> 00:00:47,661
not always the easiest audiences, especially when you don't have the right data.

13
00:00:48,121 --> 00:00:57,601
And it's real money these days. Output is up too. Merged PRs per engineer are up 43% over the same

14
00:00:57,601 --> 00:01:02,961
one-year horizon. This is phenomenal. We've not seen productivity inputs like this,

15
00:01:03,181 --> 00:01:09,721
especially on this timescale. There's plenty of research and anecdotal data to know

16
00:01:09,721 --> 00:01:13,281
that AI and agentic coding are driving these throughput gains.

17
00:01:13,581 --> 00:01:15,481
Heck, even our charts show the correlation.

18
00:01:15,981 --> 00:01:18,361
Notice where the inflection points are.

19
00:01:19,001 --> 00:01:21,541
AI spend increased in early 2026

20
00:01:21,541 --> 00:01:24,501
when throughput gains start a steady increase of their own.

21
00:01:26,201 --> 00:01:28,361
There's a tension with these two numbers, though.

22
00:01:28,921 --> 00:01:30,321
How do you reconcile them?

23
00:01:30,861 --> 00:01:32,381
Spend is up 60x.

24
00:01:32,801 --> 00:01:34,821
Throughput is also up 43%.

25
00:01:34,821 --> 00:01:37,021
Is that good? Bad?

26
00:01:37,761 --> 00:01:39,521
43% is a big number.

27
00:01:39,721 --> 00:01:57,741
But so is 60x. How do you know if you're spending enough? Is there more productivity gain locked behind a checkbook? Should we increase spend 50% 2x more? Or how do you know if you spent too much?

28
00:01:58,321 --> 00:02:00,341
I'm sure many of you have felt the tension here.

29
00:02:00,681 --> 00:02:03,941
The CFO or board are asking for the receipts of the spend.

30
00:02:04,401 --> 00:02:05,501
Can you show them?

31
00:02:06,161 --> 00:02:09,701
Will merged PRs are up 43% satisfy them?

32
00:02:10,761 --> 00:02:14,461
Also, what happens if you cap use and spend?

33
00:02:15,101 --> 00:02:18,461
Are you stopping from setting tokens on fire?

34
00:02:18,681 --> 00:02:21,121
Or are you stumping further efficiency gains?

35
00:02:21,821 --> 00:02:24,481
You can ask peers at other companies what they're paying.

36
00:02:24,881 --> 00:02:26,821
For many, it's still a moving target, though.

37
00:02:27,621 --> 00:02:30,641
But how do you equate that to your org?

38
00:02:31,041 --> 00:02:34,321
Surely, just comparing their bill to yours isn't adequate.

39
00:02:34,701 --> 00:02:42,761
We've heard the reports of companies spending $100K, $500K, even millions on AI,

40
00:02:44,061 --> 00:02:47,941
setting, then resetting their AI budgets as reality sets in.

41
00:02:48,981 --> 00:02:51,781
Ultimately, these two numbers can't tell the story.

42
00:02:52,081 --> 00:02:55,681
And even together, they missed the biggest line on the cost side.

43
00:02:58,381 --> 00:03:03,001
These two numbers leave out the biggest cost of all, developer salary.

44
00:03:03,481 --> 00:03:07,721
AI is making developers more productive on a scale we haven't seen before.

45
00:03:08,081 --> 00:03:10,761
It isn't just another tool, it's intelligence.

46
00:03:11,641 --> 00:03:15,661
This intelligence dramatically changes how much each dev can produce.

47
00:03:16,341 --> 00:03:18,761
That's why we need to think about its cost differently.

48
00:03:19,401 --> 00:03:24,461
Not often the side is overhead, but it's part of what it takes to produce each PR.

49
00:03:25,101 --> 00:03:28,861
We're going to call this intelligence costs of a PR.

50
00:03:32,761 --> 00:03:40,281
What you pay your engineers plus what you pay for AI, put them together in one cost bucket.

51
00:03:40,721 --> 00:03:46,761
AI is about 2.5% of the cost to produce another unit of merged code.

52
00:03:47,201 --> 00:03:50,381
Salary is the other 97.5%.

53
00:03:51,161 --> 00:03:56,801
This is in line with other reported figures, and this data is from August 2026.

54
00:03:58,921 --> 00:04:06,161
Here, tokens are the cheap part. But that's current relative costs, not dollars. Let's

55
00:04:06,161 --> 00:04:09,781
look at what happened to the cost of merged PR over the last year.

56
00:04:16,621 --> 00:04:21,061
The dollar cost of a merged PR fell 26% over the last year.

57
00:04:21,541 --> 00:04:24,661
Not PRs created, but PRs merged.

58
00:04:25,061 --> 00:04:29,341
Review time has factored in, and some level of quality gates has been cleared.

59
00:04:30,881 --> 00:04:37,061
That decline in cost per merged PR includes a 60x increase in AI costs over the same time frame.

60
00:04:37,361 --> 00:04:39,521
The change is nearly universal, too.

61
00:04:39,781 --> 00:04:44,421
83% of the companies we studied showed a drop in cost per PR.

62
00:04:44,941 --> 00:04:46,361
What does this tell us?

63
00:04:46,641 --> 00:04:53,641
The additional throughput gains are spreading salary thinner across PRs,

64
00:04:53,761 --> 00:04:57,441
and this is happening faster than the AI costs are increasing.

65
00:05:00,561 --> 00:05:02,361
What's the range in token spend?

66
00:05:02,781 --> 00:05:09,021
We ranked companies by token usage, then grouped them into quintiles or fifths.

67
00:05:09,021 --> 00:05:25,761
The range here is quite large. The lightest group, about 0.3% of the cost of a PR is AI. This represents about 60 million tokens per dev per month.

68
00:05:25,761 --> 00:05:32,641
The top 20% of AI spend per dev, we're going to call the heaviest spend group.

69
00:05:33,301 --> 00:05:37,081
There, 8.5% of the PR cost is AI.

70
00:05:37,401 --> 00:05:41,761
This represents about 2.5 billion tokens per dev per month,

71
00:05:42,141 --> 00:05:45,921
about two orders of magnitude more than the lightest group.

72
00:05:47,281 --> 00:05:52,541
We'll use these groups on the next slides to see how token spend influences other metrics.

73
00:05:54,201 --> 00:06:03,341
First, PR throughput measured as merged PRs per dev, split by the same AI spend quintiles on the previous slide.

74
00:06:03,701 --> 00:06:12,061
The bottom four-fifths of spenders sell PR throughput gains of around 40%, give or take.

75
00:06:12,381 --> 00:06:14,021
But the heavy spend group

76
00:06:14,041 --> 00:06:17,341
improved more at a 67% increase.

77
00:06:18,001 --> 00:06:22,381
More AI spend led to more throughput gains.

78
00:06:25,621 --> 00:06:27,101
let's look at the cost per PR.

79
00:06:28,001 --> 00:06:30,921
Positive results are seen again across the board.

80
00:06:31,361 --> 00:06:34,921
Cost per PR is dropping for all AI usage groups.

81
00:06:35,781 --> 00:06:44,141
However, again, the heaviest fifth, so the companies that spent the most, improved the most.

82
00:06:44,461 --> 00:06:50,541
They saw a cost per PR decrease of 32% over the last year.

83
00:06:51,421 --> 00:06:57,061
The other four groups saw about a 25% decrease in cost per PR.

84
00:06:58,661 --> 00:07:04,081
Investing and spending more on AI fueled more of an improvement in cost per PR.

85
00:07:04,081 --> 00:07:07,361
compared to peers who leaned in less.

86
00:07:07,921 --> 00:07:14,521
So we saw that the same as we saw in the population average,

87
00:07:14,741 --> 00:07:20,241
cost per PR changes are driven largely by PR throughput, not by AI spend.

88
00:07:20,601 --> 00:07:25,401
The heavy spenders experience PR throughput gains of 67%

89
00:07:25,401 --> 00:07:28,201
compared to about 40-ish percent for their peers.

90
00:07:29,321 --> 00:07:33,101
Spreading the same salary across more PRs in the month

91
00:07:33,101 --> 00:07:37,181
has more of an effect than the rising AI costs,

92
00:07:37,481 --> 00:07:40,441
because tokens are the cheap part.

93
00:07:42,341 --> 00:07:47,241
This is not to say that orgs should light tokens on fire

94
00:07:47,241 --> 00:07:48,601
and spend with a banding.

95
00:07:48,861 --> 00:07:51,861
Here, we're looking at tokens used on the horizontal axis

96
00:07:51,861 --> 00:07:54,221
and cost per PR on the vertical axis.

97
00:07:54,601 --> 00:07:56,541
There's a steep drop at the beginning

98
00:07:56,541 --> 00:08:01,641
where relative small AI spend results in large gains initially.

99
00:08:01,641 --> 00:08:07,341
As token usage increases, the cost per PR improvement decreases.

100
00:08:07,761 --> 00:08:09,281
This is diminishing returns.

101
00:08:09,721 --> 00:08:15,321
I want to stress, though, diminishing returns is not the same as no returns.

102
00:08:16,501 --> 00:08:25,261
We're seeing that the top 10% are spending 8x the tokens compared to the median

103
00:08:25,261 --> 00:08:30,761
and achieving a 25% lower cost per PR.

104
00:08:31,701 --> 00:08:41,841
There will be a time when tokens aren't the cheap part, but by and large, for most companies out in the wild, we're not close to that threshold.

105
00:08:42,521 --> 00:08:47,941
More tokens leads to more PRs, which leads to a lower cost of delivery.

106
00:08:55,661 --> 00:08:57,801
So far, we've talked about companies.

107
00:08:58,681 --> 00:09:03,521
Next, I'd like to take a tangent and talk about a study we did with developers.

108
00:09:04,721 --> 00:09:14,881
Here we looked at developer weeks with heavy token use compared to the same developer weeks with lower token use.

109
00:09:16,961 --> 00:09:21,461
Here we saw the same phenomenon just at the individual level.

110
00:09:21,461 --> 00:09:28,801
Compare each developer's heavy agent skill weeks with their own weeks without skills.

111
00:09:29,701 --> 00:09:44,881
They spend about twice as much on tokens, merge 27% more PRs, and reach PR costs about 18% less once salary and tokens are counted together.

112
00:09:49,441 --> 00:09:57,501
there are two fair curiosities i get when i talk about this work first does faster work

113
00:09:57,501 --> 00:10:04,201
result in worse work and second are the more prs

114
00:10:04,201 --> 00:10:10,741
the throughput gains that we're seeing are they being driven by smaller chunks of work

115
00:10:12,921 --> 00:10:17,681
First curiosity is cheaper, faster work, worse work.

116
00:10:18,001 --> 00:10:22,401
If AI makes PRs faster and cheaper, are we just shipping more mistakes?

117
00:10:22,981 --> 00:10:29,021
One proxy for this is reverts. Merge PRs that undo an earlier PR.

118
00:10:29,561 --> 00:10:34,221
The heaviest fifth started lowest before most of the token growth.

119
00:10:34,681 --> 00:10:38,601
And revert rates dropped slightly for every group.

120
00:10:38,601 --> 00:10:42,481
though these deltas are just slightly below the noise in the data.

121
00:10:42,781 --> 00:10:45,661
It's not evidence of improvement here.

122
00:10:46,041 --> 00:10:54,081
However, more importantly, we're not seeing any worrying signs of rampant or severe quality issues

123
00:10:54,081 --> 00:11:00,241
with higher token spends and faster PR throughput.

124
00:11:00,241 --> 00:11:11,641
it. The second curiosity, is throughput driven by smaller PRs? And this is demonstrably no,

125
00:11:12,401 --> 00:11:21,321
not smaller PRs. In fact, AI is driving an increase in lines of code generated per PR.

126
00:11:21,881 --> 00:11:27,901
This is no surprise to the devs in the room. Raise your hand if you've rolled your eyes at

127
00:11:27,901 --> 00:11:30,661
for both unnecessary inline AI comments,

128
00:11:31,111 --> 00:11:34,121
or a file full of mostly irrelevant tests.

129
00:11:34,881 --> 00:11:38,241
Gen AI likes to generate code.

130
00:11:39,341 --> 00:11:41,881
Also, the bar to create is low for devs,

131
00:11:41,941 --> 00:11:44,481
and there's a pool to do more per PR

132
00:11:44,481 --> 00:11:46,141
instead of spending up another ticket.

133
00:11:46,501 --> 00:11:48,601
It's just one, maybe two prompts,

134
00:11:48,801 --> 00:11:50,221
and I'll handle this edge case,

135
00:11:50,361 --> 00:11:52,121
or I can combine this and that element.

136
00:11:52,601 --> 00:11:53,881
Fewer PRs to wrangle.

137
00:11:55,001 --> 00:11:57,721
This doesn't mesh well with the PR review stress

138
00:11:57,721 --> 00:11:59,321
folks are feeling these days though.

139
00:12:02,761 --> 00:12:05,841
Also, AI is just becoming more capable,

140
00:12:06,221 --> 00:12:07,721
so we're getting more ambitious.

141
00:12:08,361 --> 00:12:12,161
It is beyond this talk to dig into what's actually driving

142
00:12:12,161 --> 00:12:15,761
those level of lines of code changes.

143
00:12:16,641 --> 00:12:19,261
However, the increase in PR throughput

144
00:12:19,261 --> 00:12:21,861
is not coming from smaller PRs.

145
00:12:22,141 --> 00:12:26,021
This is intuitive for most, but it's still good to verify.

146
00:12:27,781 --> 00:12:29,641
Back to the opening question.

147
00:12:30,101 --> 00:12:34,881
Is 60x more spend worth 43% more output?

148
00:12:35,321 --> 00:12:37,341
You can't tell in isolation.

149
00:12:37,901 --> 00:12:39,521
You should track both.

150
00:12:39,721 --> 00:12:42,901
They are important numbers, but they're supporting metrics.

151
00:12:43,541 --> 00:12:51,921
AI spend per PR is the intelligence that you're buying as a resource to help make your devs go faster.

152
00:12:52,461 --> 00:12:56,061
PRs per engineer is what you're getting with that spend.

153
00:12:57,141 --> 00:13:00,161
However, looking at those hides the trade-off.

154
00:13:05,381 --> 00:13:10,321
But put salary into the equation and take the ratio of cost to throughput.

155
00:13:10,661 --> 00:13:12,081
And there you get your answer.

156
00:13:12,481 --> 00:13:18,481
Cost per PR is down 26% over that same time period.

157
00:13:18,761 --> 00:13:24,541
It's clear AI is driving cost efficiencies despite the increasing bills.

158
00:13:27,921 --> 00:13:34,881
and the AI spend needed to get those efficiencies is small, relatively speaking.

159
00:13:35,321 --> 00:13:41,561
AI is still a sliver of what a PR costs, 2.5% on average,

160
00:13:42,061 --> 00:13:44,701
under 10% for the heavy spender group.

161
00:13:45,421 --> 00:13:47,281
Tokens are the cheap part.

162
00:13:48,001 --> 00:13:52,101
In the range that we see, more spend kept paying.

163
00:13:52,101 --> 00:14:00,461
heavier token users saw 67% PR throughput gain compared to 40% for their lower spend peers.

164
00:14:01,101 --> 00:14:09,141
This drove cost per PR down 32% compared to about 25% for the lower spend peers.

165
00:14:09,641 --> 00:14:13,461
Looking at the PR cost to token ratio curve,

166
00:14:13,461 --> 00:14:27,501
The top 10% of company months by token usage resulted in 8x the tokens, but 25% lower cost per PR compared to the median.

167
00:14:28,901 --> 00:14:33,761
AI is making developers more productive on a scale we haven't seen before.

168
00:14:34,081 --> 00:14:41,301
It isn't just another tool, it is intelligence, which changes how much each dev can produce.

169
00:14:42,081 --> 00:14:44,521
That's why we need to think about its costs differently.

170
00:14:44,921 --> 00:14:49,861
Not off to the side as overhead, but as part of what it takes to produce each PR.

171
00:14:50,981 --> 00:15:02,241
You need to add the AI spend and the human spend, the human salary spend, together to calculate your real costs per PR.

172
00:15:02,881 --> 00:15:05,641
This isn't to say spend frivolously.

173
00:15:06,721 --> 00:15:09,801
Sure, you should watch for runaway jobs.

174
00:15:10,621 --> 00:15:12,341
You should track your AI spend.

175
00:15:12,641 --> 00:15:18,521
I'm not saying don't look at how much AI spend is going in each PR.

176
00:15:19,021 --> 00:15:21,861
I'm saying don't let that be your driving metric.

177
00:15:22,901 --> 00:15:28,001
A new framing might steer how and when you think about other investments,

178
00:15:28,021 --> 00:15:32,221
like integrating model routers, post-training your own model,

179
00:15:32,881 --> 00:15:35,301
implementing token limits, et cetera.

180
00:15:36,441 --> 00:15:46,781
Will you get more efficiencies out of trying to control AI costs or by continuing to invest in things that increase throughput, like more AI enablement?

181
00:15:47,181 --> 00:15:58,721
Either way, track the total intelligence costs, human plus artificial, per PR every month next to that AI bill and next to the PR throughput.

182
00:15:59,661 --> 00:16:03,481
How are you trending? Is throughput continuing to improve?

183
00:16:03,481 --> 00:16:07,181
What about those cost efficiencies, the cost to deliver?

184
00:16:07,901 --> 00:16:12,801
Watch reverts and quality metrics beside all of this as a counterweight.

185
00:16:13,081 --> 00:16:13,861
We don't

186
00:16:14,161 --> 00:16:16,041
AI slot mucking up the gears.

187
00:16:16,681 --> 00:16:25,861
So heading into budget season, bring cost per PR to the table. Not just your token bill, not just the throughput bump.

188
00:16:26,161 --> 00:16:32,661
The bill tells you what you spent. Cost per PR tells you whether it worked.

189
00:16:32,661 --> 00:16:37,081
And right now, for most companies, it says keep investing.

190
00:16:37,981 --> 00:16:41,801
Tokens are the cheap part to get efficiency gains.

191
00:16:44,401 --> 00:16:46,681
Well done. You made it to the end.

192
00:16:47,481 --> 00:16:55,941
Here's how you can stay up to date with our latest research, which again draws from the largest engineering workflow and AI telemetry data set in the industry.

193
00:16:56,181 --> 00:17:01,381
You can follow me on LinkedIn, my colleagues, or Jellyfish.

194
00:17:02,081 --> 00:17:03,801
subscribe to our sub stack,

195
00:17:04,761 --> 00:17:07,641
or you can check out our Jellyfish blog on the website.

196
00:17:08,541 --> 00:17:11,641
If you want help tracking your org's engineering workflow

197
00:17:11,641 --> 00:17:15,761
and AI effectiveness, including cost per merged PR,

198
00:17:16,201 --> 00:17:19,381
go to our website and book a demo or reach out to me

199
00:17:19,381 --> 00:17:21,481
and I'll get you connected to the right folks.

200
00:17:22,461 --> 00:17:24,001
Again, I'm Steve Jamison,

201
00:17:24,261 --> 00:17:26,421
AI research and engineering leader at Jellyfish.

202
00:17:26,681 --> 00:17:27,781
Thanks for watching.

203
00:17:28,401 --> 00:17:28,681
Cheers.