1
00:00:00,641 --> 00:00:02,881
Not all GPUs are deployed the same.

2
00:00:03,441 --> 00:00:05,861
Thank you everyone for taking the time to be here today.

3
00:00:06,141 --> 00:00:07,061
My name is Oliver Kennon.

4
00:00:07,521 --> 00:00:10,101
I'm a research analyst at SemiAnalysis.

5
00:00:11,181 --> 00:00:15,901
Time and attention are the most valuable things any of us have, unless of course you're an

6
00:00:15,901 --> 00:00:18,201
early stage investor in open AI or Anthropic.

7
00:00:19,081 --> 00:00:21,481
With that, let's get into it.

8
00:00:21,481 --> 00:00:23,561
Not all GPUs are deployed the same.

9
00:00:23,761 --> 00:00:26,181
A GPU falls to the level of the operator.

10
00:00:27,761 --> 00:00:28,961
hardware gives you potential.

11
00:00:30,061 --> 00:00:34,261
capability is what determines how much useful work you actually are able to

12
00:00:34,261 --> 00:00:34,561
deliver.

13
00:00:35,001 --> 00:00:38,481
That leads us to the question I will attempt to answer with all of you today.

14
00:00:38,761 --> 00:00:42,241
Who can produce the intelligence you need at the lowest cost?

15
00:00:42,441 --> 00:00:43,141
Cost matters.

16
00:00:45,001 --> 00:00:50,321
today's presentation, I want to show you guys why and how operating capability changes

17
00:00:50,321 --> 00:00:54,041
that cost and why it matters in the commodity market that is compute.

18
00:00:56,961 --> 00:00:58,221
matters in a commodity market?

19
00:00:58,781 --> 00:01:04,521
A lot of tech investors or technologists may not be familiar with this framework, so a

20
00:01:04,521 --> 00:01:05,081
little crash course.

21
00:01:05,621 --> 00:01:09,281
In a commodity market, producers compete to deliver comparable output.

22
00:01:09,701 --> 00:01:13,481
A lower production cost gives more room to compete when prices fall.

23
00:01:15,121 --> 00:01:16,421
what a cost curve shows.

24
00:01:16,421 --> 00:01:20,621
producers lined up by what they spend to deliver the same usable work.

25
00:01:20,981 --> 00:01:23,721
If two producers earn the same price,

26
00:01:24,041 --> 00:01:27,021
the one spending less to deliver the work gets to keep more of it.

27
00:01:27,421 --> 00:01:28,201
Pretty simple, right?

28
00:01:30,581 --> 00:01:34,221
For us, intelligence is the economic output we care about.

29
00:01:34,861 --> 00:01:39,101
Eventually, our North Star metric will be price per successfully completed task

30
00:01:39,101 --> 00:01:40,321
at the required quality.

31
00:01:40,821 --> 00:01:42,421
Unfortunately, we're not quite there yet.

32
00:01:42,781 --> 00:01:44,381
That's a pretty hard thing to standardize.

33
00:01:45,281 --> 00:01:51,681
today price per token is a useful practical proxy especially for comparable workloads

34
00:01:51,681 --> 00:01:57,921
and quality requirements and gets us closer to delivered output much closer than the often quoted

35
00:01:57,921 --> 00:02:07,101
all you finance bros out here will have heard this a gpu dollar per hour price or as we view it a cost

36
00:02:08,301 --> 00:02:14,381
We view GPU hours as a measure of input, not output, the cost versus price distinction.

37
00:02:15,581 --> 00:02:19,181
prices are different from the producer's cost of doing the work.

38
00:02:22,301 --> 00:02:25,741
GPU hours are the input, intelligence is the output.

39
00:02:26,361 --> 00:02:30,281
On this slide we can visualize the inputs needed to create useful intelligence.

40
00:02:31,041 --> 00:02:36,381
We start with hardware and electricity, GPUs, for this presentation.

41
00:02:36,741 --> 00:02:41,741
Software and operations can be thought of as a multiplier on those first two inputs,

42
00:02:41,741 --> 00:02:44,701
with the end product being useful intelligence.

43
00:02:47,181 --> 00:02:50,901
The operator is who chooses and implements the software that serves the model

44
00:02:50,901 --> 00:02:52,701
and keeps requests moving through the system.

45
00:02:54,361 --> 00:02:58,941
The machines might be available, but you still need someone to make them deliver responses

46
00:02:58,941 --> 00:03:00,481
customers can actually use.

47
00:03:01,741 --> 00:03:05,921
What matters is how much useful work you get and whether it meets your requirements for

48
00:03:05,921 --> 00:03:06,881
the specified task.

49
00:03:07,901 --> 00:03:11,101
The conversion can differ with the same kind of chip.

50
00:03:11,401 --> 00:03:13,041
Let's look at that in the measured example.

51
00:03:14,761 --> 00:03:19,661
Here's a concrete example from our public inference X calculator.

52
00:03:20,001 --> 00:03:22,261
You guys can all check it out, semianalysis.com.

53
00:03:23,401 --> 00:03:30,201
Both estimates use B200 GPUs serving the same model with roughly 8,000 input tokens and

54
00:03:30,201 --> 00:03:32,741
1,000 output tokens, a pretty standard ratio.

55
00:03:33,741 --> 00:03:37,281
P50 target is 100 output tokens per second for each user.

56
00:03:38,001 --> 00:03:41,081
The serving stack is the software handling the model's responses.

57
00:03:41,461 --> 00:03:47,381
In the P90 scenario, one of these stacks gives about 200 generated output tokens per second

58
00:03:47,381 --> 00:03:49,641
while the other one gives about 300.

59
00:03:50,321 --> 00:03:56,321
That's a big difference and comes down to software, settings, and number of requests running.

60
00:03:56,561 --> 00:03:58,761
All choices made by the operator.

61
00:04:02,261 --> 00:04:07,601
Same chip, same target interactivity, about 50% more output.

62
00:04:07,901 --> 00:04:08,681
That's a lot.

63
00:04:08,901 --> 00:04:10,501
So what does that mean for the buyer's bill?

64
00:04:14,301 --> 00:04:17,381
What does that intelligence actually cost the customer?

65
00:04:17,381 --> 00:04:22,801
For this example, let's assume the hourly bill is the same for both deployments and

66
00:04:22,801 --> 00:04:24,001
both meet your requirements.

67
00:04:24,721 --> 00:04:28,821
At roughly one and a half times the output, your rental spend per acceptable token is

68
00:04:28,821 --> 00:04:29,841
proportionally lower.

69
00:04:30,221 --> 00:04:35,241
That is to say, the hour buys you more usable and useful output.

70
00:04:37,541 --> 00:04:38,741
counts as useful intelligence?

71
00:04:40,121 --> 00:04:42,341
intelligence has to do the job you hired it to do.

72
00:04:42,721 --> 00:04:43,361
Pretty simple, right?

73
00:04:43,361 --> 00:04:49,021
it, the answer needs to be good enough for the task, arrive quickly enough, and be available

74
00:04:49,021 --> 00:04:53,701
when you need it. Your information also needs to stay within the access boundaries you've

75
00:04:53,701 --> 00:05:00,021
agreed upon, aka it needs to be secure. This is an overlooked requirement that is not

76
00:05:00,121 --> 00:05:24,901
captured in a chip's performance metrics, but it is essential for a chip to be usable and useful. In our published security work, we found monitoring access that exposed another customer's logs and metrics. This is a case, and an unfortunately common one at that, where the GPs were fast, in the case I'm referencing really, really fast, which makes the whole thing sadder, and the service still failed horrific.

77
00:05:25,021 --> 00:05:30,421
Basically, ensuring cluster security takes ongoing work just as delivering the promised

78
00:05:30,421 --> 00:05:31,161
performance does.

79
00:05:34,461 --> 00:05:37,401
most cases, it is the operator that owns that responsibility.

80
00:05:39,581 --> 00:05:41,721
Someone has to make the system work in production.

81
00:05:42,781 --> 00:05:45,341
Suppose a faster serving update becomes available.

82
00:05:45,921 --> 00:05:50,541
Someone has to test it on the customer's workload, check whether the output still meets the requirements,

83
00:05:50,541 --> 00:05:54,741
and then decide when, where, and how to deploy it.

84
00:05:54,961 --> 00:05:57,061
That means checking the actual responses,

85
00:05:57,361 --> 00:05:58,381
testing it under load,

86
00:05:58,581 --> 00:06:01,121
and keeping a way to return to previous version

87
00:06:01,121 --> 00:06:04,081
if the update causes problems once it's deployed.

88
00:06:05,281 --> 00:06:07,401
Once it is deployed, someone has to maintain it

89
00:06:07,401 --> 00:06:08,821
and respond when issues come up.

90
00:06:11,881 --> 00:06:13,821
That work may sit with your provider,

91
00:06:14,181 --> 00:06:16,121
your own team, or both.

92
00:06:17,081 --> 00:06:19,741
Irregardless, you need to know who's responsible

93
00:06:19,741 --> 00:06:22,001
and whether they can actually do it.

94
00:06:24,501 --> 00:06:26,281
So how do we assess the operator?

95
00:06:27,181 --> 00:06:32,901
Semi-analysis cluster max, we're on version 3.0 at this point, assesses operating capability

96
00:06:32,901 --> 00:06:37,841
on managed GPU clusters, a couple hundred growing.

97
00:06:38,261 --> 00:06:40,321
Please reach out if you want to be on cluster max 4.0.

98
00:06:40,721 --> 00:06:44,981
We look at provisioning, reliability, monitoring, and security.

99
00:06:44,981 --> 00:06:50,741
hands-on testing, documentation, and customer feedback help us understand how the service

100
00:06:50,741 --> 00:06:54,121
runs, not just on a good day, but also when a machine fails.

101
00:06:56,741 --> 00:07:02,041
I want to take the chance to reiterate that cluster max ratings apply to managed clusters,

102
00:07:02,801 --> 00:07:04,781
different from bare metal, very different.

103
00:07:05,261 --> 00:07:08,421
With bare metal, more operating work usually sits with the customer.

104
00:07:08,901 --> 00:07:13,581
Applying a managed service rating there does not automatically cover every product a provider

105
00:07:13,581 --> 00:07:21,001
cells. The tiers are relative to peers. A little bit of a tongue twister. I nailed it.

106
00:07:21,521 --> 00:07:27,461
And pricing and availability are also important factors in this assessment. These are metrics

107
00:07:27,461 --> 00:07:36,261
that do not directly measure token output but are incredibly important. Assessment helps

108
00:07:36,261 --> 00:07:39,941
us understand who can deliver. Now we can move on to the next question. What does

109
00:07:40,221 --> 00:07:41,801
It's not actually cost to produce useful intelligence.

110
00:07:42,641 --> 00:07:46,601
Let's start with a compute bill and other costs that producers still have to cover.

111
00:07:49,441 --> 00:07:51,201
price and production costs are different.

112
00:07:51,621 --> 00:07:58,201
In our example, the GPU provider rents out hours, the inference producer buys those hours

113
00:07:58,201 --> 00:07:59,261
and sells responses.

114
00:08:00,341 --> 00:08:04,001
rental bill is revenue to the provider and an input cost to the inference producer.

115
00:08:04,321 --> 00:08:08,161
The GPU owner pays for the equipment, power, space and support.

116
00:08:09,121 --> 00:08:11,261
inference producer's full cost includes rent,

117
00:08:11,461 --> 00:08:15,681
plus whatever software, staff, and other services it pays for beyond that rental.

118
00:08:16,321 --> 00:08:19,441
If the inference producer owns its equipment instead,

119
00:08:19,761 --> 00:08:23,601
we spread the equipment investment over its working life and add running costs.

120
00:08:23,901 --> 00:08:27,241
What the producer charges and what it spends are separate numbers.

121
00:08:27,421 --> 00:08:29,321
To understand the cost of useful intelligence,

122
00:08:29,641 --> 00:08:32,661
we need to know how much useful work the spending actually produces.

123
00:08:33,761 --> 00:08:37,101
Better operations lowers that cost of useful intelligence.

124
00:08:37,101 --> 00:08:41,761
The B200 comparison showed why the amount of output matters.

125
00:08:41,961 --> 00:08:44,321
More acceptable responses from the same paid hours

126
00:08:44,321 --> 00:08:48,181
can spread the inference producer's rental expense over more useful work.

127
00:08:48,421 --> 00:08:50,541
For a producer that owns its GPUs,

128
00:08:50,641 --> 00:08:54,601
the same logic spreads the equipment investment and running costs over more output.

129
00:08:55,161 --> 00:08:58,561
Count all the spending needed to deliver the service over the same period,

130
00:08:58,741 --> 00:09:03,061
then divide by the output customers can actually use.

131
00:09:03,981 --> 00:09:07,741
That gives us cost for acceptable token for a comparable workload.

132
00:09:11,461 --> 00:09:13,381
operations also require investment.

133
00:09:14,501 --> 00:09:20,301
and update, deploying it safely, and keeping it working take people and money.

134
00:09:21,761 --> 00:09:24,381
extra output has to justify the extra spending.

135
00:09:25,681 --> 00:09:30,641
advantage can persist when a team repeats that work as software and workloads change.

136
00:09:30,641 --> 00:09:33,561
But one good benchmark doesn't establish it.

137
00:09:33,721 --> 00:09:35,621
You have to earn it every single day.

138
00:09:36,281 --> 00:09:41,421
Each new workload still needs checking, but a repeatable operating process, from what

139
00:09:41,421 --> 00:09:46,501
we've observed, can support a lasting cost advantage and a price advantage, too.

140
00:09:47,081 --> 00:09:51,501
Even if you do keep improving, though, your competitors are not sitting still.

141
00:09:53,041 --> 00:09:55,121
position on the cost curve is relative.

142
00:09:55,121 --> 00:10:00,361
This slide, out of all the slides, should make clear the importance of the commodity

143
00:10:00,361 --> 00:10:01,481
framework in all of this.

144
00:10:01,941 --> 00:10:05,721
Your position on the cost curve depends on what your competitors can deliver, not just

145
00:10:05,721 --> 00:10:06,101
yourself.

146
00:10:06,361 --> 00:10:07,621
It's a relative measure.

147
00:10:08,101 --> 00:10:09,621
Here's a simple hypothetical.

148
00:10:09,961 --> 00:10:14,621
We're comparing the same task, the same required quality, using made-up cost units.

149
00:10:15,741 --> 00:10:17,621
I like to imagine bars of gold.

150
00:10:17,961 --> 00:10:21,761
Your full cost per successful task falls from 10 to 8.

151
00:10:22,381 --> 00:10:25,641
In the same period, your competitors fall from 6 to 12.

152
00:10:28,681 --> 00:10:29,181
improved.

153
00:10:29,481 --> 00:10:30,041
That's awesome.

154
00:10:30,461 --> 00:10:31,621
But both of you did.

155
00:10:31,861 --> 00:10:36,721
And in this example, you started as the cheaper producer and ended as the more expensive one.

156
00:10:36,961 --> 00:10:41,941
That gives the competitor more room to lower its selling price, provided customers find

157
00:10:41,941 --> 00:10:43,361
the two services equally acceptable.

158
00:10:44,201 --> 00:10:47,081
Getting better does not automatically move you ahead.

159
00:10:47,081 --> 00:10:50,561
Instead, applying the commodity framework makes you realize that you have to compare

160
00:10:50,561 --> 00:10:56,381
your cost per useful work on a relative basis, comparing your own product with what your

161
00:10:56,381 --> 00:10:57,461
competitors can deliver.

162
00:10:58,181 --> 00:11:03,521
That comparison can be easy to overlook in moments like today where demand far outstrips

163
00:11:03,521 --> 00:11:05,881
supply and will for the foreseeable future.

164
00:11:07,501 --> 00:11:08,061
to three years.

165
00:11:09,181 --> 00:11:10,621
Visibility, good.

166
00:11:13,661 --> 00:11:15,001
can obscure those differences.

167
00:11:15,001 --> 00:11:20,981
When demand is strong and supply is constrained, producers with very different costs can find customers.

168
00:11:21,441 --> 00:11:24,021
Strong revenue can make those differences feel less urgent.

169
00:11:24,321 --> 00:11:28,681
If you're a CEO, why would you focus on your precious time and energy and money,

170
00:11:29,001 --> 00:11:33,821
shoring up your spot on the cost curve when you can be building a massive new data center?

171
00:11:34,301 --> 00:11:37,941
In moments like this, it's important to step back and have a little perspective,

172
00:11:38,621 --> 00:11:39,981
take a little historical perspective.

173
00:11:40,441 --> 00:11:44,281
Every commodity in the history of humanity has had swings in supply and demand.

174
00:11:44,281 --> 00:11:48,241
Just because we don't see when that will be yet does not mean that this cycle will be

175
00:11:48,241 --> 00:11:48,741
any different.

176
00:11:50,361 --> 00:11:54,561
Unless you're a total AI maxi, which some of us are, so I'm very open to being wrong.

177
00:11:54,681 --> 00:11:55,301
I'm wrong every day.

178
00:11:57,401 --> 00:12:02,061
But I would say if you're a business with leverage, you need to be especially conscientious

179
00:12:02,061 --> 00:12:02,441
of this.

180
00:12:02,801 --> 00:12:09,221
Being proactive with building your cost advantage might not be as sexy as making your top line

181
00:12:09,221 --> 00:12:12,941
bigger, but it matters deeply if you want to survive through a business cycle.

182
00:12:13,501 --> 00:12:15,661
That's why you want to build your operating capability

183
00:12:15,661 --> 00:12:18,201
and cost advantage while you have the opportunity,

184
00:12:19,341 --> 00:12:22,001
when the sun is shining, when you can raise capital.

185
00:12:22,661 --> 00:12:26,141
They matter even more when customers have more choices to make

186
00:12:26,141 --> 00:12:27,301
and prices come under pressure.

187
00:12:27,581 --> 00:12:31,021
So how do we make clear what the buyer actually gets?

188
00:12:32,961 --> 00:12:35,501
Comparable compute needs clear grades.

189
00:12:35,841 --> 00:12:39,901
Now let's look at the compute that an intelligence producer is buying.

190
00:12:40,061 --> 00:12:41,961
We've already seen that the same chip

191
00:12:41,961 --> 00:12:44,721
can deliver different results depending on how it's operated.

192
00:12:45,641 --> 00:12:51,501
And two offers for B200 GPU hours can include different services and leave different operating

193
00:12:51,501 --> 00:12:52,401
jobs with the customer.

194
00:12:53,161 --> 00:12:57,961
A commodity framework needs a way to describe these differences so buyers know which services

195
00:12:57,961 --> 00:12:59,101
they can treat as comparable.

196
00:12:59,621 --> 00:13:01,101
That's the job of a grade.

197
00:13:03,841 --> 00:13:07,781
grade describes a particular compute service and what its buyers can expect.

198
00:13:07,781 --> 00:13:11,361
It doesn't make intelligence interchangeable across every task.

199
00:13:11,801 --> 00:13:15,981
Once we've defined that service, we can ask what its rental price tells us.

200
00:13:18,061 --> 00:13:20,921
What does a rental price benchmark measure?

201
00:13:21,661 --> 00:13:26,661
A rental price benchmark summarizes the price of GPU hours within its defined market.

202
00:13:26,981 --> 00:13:30,961
It isn't the producer's production cost or a measure of intelligence delivered.

203
00:13:31,481 --> 00:13:33,821
We need to know where those prices come from.

204
00:13:33,821 --> 00:13:37,761
An advertised offer and a completed trade are not the same thing.

205
00:13:38,601 --> 00:13:47,521
Service scope, contract terms, whose prices are sampled, and how they're combined all shape that reference point.

206
00:13:47,941 --> 00:13:52,141
Operating quality can influence those prices without the index measuring it separately.

207
00:13:52,821 --> 00:13:56,621
Running a rental reference doesn't tell you how much useful work your deployment will produce.

208
00:13:57,001 --> 00:14:02,461
To use it as a hedge, you need to understand how it tracks the rent you actually pay or earn.

209
00:14:04,741 --> 00:14:06,861
What can compute futures hedge?

210
00:14:07,301 --> 00:14:09,781
There has been a lot of excitement lately about compute futures.

211
00:14:10,201 --> 00:14:14,941
For all of you that don't know, a futures contract sets a standardised term for future

212
00:14:14,941 --> 00:14:16,381
delivery or cash settlement.

213
00:14:16,681 --> 00:14:18,801
Cash settlement depends on a price reference.

214
00:14:19,221 --> 00:14:22,701
Buyers and sellers both face uncertainty about rental prices.

215
00:14:23,321 --> 00:14:26,621
Think back to the serving stacks and the monitoring failure.

216
00:14:26,921 --> 00:14:30,441
Assessing who runs the system helps describe the service buyers get.

217
00:14:30,441 --> 00:14:35,101
That helps us to find grades so buyers know which compute services are comparable.

218
00:14:35,521 --> 00:14:39,101
The benchmark then needs to track the rent they actually pay or earn.

219
00:14:39,601 --> 00:14:41,121
That's why I make this argument.

220
00:14:42,821 --> 00:14:48,081
Rating the operator is an essential building block for building a credible compute futures market.

221
00:14:48,441 --> 00:14:49,981
A rating informs that work.

222
00:14:50,241 --> 00:14:54,041
It doesn't by itself define a futures grade, establish a sound benchmark,

223
00:14:54,281 --> 00:14:56,901
or create enough trading to buy and sell easily.

224
00:14:57,121 --> 00:14:59,741
It also doesn't necessarily make a hedge effective.

225
00:15:00,741 --> 00:15:07,521
A rental price hedge can help offset rental price changes. It does not fix a failed deployment or ensure that your customer pays.

226
00:15:10,021 --> 00:15:22,181
Charging more than the index does not by itself make it a bad hedge. What matters is whether changes in the reference track changes in your rent. Unexpected changes in that relationship leave real price risk.

227
00:15:22,181 --> 00:15:27,241
risk, those tools can help you understand what you're buying and manage the rental price

228
00:15:27,241 --> 00:15:27,581
risk.

229
00:15:27,861 --> 00:15:31,301
For an intelligence producer, rent is one part of the cost.

230
00:15:31,621 --> 00:15:35,801
The operator still has to turn that compute into work that the customer can actually use.

231
00:15:37,481 --> 00:15:39,241
This brings us full circle.

232
00:15:45,901 --> 00:15:48,281
a GPU falls to the level of the operator.

233
00:15:48,281 --> 00:15:53,041
So who can produce the intelligence you need at the lowest cost?

234
00:15:53,261 --> 00:15:59,701
The producer who can repeatedly meet your requirements at the lowest full cost per successfully completed task.

235
00:16:00,141 --> 00:16:05,541
Count everything spent to deliver the service, then divide by the output that the customer can actually use.

236
00:16:06,221 --> 00:16:12,661
Operating capability affects both sides of that equation, what you spend, and how much more useful work you can deliver.

237
00:16:13,501 --> 00:16:16,161
Also, your position depends on that of your competitors.

238
00:16:16,161 --> 00:16:21,241
You can improve and still fall behind if your competitors improve faster than you.

239
00:16:21,941 --> 00:16:26,521
That's why understanding the operator matters through the cycle, not just at one point in time.

240
00:16:27,661 --> 00:16:31,801
Especially when customers have more choices and prices come under more pressure.

241
00:16:32,561 --> 00:16:36,321
In conclusion, if you just walked in and want the TLDR, this is it.

242
00:16:36,561 --> 00:16:39,081
A GPU falls to the level of the operator.

243
00:16:41,641 --> 00:16:43,461
Help build a better informed compute market.

244
00:16:43,621 --> 00:16:45,721
Ladies and gentlemen, thank you so much for your time today.

245
00:16:45,721 --> 00:16:50,281
at SemiAnalysis, we know that markets do not become informed and efficient on their own.

246
00:16:50,421 --> 00:16:55,141
We seek to be an active, engaged partner in that work. It takes two to tango, though.

247
00:16:56,041 --> 00:17:02,141
Ideally, more than two, actually. If you'd like to partner with us, contribute, or join the team,

248
00:17:02,321 --> 00:17:06,681
we would love to hear from you. We are always hiring great people. Please reach out.

249
00:17:08,241 --> 00:17:12,901
If this resonated, please go ahead and email sales at semianalysis.com

250
00:17:13,861 --> 00:17:15,561
for partnership opportunities.

251
00:17:15,961 --> 00:17:18,841
And if you're interested in potentially joining the team,

252
00:17:19,121 --> 00:17:22,761
my email is kenin at semianalysis.com, K-E-N-N-O-N.

253
00:17:23,601 --> 00:17:25,021
Thank you all for listening,

254
00:17:25,101 --> 00:17:26,901
and have a fantastic rest of your day.

255
00:17:28,161 --> 00:17:28,981
Thank you.