Same GPU, 50% More Output: Why the Operator Matters — Oliver Kennon, SemiAnalysis

Thank you everyone for taking the time to be here today. My name is Oliver Kennon. I'm a research analyst at SemiAnalysis. Um, time and attention are the most valuable things any of us have, uh, unless of course you're an early-stage investor in OPA-- OpenAI or Anthropic. Um, with that, uh, let's get into it. Uh, not all GPUs are deployed the same. A GPU falls to the level of the operator. Um, buying hardware gives you potential. Uh, operating capability is what determines how much useful work you

actually are able to deliver. Uh, that leads us to the question I will attempt to answer with all of you today. Who can produce the intelligence you need at the lowest cost? Cost matters. Uh, who c-- Uh, in today's presentation, I wanna show you guys why and how operating capability changes that cost and why it matters in the commodity market that is compute.

What matters in a commodity market? Um, a lot of tech investors or technologists may not be familiar with this framework, um, so we'll crash course. Uh, in a commodity market, producers compete to deliver comparable output. Uh, a lower production cost gives more room to compete when prices fall. Uh, that's what a cost curve shows. Producers lined up by what they spend to deliver the same usable work. If two producers earn the same price, the one spending less to deliver the work gets to keep more of

it. Pretty simple, right?

For us, intelligence is the economic output we care about. Eventually, our North Star metric will be price per successfully completed task at the required quality. Uh, unfortunately, we're not quite there yet. That's a pretty hard thing to standardize. Um, today, price per token is a useful practical proxy, especially for comparable workloads and quality requirements, and gets us closer to delivered output, m- uh, much closer than the often quoted, uh, all you finance bros out here will have heard

this, a GPU, uh, dollar per hour price, um, or as we view it, a cost. Um, we view GPU hours as a measure of input, not output, the cost versus price distinction. Customer prices are different from th-the producer's cost of doing the work.

GPU hours are the input. Intelligence is the output. Uh, on this slide, we can visualize the inputs needed to create useful intelligence. Uh, we start with hardware and electricity, GPUs, uh, for th-the-- this presentation. Software and operations can be thought of as a multiplier on those first two inputs, with the end product being useful intelligence.

The operator is who chooses and implements the software that serves the model and keeps requests moving through the system.

The machines might be available, but you still need someone to make them deliver responses customers can actually use. What matters is how much useful work you get and whether it meets your requirements for the specified task. The conversion can differ with the same kind of chip. Let's look at that in the measured example.

Here's a concrete example from our InferenceX-- public InferenceX calculator. You guys can all check it out, semianalysis.com. Uh, both estimates use B200 GPUs serving the same model with roughly eight thousand input tokens and one thousand output tokens, a pretty standard ratio. The P50 target is one hundred output tokens per second for each user. Uh, the serving stack is the software handling the model's responses. In the P90 scenario, one of these stacks gives about

two hundred generated output tokens per second, while the other one gives about three hundred. That's a big difference and comes down to software, settings, and number of requests running, all choices made by the operator.

Same chip, same target interactivity, about fifty percent more output. That's a lot. So what does that mean for the buyer's bill?

What does that intelligence actually cost the customer? For this, this example, let's assume the hourly bill is the same for both deployments and both meet your requirements. At roughly one and a half times the output, your rental spend per acceptable token is proportionally lower. That is to say the hour buys you more usable and useful output. What counts as useful intelligence? Useful intelligence has to do the job you hired it to do. Pretty simple, right? The

answer needs to be good enough for the task, arrive quickly enough, and be available when you need it. Your information also needs to stay within the access boundaries you've, you've agreed upon, AKA, it needs to be secure. This is an overlooked requirement that is not captured in a chip's performance metrics, but it is essential for a chip to be usable and useful. In our published security work, we found monitoring access that exposed another customer's logs and metrics. This is a case, and an

unfortunately common one at that, where the GPUs were fast, in, in the case I'm referencing really, really fast, which makes the whole thing sadder, uh, and the service still failed horrifically. Um, ensuring cluster security takes ongoing work, just as delivering the promised performance does.

In most cases, it is the operator that owns that responsibility.

Someone has to make the system work in production. Suppose a faster serving update becomes available. Someone has to test it on the customer's workload, check whether the output still meets the requirements, and then decide when Where and how to deploy it. That means checking the actual responses, testing it under load, and keeping a way to return to previous version if the update causes problems once it's deployed. Once it is deployed, someone has to maintain it and respond when issues come up.

That work may sit with your provider, your own team, or both. Irregardless, you need to know who's responsible and whether they can actually do it.

So how do we assess the operator? Uh, SemiAnalysis ClusterMAX, we're on version three point oh at this point, uh, assesses operating cap- capability on managed GPU clusters. Couple, couple hundred, uh, growing. Please reach out if you wanna be on ClusterMAX 4.0. Um, we look at provisioning, reliability, monitoring, and security. Uh, hands-on testing, documentation, and customers' feedback help us understand how the service runs, not just on a good day, but also when a machine fails.

I wanna take the chance to reiterate that ClusterMAX ratings apply to managed clusters, uh, different from bare metal. Very different. Um, with bare metal, more operating work usually sits with the customer. Uh, applying a managed service rating there does not automatically cover every product a provider sells. The tiers are relative to peers. Little bit of a tongue twister. Uh, I nailed it. Pretty happy with that. And pricing and availability are also important factors in this assessment. Uh, these are

metrics that do not directly measure, uh, token output, uh, but are incredibly important. Obviously, very important. Uh, assessment helps us understand who can deliver. Now we can move on to the next question. What does it actually cost to produce useful intelligence? Let's start with the compute bill and other costs that producer still has to cover.

Rental price and production cost are different. In our example, GPU provider... The GPU provider rents out hours, the inference, uh, producer buys those hours and sells responses. The rental bill is revenue to the provider and an input cost to the inference producer. The GPU owner pays for the equipment, power, space, and support. The inference producer's full cost includes rent, plus whatever software, staff, and other services it pays for beyond that rental. If the inference producer owns its own i- equipment

instead, we spread the equipment investment over its working life and add running costs. What the producer charges and what it spends are separate numbers. To understand the cost of useful intelligence, we need to know how much useful work the spending actually produces. Uh, better operations lowers that cost of useful intelligence. The B200 comparison showed why the amount of output matters. More acceptable responses from the same paid hours can spread the inference producer's rental expense over more useful work. For a

producer that owns its GPUs, the same logic spreads the equipment investment and running costs over more output. Count all the spending needed to deliver the service over the same period, then divide by the output customers can actually use. That gives us cost per acceptable token for a comparable workload.

Better operations also require investment. Testing an update, deploying it safely, uh, and keeping it working take people and money. The extra output has to justify the extra spending. Uh, this advantage can persist when a team repeats that work as software and workloads change. But one good m- benchmark doesn't establish it. You have to earn it every single day. Uh, each new workload still needs checking, but a repeatable operating process, from

what we've observed, can support a lasting cost advantage and a price advantage too. Uh, even if you do keep improving though, your competitors are not sitting still. Uh, your position on the cost curve is relative. This is... This slide out of all the slides should make clear the importance of the commodity framework in all of this. Um, your position on the cost curve depends on what your competitors can deliver, not just yourself. It's a relative measure. Here's a simple hypothetical. We're comparing the same

task, the same required quality, using made-up cost units. Uh, I like to imagine bars of gold. Uh, your full cost per successful task falls from ten to eight. Uh, in the same period, your competitor's falls from six to twelve.

You improved. That's awesome. But both of you did. And in this example, you started as the cheaper producer and ended as the more expensive one. That gives the competitor more room to lower its selling price, provid- provided customers find the two services equally acceptable. Uh, getting better does not automatically move you ahead. Applying the commodity framework makes you realize that you have to compare your co- your cost per useful work on a relative basis, comparing your own product with what your competitors can deliver.

That comparison can be easy to overlook in moments like today, where demand far outstrips supply, and will for the foreseeable future. Uh, two to three years visibility. Good.

Scarcity can obscure those differences. When demand is strong and supply is constrained, producers with very different costs can, can find customers. Strong revenue can make those differences feel less urgent. If you're a CEO, why would you focus on your precious time and energy and money shoring up your spot on the cost curve when you can be building a massive new data center? Uh, in moments like this, it's important to step back and have a little perspective, uh, take a little historical perspective. Uh, every commodity in the history of humanity has had

swings in supply and demand. Just because we don't see when that will be yet does not mean that this cycle will be any different. Uh, unless you're a total AI maxi, which some of us are, so I'm very open to being wrong. I'm wrong every day. Um, but I would say if you're a business with leverage, you need to be especially conscientious of this. Being proactive with your built-- with building your cost advantage, uh, might be, not be as sexy as making your top line bigger, but it matters deeply if you wanna survive through a business cycle.

Uh, that's why you wanna build your operating capability and cost advantage while you have the opportunity, uh, when the sun is shining, when you can raise capital. Uh, they matter even more when customers have more choices to make and prices come under pressure. So how do we make clear what, uh, the buyer actually gets?

Comparable compute needs clear grades. Now let's look at the compute, uh, that an intelligence producer is buying. We've already seen that the same chip can deliver different results depending on how it's operated. Uh, and two offers for B200 GPU hours can include different services and leave different operating jobs with the customer. A commodity framework needs a way to describe these differences so buyers know which services they can treat as comparable. That's the job of a grade.

The grade describes a particular compute service and what its buyers can expect. It doesn't make intelligence interchangeable across every task. Once we've defined that service, we can ask what its rental price tells us.

What does a rental price benchmark measure? Uh, a rental price benchmark summarizes the price of GPU hours within its defined market. It isn't the producer's production cost or a measure of intelligence delivered. We need to know where those prices come from. An advertised offer and a completed trade are not the same thing. Service scope, contract terms, uh, whose prices are sampled and how they're combined all shape that reference point. Uh, operating

quality can influence those prices without the index measuring it separately. Running a rental reference doesn't tell you how much useful work your deployment will produce. To use it as a hedge, you need to understand how it tracks the rent you actually pay or earn.

Uh, what can compute futures hedge? There's been a lot of excitement lately about compute futures. For all of you that don't know, a futures contract sets a standardized term for future delivery or cash settlement. Cash settlement depends on a price reference. Buyers and sellers both face uncertainty about rental prices. Think back to the serving stacks and monitor-- and the monitoring failure. Assessing who runs the system helps describe the service buyers get. That helps us define grades so buyers know which compute services are

comparable. The benchmark then needs to track the rent they actually pay or earn. That's why I make this argument. Rating the operator is an essential building block for building a credible compute futures market. A rating informs that work. It doesn't by itself define a futures grade, establish a sound benchmark, or create enough trading to buy and sell easily. It also doesn't m- necessarily make a hedge effective. A rental price hedge can help offset rental price changes. It does not

fix a failed deployment or ensure that your customer pays.

Charging more than the index does not by itself make it a bad hedge. What matters is whether changes in the reference track changes in your rent. Unexpected changes in that relationship leave real price risk. Those tool-- uh, those tools can help you understand what you're buying and manage the rental price risk. For an intelligence producer, rent is one part of the cost. The operator still has to turn that compute into work that the customer can actually use. This brings us full circle.

So who-- A GPU falls to the level of the operator. So who can produce the intelligence you need at the lower cost-- at the lowest cost? The producer who can repeatedly m- meet your requirements at the lowest full cost per successfully completed task. Count everything spent to deliver the service, then divide by the output that the customer can actually use. Operating capability affects both sides of that equation, what you spend and how much more useful work you can deliver. Also, your position depends on that of your

competitors. You can improve and still fall behind if your competitors improve faster than you. That's why understanding the operator matters through the cycle, not just at one point in time, uh, especially when customers have more choices and prices come under more pressure. In conclusion, if you just walked in and want the TLDR, this is it. A GPU falls to the level of the operator.

Help build a better-informed compute market. Ladies and gentlemen, thank you so much for your time today. At SemiAnalysis, we know that markets do not become informed and efficient on their own. We seek to be an active, engaged partner in that work. It takes two to tango, though. Uh, ideally more than two, actually. Um, if you'd like to partner with us, con- uh, contribute or join the team, we would love to hear from you. We are always hiring great people. Please reach out. Um, if this resonated, uh, please go ahead and email

sales@semianalysis.com, um, for partnership opportunities. Uh, and if you're interested in potentially joining the team, my email is kennon@semianalysis.com, K-E-N-N-O-N. Um, thank you all for listening, and have a fantastic rest of your day.