Same GPU, 50% More Output: Why the Operator Matters — Oliver Kennon, SemiAnalysis

0:01 [music] >> Thank you everyone for taking the time to be here today. My name is Oliver Cannon. I'm a research analyst at SemiAnalysis. Um time and attention are

0:21 the most valuable things any of us have. Uh unless of course you're an early stage investor in OpenAI or Anthropic. Um with that, uh let's get into it. Uh not all GPUs are deployed the same. A

0:32 GPU falls to the level of the operator. Um buying hardware gives you potential. Uh operating capability is what determines how much useful work you actually are

0:42 able to deliver. Uh that leads us to the question I will attempt to answer with all of you today. Who can produce the intelligence you need at the lowest cost? Cost matters. Uh who uh in today's

0:53 presentation I want to show you guys why and how operating capability changes that cost and why it matters in the commodity market that is compute. What matters in a commodity market? Um a

1:07 lot of tech investors or technologists may not be familiar with this framework. Um so a little crash course. Uh in a commodity market producers compete to deliver

1:17 comparable output. Uh a lower production cost gives more room to compete when prices fall. Uh that's what a cost curve shows. Producers lined up by what they spend to

1:27 deliver the same usable work. If two producers earn the same price, the one spending less to deliver the work gets to keep more of it. Pretty simple, right?

1:39 For us, intelligence is the economic output we care about. Eventually, our North Star metric will be price per successfully completed task at the required quality. Uh unfortunately,

1:50 we're not quite there yet. That's a pretty hard thing to standardize. Um today price per token is a useful practical proxy, especially for comparable workloads and quality

2:01 requirements, and gets us closer to delivered output. Uh much closer than the often quoted uh all you finance bros out here will have heard this uh GPU uh dollar per hour

2:12 price. Um or as we view it, a cost. Um We view GPU hours as a measure of input, not output, the cost versus price distinction.

2:23 Customer prices are different from the producer's cost of doing the work. GPU hours are the input, intelligence is the output. Uh on this slide, we can visualize the inputs needed to create

2:38 useful intelligence. Uh we start with hardware and electricity, GPUs uh for the the this presentation. Software and operations can be thought

2:47 of as a multiplier on those first two inputs, with the end product being useful intelligence. The operator is who chooses and implements the software that serves the

2:59 model and keeps requests moving through the system. The machines might be available, but you still need someone to make them deliver responses customers can actually use.

3:10 What matters is how much useful work you get and whether it meets your requirements for the specified task. The conversion can differ with the same kind of chip. Let's look at that in the

3:21 measured example. Here's a concrete example from our InferenceX public InferenceX calculator. You guys can all check it out, semi-analysis.com.

3:31 Uh both estimates use B200 GPUs serving the same model with roughly 8,000 input tokens and 1,000 output tokens, a pretty standard ratio.

3:42 The P50 target is 100 output tokens per second for each user. Uh the serving stack is the software handling the model's responses. In the P90 scenario, one of these stacks gives

3:53 about 200 generated output tokens per second, while the other one gives about 300. That's a big difference and comes down to the software, settings, and number of requests running. All choices

4:06 made by the operator. Same chip, same target interactivity, about 50% more output. That's a lot. So, what does that mean for the buyer's bill?

4:23 What does that intelligence actually cost the customer? For this this example, let's assume the hourly bill is the same for both deployments and both meet your

4:32 requirements. At roughly 1 and 1/2 times the output, your rental spend per acceptable token is proportionally lower. That is to say, the hour buys you more usable and useful

4:43 output. What counts as useful intelligence? Useful intelligence has to do the job you hired it to do. Pretty simple, right?

4:52 The answer needs to be good enough for the task, arrive quickly enough, and be available when you need it. Your information also needs to stay within the access boundaries you you've agreed

5:02 upon. AKA, it needs to be secure. This is an overlooked requirement that is not captured in the chip's performance metrics, but it is essential for a chip to be usable and useful.

5:15 In our published security work, we found monitoring access that exposed another customer's logs and metrics. This is a case, and an unfortunately common one at that, where the GPUs were

5:25 fast, and in the case I'm referencing, really, really fast, which makes the whole thing sadder. Uh and the service still failed horrifically. Um ensuring cluster security takes ongoing work,

5:37 just as delivering the promised performance does. In most cases, it is the operator that owns that responsibility. Someone has to make the system work in

5:50 production. Suppose a faster serving update becomes available. Someone has to test it on the customer's workload, check whether the output still meets the requirements, and

5:59 then decide when, where, and how to deploy it. That means checking the actual responses, testing it under load, and keeping a way to return to previous version if the update

6:10 causes problems once it's deployed. Once it is deployed, someone has to maintain it and respond when issues come up.

6:20 That work may sit with your provider, your own team, or both. Irregardless, you need to know who's responsible and whether they can actually do it.

6:33 So, how do we assess the operator? Uh SemiAnalysis ClusterMax, we're in version 3.0 at this point, uh assesses operating cap- capability on managed GPU clusters, couple couple hundred, uh

6:46 growing. Please reach out if you want to be on ClusterMax 4.0. Um we look at provisioning, reliability, monitoring, and security. Uh hands-on testing, documentation, and customer's feedback

6:57 help us understand how the service runs, not just on a good day, but also when a machine fails. I want to take the chance to reiterate that ClusterMax ratings apply to managed

7:10 clusters, uh different from bare metal, very different. Um with bare metal, more operating work usually sits with the customer. Uh applying a managed service rating there does not automatically

7:20 cover every product a provider sells. The tiers are relative to peers. Little bit of a tongue twister. Uh I nailed it, pretty happy with that. Uh >> [laughter]

7:31 >> and pricing and availability are also important factors in this assessment. Uh these are metrics that do not directly measure uh token output, uh but are incredibly important.

7:42 Obviously, very important. Uh assessment helps us understand who can deliver. Now we can move on to the next question. What does it actually cost to produce useful intelligence?

7:51 Let's start with a compute bill and other costs that producers still has to cover. Rental price and production cost are different. In our example, GPU provider

8:02 the GPU provider rents out hours, the inference a producer buys those hours and sells responses. The rental bill is revenue to the provider and an input cost to the

8:11 inference producer. The GPU owner pays for the equipment, power, space, and support. The inference producer's full cost includes rent plus whatever software,

8:21 staff, and other services it pays for beyond that rental. If the inference producer owns its own it equipment instead, we spread the equipment investment over its working

8:30 life and add running costs. What the producer charges and what it spends are separate numbers. To understand the cost of useful intelligence, we need to know how much useful work the spending

8:40 actually produces. Uh better operations lowers that cost of useful intelligence. The B200 comparison showed why the amount of output matters. More acceptable

8:51 responses from the same paid hours can spread the inference producer's rental expense over more useful work. For a producer that owns its GPUs, the same logic spreads the equipment

9:00 investment and running costs over more output. Count all the spending needed to deliver the service over the same period, then divide by the output customers can actually

9:11 use. That gives us cost for acceptable token for a comparable workload. Better operations also require investment. Testing an update, deploying

9:24 it safely, uh and keeping it working take people and money. The extra output has to justify the extra spending. Uh this advantage can

9:34 persist when a team repeats that work as software and workloads change, but one good benchmark doesn't establish it. You have to earn it every single day. Uh each new workload still needs

9:46 checking, but a repeatable operating process, from what we've observed, can support a lasting cost advantage and a price advantage, too. Uh even if

9:56 you do keep improving, though, your competitors are not sitting still. Uh your position on the cost curve is relative. This is This slide, out of all the slides, should make clear the

10:07 importance of the commodity framework in all of this. Um your position on the cost curve depends on what your competitors can deliver, not just yourself. It's a relative measure.

10:16 Here's a simple hypothetical. We're comparing the same task, the same required quality, using made-up cost units. Uh I like to imagine bars of gold. Uh

10:26 your full cost per successful task falls from 10 to 8. Uh in the same period, your competitor's falls from 6 to 12. You improved. That's awesome.

10:39 But both of you did. And in this example, you started as the cheaper producer and ended as the more expensive one. That gives the competitor more room to lower its selling price, provide

10:49 provided customers find the two services equally acceptable. Uh getting better does not automatically move you ahead. Applying the commodity framework makes you realize that you

10:58 have to compare your your cost per useful work on a relative basis. Comparing your own product with what your competitors can deliver. That comparison can be easy to overlook

11:09 in moments like today, where demand far outstrips supply and will for the foreseeable future. Uh two to three years

11:17 visibility. Good. Scarcity can obscure those differences. When demand is strong and supply is constrained, producers with very different costs can can find customers.

11:30 Strong revenue can make those differences feel less urgent. If you're a CEO, why would you focus on your precious time and energy and money shoring up your spot on the cost curve

11:39 when you can be building a massive new data center? Uh in moments like this, it's important to step back and have a little perspective. Uh take a little historical perspective. Uh every

11:49 commodity in the history of humanity has had swings in supply and demand. Just because we don't see when that will be yet does not mean that this cycle will be any different. Uh

11:59 unless you're a total AI maxi, which some of us are. So, I'm very open to being wrong. I'm wrong every day. Um but I would say if you're a business with leverage, you need to be especially

12:09 conscientious of this. Being proactive with your built with building your cost advantage uh might be not be as sexy as making your top line bigger, but it matters deeply if you want to survive

12:20 through a business cycle. Uh that's why you want to build your operating capability and cost advantage while you have the opportunity uh

12:28 when the sun is shining, when you can raise capital. Uh they matter even more when customers have more choices to make and prices come under pressure. So, how do we make

12:37 clear what uh the buyer actually gets? Comparable compute needs clear grades. Now, let's look at the compute uh that an intelligence producer is buying. We've already seen that the same

12:50 chip can deliver different results depending on how it's operated. Uh in two offers for B200 GPU hours can include different services and leave different operating jobs with the

13:00 customer. A commodity framework needs a way to describe these differences so buyers know which services they can treat as comparable. That's the job of a grade.

13:12 The grade describes a particular compute service and what its buyers can expect. It doesn't make intelligence interchangeable across every task. Once we've defined that service, we can

13:22 ask what its rental price tells us. What does a rental price benchmark measure? Uh a rental price benchmark summarizes the price of GPU hours within its

13:34 defined market. It isn't the producer's production cost or a measure of intelligence delivered. We need to know where those prices come from. An advertised offer and a

13:44 completed trade are not the same thing. Service scope, contract terms, uh whose prices are sampled and how they're combined all shape that reference point. Uh

13:56 operating quality can influence those prices without the index measuring it separately. Running a rental reference doesn't tell you how much useful work your deployment will produce.

14:06 To use it as a hedge, you need to understand how it tracks the rent you actually pay or earn. Uh what can compute futures hedge? There's been a lot of excitement lately

14:17 about compute futures. For all of you that don't know, a futures contract sets a standardized term for future delivery or cash settlement. Cash settlement depends on a price reference.

14:27 Buyers and sellers both face uncertainty about rental prices. Think back to the serving stacks and monitor in the monitoring failure. Assessing who runs the system helps describe the service

14:38 buyers get. That helps us define grade so buyers know which compute services are comparable. The benchmark then needs to track the rent they actually pay or earn.

14:48 That's why I make this argument. Rating the operator is an essential building block for building a credible compute futures market. A rating informs that work. It doesn't by itself define a

15:00 futures grade, establish a sound benchmark, or create enough trading to buy and sell easily. It also doesn't necessarily make a hedge effective. A rental price hedge can help offset

15:11 rental price changes. It does not fix a failed deployment or ensure you that your customer pays. Charging more than the index does not by itself make it a bad hedge.

15:23 What matters is whether changes in the reference track changes in your rent. Unexpected changes in that relationship leave real price risk. Those tool Those tools can help you understand what

15:33 you're buying and manage the rental price risk. For an intelligence producer, rent is one part of the cost. The operator still has to turn that compute into work that the customer can

15:43 actually use. This brings us full circle. So, a GPU falls to the level of the operator. So, who can produce the intelligence you need at the lower cost?

16:00 At the lowest cost? The producer who can repeatedly meet your requirements at the lowest full cost per successfully completed task. Count everything spent to deliver the

16:10 service, then divide by the output that the customer can actually use. Operating capability affects both sides of that equation, what you spend and how much more useful work you can deliver.

16:22 Also, your position depends on that of your competitors. You can improve and still fall behind if your competitors improve faster than you. That's why understanding the operator

16:32 matters through the cycle, not just at one point in time. Uh especially when customers have more choices and prices come under more pressure.

16:41 In conclusion, if you just walked in and want the TLDR, this is it. A GPU falls to the level of the operator. Help build a better informed compute market. Ladies and gentlemen, thank you

16:53 so much for your time today. At SemiAnalysis, we know that markets do not become informed and efficient on their own. We seek to be an active engaged partner in that work. It takes

17:02 two to tango though. Uh ideally more than two actually. Um If you'd like to partner with us, uh contribute, or join the team, we would love to hear from you. We are always

17:13 hiring great people. Please reach out. Um If this resonated, uh please go ahead and email sales@semianalysis.com uh

17:22 for partnership opportunities. Uh and if you're interested in potentially joining the team, my email is kennon@semianalysis.com, k e n n o n. Um thank you all for

17:33 listening and have a fantastic rest of your day. >> [applause]