Enjoying this issue?
Get tomorrow's AI & engineering digest in your inbox — hand-picked, summarized, and always spam-free.
TLDR
Semi Analysis’s ClusterMAX 3.0 reveals that most NeoClouds have terrible cybersecurity, with issues like outdated software and exposed network configurations that let testers see other customers’ data—including national intelligence. The compute market has become the toughest ever for startups because inference providers are now profitable and compete for the same GPUs. The report is designed for buyers of managed clusters, not stock pickers, and explicitly separates managed cluster quality from a company’s overall business or data center capabilities.
Key points
ClusterMAX 3.0 tests 323 NeoCloud providers on 10 criteria including security, reliability, and performance.
Most NeoClouds fail basic security checks like outdated software and misconfigured InfiniBand keys.
Google has risen from bronze to gold across ClusterMAX generations, but DeepMind now has less R&D compute than OpenAI or Anthropic.
Getting GPU compute is harder than ever because inference providers have become profitable and compete for the same nodes.
Semi Analysis releases a free CLI called CMAX for users to check software versions on their own clusters.
Tools mentioned
Stop scrolling. Start reading smarter.
Receive the day's most important AI & engineering updates in one concise email. No spam.
Transcript (captions)
[music] Uh hi uh hi Jordan um congrats on launching cluster max um what's your brief history uh as a personal introduction and also journey into
building cluster max >> sure yeah I joined semi analysis uh last summer last June and uh been working on the technical staff we've grown a lot um over the past year anyway The uh I guess
personal background is that I spent 10 years working at Hilopackard Enterprise. I was designing a lot of the hardware systems that these Neoclouds were buying back you know before Chief D launched
and then with the big craze in 22 23 when a lot of them came into the market. was a big semi- analysis reader, read every word, knew a bunch of the guys in like group chats on the side, and uh saw
a lot of my opinions represented in Clustermax one when Dan and the team put it out. And uh the best way to get a job at semi analysis is to find something that semi analysis is doing and
criticize it and then Dylan hires you to fix it, which was what happened in my case. >> You got to tell us what were the criticisms of Cluster Max one?
Well, I did two all by myself so I can give you the critiques. >> Oh, no criticisms there. >> Uh, [laughter] >> what changed from one to two? And then,
you know, let's let's click into what's new. >> You can't see it on screen, guys. It's perfect as is. I don't know what what you would criticize. Um, okay. Yeah,
cluster max one. I mean there was just a lot of providers missed like 24 total providers and a lot of them um big ones you know they should have been testing and and weren't small team didn't have
the time and then there's a big focus on slurum to me kubernetes was really important need to you know test that as well um in terms of like criteria and stuff we haven't changed a lot but yeah
all the way back to cluster max one you know the rankings page looks a lot different or there's just a lot less names and um releasing the market view also I believe at the time it was around
120 total providers yeah 111 emerging NeoClouds there in the middle and um we didn't release the full market view this time because we paywalled that behind our neocloud model or we track all of
these providers a lot of them are public a lot of them are going IPO soon so we've got this whole financial model for the industry now but yeah 3.0 I know we're up at 323 total providers that we
track. Um, and I have some people this morning in my inbox reaching out saying that they've got some 3,000 GPUs and they want us to test and we got to, you know, anyway, um, it's been a big
evolution. I think the big thing in terms of testing is that with cluster one, you know, all the testing was hands-on. This is precloud code, pre-codeex
getting serious. And so we did a lot of the testing hands-on by like just running benchmark scripts manually. And now we've been able to develop like this whole repo over the past year and a half
of industry standard benchmarks, custom benchmarks, stuff for reliability and burn-in. Um, you know, it's it's like very automated now where we can launch uh this stuff with the coding agents and
have them run and fix a bunch of issues autonomously without us involved. And um yeah, I mean just the the breadth of coverage as well as the depth of coverage I would say has improved over
time. >> I mean I guess introduce it. What is cluster max? What is NeoCloud ratings and rankings? Um what are like the main things you guys are actually testing for
here? >> Yeah. So there's 10 criteria described on the website. Basically a neocloud we define as anybody who rents access to compute um sells access to compute. So
Neo cloud implying that there was hyperscalers and legacy clouds that existed before chatbt. Um this is AWS, Google, Oracle, Azure also some of the old school clouds like a Rackspace or
Digital Ocean or even like a Cloudflare AI. Anybody who's like running a big data center footprint and renting access out um there would be a cloud and then the Neo clouds are the new ones that
have come along. We say everybody's becoming a NeoCloud because what makes NeoClouds different is that they're specifically focused on AI. So they deploy GPUs and other accelerators for
AI workloads. This is training inference increasingly RL and um that's what we try and test on is who's going to give you the best performance, security, reliability, and uh just like ease of
use, pricing, availability. We try to go through comprehensive assessments instead of individual benchmarks. Back when JGBT came out, a bunch of people were reaching out to semi analysis
saying like, "Hey, I'm trying to decide between five providers like how should I think about this? How should we go about it?" They were, you know, bothering Dylan with the same questions over and
over because they're like, "Hey, I just raised $20 million. I'm about to go give 105 million of it to a provider. You know, it's my whole seed round. Like, this is a really critical decision for
me. I need to get everybody's feedback and so instead of Dylan and team like answering that question individually custom every single time I was like why don't we write an article to give our
like course grain first thoughts here's our like quick tier list rankings um podium and then you can engage us for more detailed stuff you know consulting DD whatever it is for your specific you
know requirements and we can dig in further from there >> there's a lot there Uh uh the the one that comes to mind is the is the criteria of of the 10 criteria. Uh the
one that comes to mind is the one that you guys quoted which is Ilia Sgiver saying uh every all of them have terrible cyber security. What's what is the state of cyber
security and neoclouds? >> I mean you said it terrible. Yeah. Um it depends who we're talking about. Look uh we Yeah. >> And also I I'll just say hi to Dylan who
is who's just joined us. [laughter] Dylan, >> so sorry for being late. >> Dylan, what's the state of NeoCloud cyber security right now?
>> So, so I mean we, you know, I think um security is really bad. Even back from the first cluster max, um there was uh pretty significant security issues. Um people not properly implementing network
security controls. And so we were able to see other people's uh jobs, see other people's data, see other people's storage, etc. in the first generation of cluster max um we escalated to the firm
as well as Nvidia uh you know and and rank that company underperform of course and strongly recommended people don't use them um and in the second generation of cluster max we saw similar stuff I
think a lot of people got better because we highlighted stuff um but in the third generation of cluster max we also tested I think you know Jordan and his team tested this and and saw pretty major
security issues so we posted a a newsletter titled told uh most Neoclouds uh suck at security. Um and then uh you know a handful of hours later, Ilia tweeted about most Neocloud security
sucking. Um they're obviously a very security conscious firm. Um you know they they have a lot of folks who are from the former cyber security world. So I I imagine they've done their own work
on a number of clouds there uh to see um and test and it's it's pretty clear, right? What? It's so bad. In one case, there was liter we were literally able to see national intelligence of a
certain country's like stuff, right? Like they had multiple national security agencies and military style uh intelligence stuff running in that Neocloud and it's like good god, we're
just like here to test the cluster and we can like see this stuff. Imagine if we were malicious. I'm sure there were people who were malicious or could have been malicious um on that cluster as
well. So thankfully they fixed it I think. Um but like you know we shouldn't be the one that has to find whether or not someone is uh you know good or bad at security but you
know I think the the industry generally needs that service. Um and so that's that's a big area of our criteria. >> Yeah and I think on the technical side like this is our checks are very simple.
Is your software version up to date? Are your drivers up to date? like is it configured correctly? I mean um these are relatively simple checks but when people are running versions of
software that are 2 3 years old and they have well doumented CVEes online it's like you don't need a Frontier model to create a PC exploit of this. Um you just need to know the right
place to look. And so as part of the article, we also put out a free CLI. People can install cluster max CMAX and then use it to check the software versions on their clusters and identify,
you know, a quick like you get a link in the terminal output to show you where to upgrade stuff. Trying to help people keep the software up to date. This is the thing is NeoCloud security.
>> The other thing is the network keys, right? There's um Infiniban and Ethernet have various like at least on Infiniban side it's like M keys, P keys, all these things. Are those properly configured
and deployed? Um, it's it's it's shocking how bad most NeoClouds are though. Like it is really bad. >> Well, you guys >> I guess any anything interesting, any
major jumps, any improvements that they've made from one, two, three? Anything you want to highlight? Our >> our our bar on cluster max is, you know,
first generation was like super low. Um, and each generation we've made it harder and harder. Hence, the trend has been for people to get downgraded. There have been a few folks who have been upgraded,
of course. Um, Google's done a really good job at getting upgraded across the cluster max generations. Um, I think they started out as bronze and moved to silver and now gold. Um, or just silver,
sorry. Oh, no, gold. Yeah. Yeah. So, Google's Google's consistently upgraded while there have been a number of clouds who have been downgraded across the generations because they sort of
stalled. They improved a little bit, but they didn't improve a lot. and and we keep trying to raise the bar um and and teach everyone, hey, these are all the things you should do to be better. Um
>> yeah, and the the bar for this time was like latest and greatest. So, are you deploying GB300 NVL72 in 2026? You know, this is incredibly popular chip. It's what all the Frontier Labs
want to buy. It's what all of the Neols want to buy. And the typical customer that comes to us and says, "I've raised $100 million. I want to spend it on compute." They want a GB300 NBL72
cluster. And so some of the providers that were in gold originally do not offer GB300 NVL72. Why not? Well, you got to contend with a whole bunch of different things. There's
a new network. NVL72 implies it's a scaleup domain in the rack. You got to go to ARM CPUs. It's a Blackwell GPU. You have an 800 gig network. It's mandatory direct liquid cooling. So from
the physical data center to how you deploy the operating system to how you like manage the drivers in the network just about everything's different and when people say oh we just like made a
strategic business decision choice not to do GBs well what what happens when Vera Rubin comes next year who are you going to trust and so the trend is like if people are going out and they're
deploying 10,000 GB300s and we test the cluster and it works and we engage with the support team and we we give them feedback on some things we we have issues with and they fix it in
an hour, not two weeks. They're moving up the rankings. Other people that are not deploying these latest and greatest GPUs or they're doing it and it's completely broken, it's taking them a
long time to fix it or it's we find security issues or we hear from customers huge issues with reliability, which is probably the number one criteria that like these top customers
care about. Um, they're moving down the list. So, that's that's a bit of the dynamic there. So assuming most of these you know bronze silver honorable mentions are good to rent from. What's
the state of availability cost compute? So say you are a neolab you just raised tens of millions of dollars. You want to start to rent. Um how does that market look now compared to one two three years
ago? Things getting better worse. >> Good luck. [laughter] >> Yeah way worse man. >> It's it's really tough to it's it's the toughest it's ever been to compute. Um
obviously Anthropic and OpenAI have been buying smaller and smaller cluster sizes. Um but also all of the inference providers are more and more profitable. It was one thing when say base 10 was
running at 10% gross margins which I think might have been like six months ago right um on inference or nine months ago but it's another thing when you know base 10 fireworks together and many
other companies even small companies like morph are running at 60% gross margins on open models using VLM or slaying using just a little bit of optimization beyond that um and able to
do an awesome job and then you know companies like inact and radics arc as well so so anyone who can like get access to a GPU can immediately make money. Um, and so ultimately it's like,
well great, I'm a Neil lab who wants to train an interesting cool new model. Well, I'm also competing against people who make money now. Whereas before I was mostly competing against money losing,
you know, open, money losing entropic, money losing labs. Very few people were making money off of GPUs. Now everyone can make money off of GPUs. And so it's just very, very difficult to get any
sort of compute. >> What level are the labs Yeah. What level are the labs going down to compared to like major? >> They they used to only go like 8k, you
know, uh 8k GPUs from a rider. Um, you know, and and it's it's steadily gone down. Um, we've even seen as small as 1,000 GPUs get rented by the labs now. Um, and so 1,000 GPUs is closer to what
a neolab wants. Um, and so you've squeezed out the market a lot. Um and then when you look at like the vase 10s, the fireworks, the togethers and all these other inference providers, they're
cool with like they're cool like four nodes like they're modal, right? These guys are cool like even four nodes at a cloud. Um so ultimately it's like you know obviously their orchestration is
tough. Um modal's talked a lot about that actually, right? Their orchestration across like 20 different clouds. Well, it's Yeah. So, so ultimately it's like a very um it's
challenging for anyone to get any sort of compute if you know modal can make money hand over fist off of even four nodes from a company. >> Yeah, it's bad. I mean like even
individual you can't spot rent single nodes anymore like down the whole stack is cooked. Um what about the state of the labs? How do you segment the large players OpenAI, Anthropic, Google? Where
where's the split of who has the most compute? uh anything interesting on the road maps of where you think who's investing the most? >> Uh I mean it's clearly OpenAI and
Enthropic that are leading the way in terms of both renting from like large scale bare metal hyperscalers. In other words, you know, OpenAI is a driver of Core's business also Microsoft. They're
also exploring with other hyperscalers. They're doing self-filled as well like massive stuff with Oracle. And then Enthropic obviously they got their big AWS partnership with Rainineer. They've
also got a lot of TPUs with Google Cloud. They signed stuff with Corewave. They've signed stuff with, you know, others. And um so they're they're the big driver. I mean, you just look at ARR
and something has to drive that. And so it goes to the compute side. Um with that, >> another aspect of this is um you know, a lot of these providers like like Crusoe
have fantastic bare metal, right? Um but that's not what we're testing in cluster max. We're not testing um bare metal. I mean, it is part of the criteria. Um, but a lot of the criteria is a managed
cluster, right? Managed slur, managed Kubernetes, all of that sort of stuff. So, it's it's a different level of um service that people want on top of the GPU. Um, the original cluster max was
titled also like a big chunk of it was called how to rent a GPU. We're also just teaching people what they should do to rent a GPU. Um, and in our view, most of these smaller folks, you know, if
you're super small, you don't care about, you know, your node being bare metal. Um, if you're super super big, you want bare metal because you're going to put your own, you know, uh, fork of
of slurm/kubernetes, um, and do your own stuff on managed services and such like that. Um, but most customers want a managed slurm, managed Kubernetes, um, and for the
cloud to like hot swap nodes for them when one fails. Uh, you know, the biggest customers will manage stuff like this, but, you know, the very the the midsize and smaller customers want this
sort of stuff managed for them. And so ClusterMax is focused around that. Um so companies like Crusoe um will rent bare metal, right? They have massive contracts with companies. Um or they'll
do great in data centers or they'll they're starting to do well in, you know, inference API, but on managed services, they're not doing well. Um and and that specifically is what cluster
max is targeting is the managed service layer, right? Because it's otherwise it's like too complicated to uh you know, it's it's it's many different areas people want to rent from. Some
people want bare metal, some people want mayor services, some people want tokens out. Um, and so ultimately it's a range of options. Um, as far as you know, even Google is renting some compute from
folks, right? They signed a deal with C cororeavee. They signed a deal with SpaceX. Um, so so pretty large amounts of compute that they've signed over, I think over hundred billion dollars worth
of compute that they've signed from various uh customers. Um, Nebius also has deals with the labs and such. And so um but these guys are only going for bare metal. Um open and thropic are
definitely lion share buying the most. Um Google Google is definitely falling behind at this point. Open andropic have as much compute if not a little bit more. By the end of the year they'll
both have more compute dedicated to R&D than deep mind. So yes, Google still ultimately has more compute than open eye for now. Um, but a lot of that is sold out to other places and dedicated
to other offerings. Um, but DeepMind's R&D compute versus openthropics R&D compute. The two labs have more compute now because they've reached that scale. Um, so this whole like, oh, Google has
the most compute, they'll win is is actually not really the case anymore, right? they still have a cost advantage because they build the TPU and um it's much lower cost but ultimately it's it's
it's less deep mind has less compute than open anotropic. >> Yeah, there's a big big shift like one of the uh uh the biggest insights for me at AIE was running into some deep mind
researchers and they were very depressed because they were like look our leadership keeps selling the compute that we want to other people. Yep. [laughter]
>> I mean, imagine being XAI, right? Like, >> well, yeah. You know, I thought SpaceX was a Neoclad. Like, what the hell? Why are we Why are we unavailable? >> They're on here.
>> Yeah. Well, as soon as we can test, they'll move up the rankings, you know. Um, they'll be a competitor. I think uh SpaceX's and the Google deal for example is super interesting as well as the
anthropic one because it goes to what Dylan was saying about the fact that the labs are considering becoming neoclouds on this opportunistic basis. Um Meta's considering the same thing. You see
Mistrol and Poolside in there as well. These companies have this skill set on building data centers and manage managing them and then when there's an opportunity to monetize the compute
either through renting it to other people or through serving inference they want to pursue that but only to a certain level and they always want the optionality bring the compute back and
use it for those research or hero runs as soon as they can afford to. They want to use the maximum amount of compute they can afford to research right now. And all the extras are, you know,
opportunistically making the money to fund all that research. And so SpaceX's contract with Google, for example, has mutual 90-day cancellation rights. So SpaceX's research starts to work, they
can pull the compute back. Google stops making money off of that compute on some inference endpoint, they pull the compute back. It's a really interesting structure where both are kind of, you
know, in this little standoff thing who's who's going to who's going to blink first who who believes in AGI more um as going to the other aspect of of uh at least for SpaceX, right? If you're a
XAI employee and you got bought by SpaceX and then SpaceX sold all the compute, at least you're not depressed because like you know, you're like, "Okay, look, Elon's building a shitload
more. I'm gonna have access to this and he's doing this because it helps us be more financially solve it. So for the IPO um with Google it's like why the hell am I selling the compute? You know
I have I'm already a very profitable company. Why why can't we just use the compute to build models and sell those models at way higher margin instead of giving to anthropic or openai or other
firms? Um you know SSI is also a customer of Google. What do you think about the um the dynamic of like Google funding the people that leave to go do a Neolab
to buy GCP? In other words, like Dave Silver, Jeff Dean, they spin out, they get funding from Google Ventures to then go spend on GCP. >> I mean, I think it's it's it's a
fantastic move if if if you look at just GCP as its own company, right? Thomas Curran continues to pump his um you know his revenue, his operating margin, his operating income, but for Google as a
whole, it's really dumb, right? Like you should not incentivize your best researchers leaving. You should not incentivize u sold at 20 $30 million a megawatt when
you could run models on it for $50 million a megawatt. So if Discovery Loop has a big breakthrough, do you think Google is able to keep them in their orbit or do you think Jeff Dean goes and
takes that breakthrough and does something without Google being exposed to any of the upside? Like they would be an investor. They're obviously the cloud service provider.
>> I mean I mean go look at uh Anthropic, right? Like yeah, true. Google did invest in Enthropic. Um but if Google didn't invest in Enthropic, would Enthropic be here today? Because Google
was a really early believer. um like Opus 3 and and Sonnet 3 and 35 like those I think were just trained on Google's infra for sure um you know whereas and then they later brought on
Amazon which was uh you know Amazon's also great but at least Amazon you know is just like they don't have a business of training models and selling them right in the same way they have some
folks but like nothing nothing crazy but Google does and so it's like did Google like stand up a ultimately their biggest competitor uh and fighter now Google now anthrop ropic is not only renting from
Amazon and Google but also from numerous other clouds. We have deals with like 10 plus clouds now. Um and then Enthropic is also building their own data centers, leasing their own data centers, uh
starting to fill them up, um working with Fluid Stack to deploy TPUs in their own data centers. So sure, Google's still getting revenue, but it's like, well, I used to rent from you. Now I'm
buying TPUs from you, which is less margin for you. And now in the future, I'm going to I'm now building my own chip. I'm gonna do everything without you. So, so like it's it's pretty clear
what happened with Entropic. Um Jeff Dean, if he does something amazing um in Discovery Loop, then why why would they continue? Why would they be stuck to Google, right? They can use any cloud.
Um switching clouds is ultimately not impossible. It's not a it's it's not, you know, data gravity and all this stuff that people used to talk about with clouds in the pre-AI era is a lot
less relevant in the post in in the AI era because AI can just help you migrate everything. and a GPU. A GPU is not just a GPU, but like you know a GPU is largely um workable as as a fungeible
asset across clouds if you're good. >> So, what do you think? You think Google's cooked, man? >> I mean, I think you guys are much more of an expert than I am. Uh and there's
like Yeah, there's Google's just like a conglomerate of a bunch of different things, right? Uh so I I I do think like the the data advantage I mean there's data gravity as as NeoCloud but uh the
data advantage of Google is the the thing that most people could keep coming back to right um and uh there's just organizational challenges and like you know now there's new new leadership um
you know God bless to Cororey who has the hardest job in the world to uh to fix this but like I mean he's a very very capable guy um and they like I think I think they they know like like
you know it's it's like not a secret that like uh they do need a reset and um to me Demis like basically stepping out of the leadership seat was just like a complete black swan like I just like I
never thought it was possible like maybe I just don't know him personally where like he's like yeah actually like I just care a lot about isomorphic but like um I mean you know he he he founded it like
he's he's um you know he's he's sort of like the spiritual leader of deep mind And and so like this this like this is probably the first major uh Frontier Lab transition, leadership
transition fully. >> Um can we say that there were >> like two days where Sam left, right? But uh >> I mean he never left in in anyone's like
hearts. Uh but yeah, this is the Tim Cook of like >> Yeah. >> Ilia leaving OpenAI. >> No, not the leader.
>> Yeah. Yeah. It was like kind of like the Yeah. Yeah. I mean, yes. Okay. So, like obviously people split, right? Like Anyway, um so uh that's my rough take. I don't think I'm going to have like uh
spicier takes than you guys. I did want to pick up on one thing. This is like just a chance to like catch up on like the last six months of like crazy news. >> You guys mentioned Poolside. You've
never never actually written about Poolside because they're not that big. Um but uh there's they do have >> uh they got bought by uh their their employees got bought by Nvidia for 12
billion. What's going on? >> Okay. Yeah. Um >> it's really cool, right? Cuz Nvidia is paying for all these researchers so they can make Neotron better. Um,
>> no, this is defense. Poolside is Poolside Poolside like you know from the beginning like they they just like all the cloud NeoClouds kind of sucked and so they started building a lot of their
own infra and they got better and better at building their own infra and then they were trying to deploy their own clusters and had a partnership with coreweave on this and stuff like that
but they've just been slowly moving down the stack maybe the first AI lab to uh become you know running their own infra even before open and enthropic were trying to build their own infra was
trying to >> which is wild as a side project it's like people have to do this. Is it a full-time job? This is a side project for them [laughter] almost.
>> Well, now it's their full-time job, right? Because they sold all their researchers to uh Nvidia for Neotron. Um because Nvidia clearly wants Neotron to be a lot better. I think they've sort of
lost hope in a lot of the uh um open source folks. I mean, sure, Reflection is still trying um thinking machines open source some stuff, but ultimately I think Nvidia is losing sort of hope in
American open source. So, they think they just have to do it themselves. Um, so they've bought a bunch of employees. Um, and yes, I thought that was $7 billion, not 12. But anyways,
regardless, um, I thought it was like a $7 billion and then a $1 billion investment out of 12. >> We have the we have the details in 12 billion.
>> I take the bigger number. >> License and employees are sick. The number is six. >> Always always take the bigger number. But, you know, I think at the end of the
day, it's like Nvidia can say, well, actually, we only paid 1/4 the price because all of this money is just going back to buying GPUs. So, did they pay, you know, $7 billion,
six for the employees and one in an investment, or did they actually pay, you know, one and a half because all seven of that is going into buying GPUs? Um, and in fact, them doing this is
going to incentivize other people to invest a bunch of money into them. And so, actually, it's a it's probably a positive thing to hire all these people. Like, they they actually didn't spend
any money because now they've created a new customer, a new cloud that's going to build a massive cluster, right? Like imagine, you know, they've got $7 billion now of cash. Um they raise
another few billion dollars. Let's just say they have $10 billion of cash on the balance sheet. Um and they're able to get a loan to value of like 75% um you know, 25% uh down and 75% loan.
So now they have $40 billion. They can build a gigawatt. Awesome. This is another gigawatt scale Neocloud now. Um so they've just made a real new competitor.
>> We have that lower down in the in the article. Um it's uh yeah I mean I I don't know I don't know uh if that I mean this is the playbook if this is the playbook honestly um uh lower down lower
down um uh with the the guy who's leading it Robert Bonar um and like if this is the playbook a lot more people should be doing this >> should be doing what should should be
building should be doing deals with Nvidia like this because Nvidia will kingmake you just cuz they they want more clouds. Yeah, I mean not everybody releases the second best open American
model. >> That's true. >> And spends years building it. I mean, we're kind of brushing over the fact that Poolside did build something
interesting. And to me, both this and hugging face are defensive acquisitions by Nvidia who does not want their competition snapping up a lab that actually has something going for them
and having some momentum where they can build the real experience that's required to build real training and inference clusters. In other words, it's so important for building an
infrastructure company that you actually understand how the workload works and you have opinions that are informed by training models and running to get represented in the not crypto miners who
decide hey I found some site I got some power I have some money they don't have the ability to execute like somebody who has proven they can train the second best American open model in the world
which poolside did Right. >> Yeah. >> Say more on the >> I was going to say you mentioned Hugging Face in there. How do you guys see the
Hugging Face acquisition? Anything you want to comment on there? >> I think it's quite quite similar. I think it's defensive. I don't think that Nvidia wants the influence in the
ecosystem of other companies coming in and owning somebody that's like critical in >> there's a Galaxy brain take here, right, which is uh now now Hugging Face, you
know, can can sue the out of OpenAI for hacking them. Um, and now Nvidia can can has has has that has that hammered? No, I don't think so. I don't think so. Um, but I think I think I think Hugging
Face is obviously like uh you know leagues ahead of everything else. Um, some some folks on my team have started submitting a bunch of PRs for model scope uh to work across the ecosystem.
Um, >> by leagues ahead in what sense? >> Just like for for being the open source repository for >> I see.
>> Yeah. uh just like what hugging face does. Um but then ultimately a lot of them are uh you know like it's it's it's you know the question is like does hugging face you know there's a way you
can just like deploy hugging face models um on Amazon tranium if you want right you can just click a button and it'll deploy them um does that go away and is it only a GPUs from here on out um or is
there any closing up that happens um we'll see but ultimately it's it's it's uh I don't know I I also love that the price was uh the emoji code. >> So sick.
>> Well, and the team's incredible. Look, I I think we're we're kind of brushing over the fact that HuggingFace and Poolside have incredible talent in both these organizations. And like a lot of
people would see that's the motivation for Nvidia to acquire them. I say it's defensive because if you look at Nvidia's business, they're generating almost $50 billion of free cash flow a
quarter. So 200 billion a year. And there is a corporate strategy team at Nvidia that gets some amount of that money and says what what what we need to deploy it right they can make
investments they can make acquisitions and 10% of free cash flow to acquire hugging face and poolside. I mean it's a no-brainer to me that these would be good assets on your team and working at
least not for your competition. Let's say Hugging Face stays independent and they follow the letter of the law and the acquisition and they keep supporting all these other chips and stuff. If they
were acquired by somebody else, they would be actively working against Nvidia's interests. And I think that's not what they want. So, I think you're going to see more of this. If companies
grow up a little bit, they build this team, 100 or 200 incredible people, um they want to bring them into make sure that they stay in the Nvidia ecosystem as opposed to going and competing
against them with somebody else. Yeah, makes sense. Not to discredit anything Pullside's done. We're uh pretty big fans. We had them on the pod a month before they got acquired. For
people interested, the model factory work, the approach to training foundation models, uh very very good way that they built all this out. >> Yeah. To to me, you know, coming, you
know, maybe maybe if this brings us back to Clustermax, uh the way that you that all NeoClouds should be proving like custom 4 is like just train a frontier model, please. uh your yourselves,
right? Uh and then like show that everything works. >> I don't know if everyone should everyone should have the skill set to train a front. That's just a kind of
>> No, but there's a lot of open recipes and stuff, right? >> We need mosaic. >> Let's let's talk about what it takes to get there, right? If you're going to
train I think also I think poolside and and probably hugging face probably both entertained uh acquisition options in the past to the tune of similar money and it's
>> Yeah. Specifically, Nvidia tried to buy Pullside uh sorry uh Nvidia tried to buy Hugging Face uh at the start of the year, right? In case people didn't know. >> Well, I mean, I think also Nvidia is on
a bit of a acquisition craze. There were rumors of thinking machines and a whole bunch of Nvidia acquisition. >> Yeah. Founders kind of have the they have the option to like cash in when
they want for a lot of these people that have built these things, right? So, when do you decide to cash in? I think Pulse is probably looking out there and going, "Okay, I've built the 10,000 GPU cluster
I needed to get Lagona out." But if I want to compete, I need a research cluster that's 100,000 GPUs, 400,000 GPUs. Like, how do I keep up with the open frontier into next year? I think
that's pretty hard to keep going. And the money comes and they're like, "Okay, sure. Let's just cash in and build it with the other people. That that seems fine." Um, but the the other thing worth
pointing out is like on the technical side, pool size papers are really good on the infrastructure stuff. I've talked to the team if you're going to train a frontier
model, you need to have health checks on your cluster. You need to have high reliability. You can't just have this model like not make progress as you're training and constantly be failing. Um,
has that. They have like an incredible suite of um they they just have incredible infrastructure. So the uh the idea that people with that talent would be able to provide that as a service to
other people I think is like just obviously true. Uh and there's only limited people that have that experience as we're saying like even when we get into the silver and bronze tier here um
people are not training frontier models on most of the silver tier and all the bronze tier let's say. uh a lot of inference, bit of RL, a lot of research, but I wouldn't trust any of those guys
frankly to build except Azure, right? But to build a 100,000 GPU cluster like who has proven they can do that already? >> Yeah. >> Five companies.
>> Yeah. >> Yeah. Um Okay. Um I do want to maybe uh dive in on one or two things that maybe you don't uh don't actually cover. Maybe the best way uh given the time that we
have left is to just address the criticism. Um uh you know you've re uh obviously this is a very valuable list like people do make purchasing the students offer bits uh periodic lads uh
did come out and say like yeah actually like uh you know this this helps us decide what what what we spend on um uh yeah I mean you know what is the criticism that most people have to
>> I think I think there's um a lot of criticism for a lot of things like first and foremost uh this is testing managed clusters um So that does not mean that a company is not good at other things,
right? Crusoe it was downgraded for managed clusters. Uh but they're one of the best data center builders in the world. Um if not like the best. They have they have they have nearly the most
contracted or or the most contracted gigawatts in their pipeline and a bunch of sites that they're building on really rapidly. But this is not a criticism against their you know data center
building capabilities. It's only a criticism against their managed cluster offerings that they were they're still one of the better ones, right? but they were downgraded, right? Um, you know,
likewise, you know, other companies together has great inference endpoints that has nothing to do with managed services. Um, you know, and and so I think there's a lot of uh confusion
around what this is testing, right? This is for managed clusters, not for data center construction or bare metal clusters or for inference endpoints, which become a blurriier line because
everyone is doing these things, right? Um the other question, the other big major criticism we have is like oh that you know people are like oh are they taking money for this? It's like no we
do not take money for any ranking uh that would be illegal. Um I myself have personal investments in Crusoe. I have personal investments in Fluid Stack and neither of those companies um you know I
did SPVS in both of them. Neither of those companies are in the top, you know, five clouds. Um because because that's not what, you know, I'm not here to be biased, right? Jordan will say
he's like he he's like he has free reign to like do whatever he wants to do. Technically, he doesn't even know all the companies we do contracts with. Um if I look at the top 20 companies in the
world, there are five companies that are not our customers. Um you know, like Eli Liy and Saudia Ramco and stuff like that, but ultimately it's like those are our much bigger customers than Core
Weaver Nebius could ever be, right? because you know they they just have way they're way bigger companies with way bigger budgets um you know and so like ad AWS is a bigger customer than core we
have Nebius for example right um and yet that has nothing to do with the rankings right everyone in the infra supply chain buys our stuff um it just is what it is right if you're tracking data centers
you want to know other people are doing in their data centers if you're tracking accelerator supply chains if you're trying to understand the cost of various kinds of chillers if you're trying to uh
understand the supply chain for networking and optics you're going to buy our research in our data services that does not mean um and you'll often commission us to do uh work beyond that,
right? Um you know, but that doesn't mean that we're going to ever bias our rankings. Um you know, so so sort of I think you know there's a lot of criticisms around that, but ultimately
like first of all, if we were taking money and ranking people and not disclosing it, that'd be illegal. Um but second of all, right, like you know, it makes no sense, right? The the value we
have to the industry is that we tell the truth and what we believe. Um if if we didn't do that then no one would care what we would say and we'd be screaming into the void. Um so our reputation is
like what matters the most right over a quick buck. >> Yeah. Maybe one way to put it is like uh you're you're you are a buy side service and people and people are accusing you
of taking money for the sell side and they don't understand the buy side is like probably 1,000 times larger uh in terms of revenue for you than than the sell side.
I mean, I think people trying to poke at it's a paid list, you know, and that's just one IQ. The the broader point is like, okay, people are misunderstanding what you're trying to rank here. And a
lot of the claims are just, well, it's crazy that this is a better cl x is a better cloud than y, right? But that's that's not what you're actually testing. I hear two uh clear criticisms about the
writing from people where I listen to the criticism because I you know respect their work and I uh you know they're not like anonymous just shouting into the void and those criticisms are uh you
guys didn't write enough you should have released more data you should have written more or this is 30,000 words I can't read this all. So it's really hard for us to balance
between those two. >> But it's it's a new >> Yeah. You're evaluating neo clouds of course like >> um so for for the for the former I'm I'm
you know interested in that criticism and and we we want to do better. And so we have this research product called the cloud TCO model that anybody can come and license. It's a yearly fee. We write
notes every week about um the trends in the industry. We have a dashboard which we took a redacted screenshot of below where you can get all of our data. All of our performance data is on there. Um
yeah, it's it's like you know you you get us working on the research for you year round weekly rather than just uh for this individual you know kind of once a year release or every six months
or something is what we do. >> So that's for the people who want more detail. Um >> there's another crowd of criticisms which is like um you know it's
oftentimes someone who like owns one of the stocks and we rank them poorly or positively right like I remember last time people were like like by the way like our view on a company doing better
or worse in the stock market has nothing to do with uh Jordan and his team right there's uh you know we we do we do estimate revenue for all these companies going on a forward basis um and then you
can like license our data and see what you think that should mean you should trade. We never recommend us to trade. Um but there have been times where like for example um we thought Nebius was
better than Corewave. And the reason we thought Nebius was better than Core even though we think Coree is still a better cloud than Nebus is because Nebius's contracts are shorter term and so with
prices of GPU spiking, Nebius gets to take the upside better, right? Like that's something that we've released in our research in a separate separate group of people, not Jordan and the
technical staff that are working on cluster max or or hey like Iris energy has this massive data center that's going to come online. and we think it's going to get leased and we would put a
report out about that while at the same time saying you should never use Iris Energy's um Iron's uh managed services. Um and so like it's like it's it's it's it's completely different things, right?
You know, they sold that data center as a bare metal to Microsoft. And so we were, you know, our research on the institutional research side for someone who might be buying stocks is completely
different from like, hey, cluster max is really made for people like periodic labs or applied compute or companies like that who have publicly came out and said they like like cluster max and what
we do there. Um you know, even OpenAI said they like cluster max and we have like a quote from them on on that. Um, we have a variety of different buyers who said they really appreciate the
methodology and that we are pushing the bar up for all the clouds. And so I think it's it's it's you know, if you're trying to buy stocks based off cluster max, that's like completely wrong. It
has nothing to do with stock performance. >> Yeah, it's 100% man. This is strictly for buyers. This is for helping people buy compute. It is not for people to
pick what stock to yolo all their savings into. I mean it it will correlate uh at at some kind of limit. Um I I think like I think isn't a tendency to literally
literally from you guys are so okay. All right. All right. All right. Gez. Oh my god. >> No. Literally, right? Like someone someone who's terrible at running a
cloud but has a gigawatt data center that they're building and and it's like actually credible like a gigawatt data center. Well, they could be the worst ranked cluster max and they could still
make shitloads of money. Um it like like does it matter >> at the current price at the current price of the stock? Many names are cheap, many names are expensive like
trust. >> Yeah, I I do I do I do get it. Fundamentals versus uh you know, one offering uh that contributes to the overall fundamentals. It contributes to
the overall uh valuation and direction. I used to be a hedge fun guy. I get it. You know, I'm just saying like this is like you guys are the domain experts and people do value the [clears throat]
rating on that element. Uh but obviously this is way more buy side uh and buy as in uh you know the the the neolab to just keep stressing this but the we genuinely have a core research project
where people report on stocks and they give opinions about the names and many of the guys that work on that research look they ask us questions but then they come out and they say people in the
underperforming or unavailable tier were positive on the name this is different than what we put out publicly and say you should rent clusters from and maybe I need another disclaimer on the future
one that's like this is not a cosign of the current valuation of their stock. [laughter] >> It could be, you know, it could be really interesting for like a 2 by two
or like a two-dimensional chart where like here's the cluster max ranking and then here's the uh same analysis like stock ranking and like it it it the differences are going to be interesting
research. Maybe worth saying though there is uh a lot more coming which is that uh Dylan mentioned it individual names and logos on here have businesses that are much larger than just managed
clusters which is what we're raking here and so we are going to be addressing that on the newsletter for inference endpoints for bare metal for R infrastructure for sandboxes for you
know part of RL infrastructure but kind of different uh we're going to be writing about all of that and stay tuned for that because I think people will start to have a more holistic view of
these businesses that compete in different market segments across inference, managed clusters, and bare metal. And we've got a lot of focus on managed clusters here, but frankly, so
many people are reaching out to us for help deciding on which endpoint provider to go with or which hosted RL training provider to go with or which sandbox provider to go with. And that's a
different question than who to go with for your managed cluster. >> Yeah. So, so is the solution more maxis, more benchmarks? Yeah, that's Yeah. Okay.
>> How many do you have? [laughter] You have >> infrance. >> You have one on like in >> when you go shopping when you go to the
grocery store when you're hungry. It's the same way, right? We want to do everything. We'll see. We'll see. Hopefully, we can execute on all of it. >> Yeah. Maybe should be clear though.
Inference X today. The name is because it's for inference performance, but it's about chips. It's about comparing chips and systems. And we're going to be releasing something called endpoint X
where we test the serverless inference endpoint providers and kind of dig into those details on uh cuz I think there's a very similar methodology. We know there's a a similar methodologies that
we should apply where we we go beyond just performance. We you know we look at cost, we look at reliability, we look at security, we look at monitoring infrastructure. There's all of this
stuff that goes into an inference endpoint that says just because somebody hits a higher uh target on batch one, you know, contrived tests doesn't necessarily mean they're the right
partner to scale up your inference requirements with when you need all that extra. >> Yeah, there's there's providers with better performance, like raw better
performance, better optimization engineers, but then they're like API is less stable, it's flakier. Um, you know, there there's there's all sorts of reasons why it may not be as good for
the endpoint end customer even though it's like more tokens per second per user and lower cost. >> Yeah. Amazing. Um, do you ever uh how will you cover the let's call it the the
non Nvidia universe. Um, so which is Ryzen. >> You have some in here, right? Like Tensor Wave is AMD cloud. Yeah, that's another one where I explicitly have an
investment by the way guys and and we still rank them poorly. um you know time intellect I had investment in the first cluster max I put we put them as underperform right I think like a lot of
this bias against like it's it's yes I might make an investment I'm going to talk to them and try and help them but at the end of the day the ranking is the ranking um so we we do want to extend
beyond uh but ultimately like you know fluid stack is not going to rent TPUs to us right because they're renting everything to anthropic right um >> maybe
>> we'll see I hope I hope >> I'm paying them >> yeah it's kind of unexcusable that they're unavailable Like, come on, guys. Like,
>> no. I mean, it's it's it's valid at the same time, right? It's like, look, it's unavailable to us, so we can't we can't we can't uh rent it. Won't be able to rent it either unless you're really
really freaking cool. Um because they're selling everything like SpaceX is not going to rent to us. They're not going to rent to you. They're not going to rent to, you know, random companies.
They're going to rent massive >> Dylan. I think I think you need to work on your uh I think you need to This is the first time I'm ever giving you this feedback, but I think you need to work
on your self-confidence, man. We we we can get it in there. I think we can get it in there with these guys. Yeah. >> I think over time like today >> a lot of these providers um even Fluid
Stack and and people that we know who are at SpaceX AI now um see value in reading the article and they see value in getting an outside opinion even if they're not necessarily going to provide
the service publicly that they offer to us. they they you know they they enjoy a third you know a technical opinion that's outside the company that's honest that's going to give them real feedback
and we're we're here to help for all of that stuff. Um to answer the question about multisilicon going to be huge next year like with the verbin generation with
Nvidia we're going to see it because they're going to bring the LPUs in from croc the LPX systems right so even if we're just testing Nvidia we're going to have multisilicon to test AMD obviously
has their own thing cerebrus is becoming a neocloud for open AI are building their own data centers right um moving from just inference endpoint stuff to like really managing clusters of their
chips for everybody else we see Samov coming to market. Lots of the chip startups like positron, etched, mad x, like lots of people are launching stuff like this. Um there's rumors, you know,
obviously fluid stacks got TPU outside of GCP. There's more stuff coming there. Tranium, lots AWS can produce a lot of chips. Maybe they won't self-build all of it in the future. They've got
announcements on heterogeneous tranium with cerebrus and stuff like that. So I I think there's going to be a lot more for that next year and and we're going to have to contend with multisilicon as
part of a comprehensive offering from the leading neoclouds. they will need to be able to support heterogeneous compute in the future >> and that's all focused on inference
mostly right like uh because of it's you know it's like >> I mean most anthropic for example uses TPUs for training primarily um so in that case it's sort of the opposite
right they don't actually most of their inference is on trainium and GPU most their training is actually especially like you know fable 5 was trained on TPUs for example for for the
pre-training and then the post- trainining and inference is mostly training and GPUs. >> So it's it's a multi-silicon. I would say in most cases other silicon
offerings are for inference but there are some cases where people are using it for training. >> Yeah. But the but I think the key thing is inference versus training is not this
hard split now where there's lots of forward passes during RL when you're doing research and when you're actually post-training the model. And so um >> in fact most of the workload is is
forward passes right and so you might call that classically inference even though it's for re you know forward passes for reinforcement learning. Um >> yeah inference is part of the loop. Yeah
>> there's a whole bunch of benefits of using uh let's say cheaper chips capable of high throughput for the forward pass NRL. But then there's a bunch of challenges with numerical correctness if
you're using two different chips. even just two different generations of Nvidia GPUs, B200 versus B300, you might you have different amounts of memory, different batch sizes. There's there's
all these effects. So, uh I I absolutely expect there to be heterogeneous silicon used across training and inference. Pre-training probably a homogeneous cluster. People probably not going to
risk that at like 400,000 GPU scale, but we'll see. >> Yeah, but like you know RL is like as big or sometimes bigger than the pre-trained uh workload right now,
right? And so that actually means that you need uh like maybe a a new form of disagregation where like you have the different kinds of compute in the same data center [laughter]
that you shuttle back. >> It doesn't even have to be the same data center, right? People are doing RL across many many sites, right? This data center is doing token generation. That
data center is doing token generation. And it's actually fine to do uh different types of chips there. It's just a very complex info problem because you're going to shard your model
differently and weight updates happen at different times and there's all this >> async for people listening to this right now are like god no please no [laughter] >> you have to data center multisilicon RL
updates on the fly god please no >> you're fighting speed of light how how are you going to transfer all this this data >> I mean there's interesting architecture
stuff being done on disagregated like decentralized RL Well, uh, the MIMO 2.6 that just came out, it's a good open paper on how they super scale up the RL side of things. Um, but on that point as
well, you mentioned a lot of the custom chips. Anything on Huawei, Chinese chips? Um, do you guys look at them? >> Yeah, we're tracking them. The Huawei chips are really uh getting better much
faster. For a while, they were like, you know, basically when they were banned by the US government, it took them like a few years to reset. um you know and the Ascend 910 B and C uh but now they've
launched the 940 950 and 960 in pretty quick succession. Um they you know we're trying to sort of get access to them and test them along all bounds, right? Whether it be renting them so we can
test the performance, whether it be um you know looking at the specs and the manufacturing capacity and things like that. um also trying to access them so we can do a reverse engineering analysis
out of our Oregon reverse engineering lab. Um we're trying to do work across all of these um dimensions. Um you know >> how's the how's the availability? Are there clusters like are the labs okay
without Nvidia? Any any early signs >> in general? Yeah, I mean people are super I mean the labs are using you know anthropic uses four different >> No, I mean like the the the Chinese labs
using their own chips. Yeah. So I think it was like um GLM said explicitly right that they had uh 50,000 chips uh that were not Nvidia. They were Chinese-made uh AI accelerators for
inference of their newest model. Um as we extend out to other firms um you know Cameracon, Ilvatar, Inflame, and many others, there's there's a handful of different companies. Um they're all
pushing up the curve. You know, obviously they're further behind Nvidia. Huawei is definitely the best of them all. Um and then uh BU is spinning out their chip business uh as a completely
separate company. Um so there's a lot of developments in the Chinese chip ecosystem and and especially for inference I think you know the Chinese you know as as frontier models um become
and even even now the open source models become better and better at writing kernels uh it's easier and easier to port stuff um and and so ultimately you're you're going to have more
diversity and heterogeneity uh which is why Nvidia's moved to this super fast pace of trying to release new chips every year and new systems every year because they know they have to run as
fast as possible at the speed of light otherwise they will get caught up to um because the CUDA mode is rapidly being deteriorated >> but it's just volume right Dylan like
the um these chips do work today but they can't produce enough of them to really serve all the domestic demand. >> So >> yeah I mean it's it's you know you know
if you look if you do an H100 equivalent or let's say if you do an GV200 equivalence um Nvidia's making you know millions and millions of chips, Google's making millions and millions of chips.
Amazon's making millions and millions of chips. AMD is making millions of chips, right? But then if you look at the Chinese companies, really, Huawei is the only one that even in gross chips,
they're making a million plus. Um, and then the Huawei chips are are behind. So then you have to discount them for that. Everyone else is still in the hundreds of thousands of units and scaling
production is very difficult, especially given export controls and stuff. Um, I think China will ramp production rapidly across next year and the year after. Um and they will get to even tens of
millions of chips produced by 2028. Um but it does take time. >> Yeah. I mean uh let's not forget that they they're the first to have a national plan around all this stuff. You
know when when they get going they get going in real scale like unlike we've ever seen in human human history. Um okay lots to discuss here. Uh congratulations on cluster max uh and
all the stuff that you guys are doing. Actually I think we managed to hit on like many of the major topics of like the year so far which is great. get a catch up on like the the hardware lab
that you guys are building in Oregon. I'd actually love to see it. We should we should do like some kind of trip. >> Yes. Fly out. Fly out. We will uh give you a tour.
>> Yeah. >> Yeah. Give me a heads up whenever you are going like we'll do a tour. Uh I want to >> Dylan will give you a ride in his orange
jeep. >> Um we I I want to go to Abalene. Like let's let's just do all these things, >> you know? Uh I mean obviously you guys have been there as well. Anyway, uh you
guys got to go. Uh it's great to catch up. Uh, congrats on everything and also like I think there's just a lot of more work to do. Jordan gave us a sneak peek at the the kind of things you're doing.
Uh, endpoint X sounds interesting. Uh, and like the the market is really segmenting, right? Like to me that's that's what you guys are identified early and are raising the standard for
everyone. So I think for you know on behalf of everybody like thank you for doing this. >> Thank you man. >> Thank you.
>> Yeah. Good to talk to you guys. Bye.