Enjoying this issue?
Get tomorrow's AI & engineering digest in your inbox — hand-picked, summarized, and always spam-free.
TLDR
The September 3 outage that knocked out Claude, Grok, and ChatGPT was caused by a single failure at SpaceX AI's Memphis compute center, not independent incidents. Anthropic leases the entire Colossus 1 facility in Memphis for $1.25 billion per month, and Grok's parent company SpaceX AI runs the same campus. The real story is that frontier models cannot be quickly failed over because they require pre-loaded GPU racks, making vendor diversity an illusion when providers share physical infrastructure.
Key points
Three major AI services experienced outages within 90 minutes on September 3, 2024.
The cause was a failure at SpaceX AI's Memphis compute center housing 220,000 chips.
Anthropic leased the entire Colossus 1 facility from SpaceX AI for $1.25 billion per month.
Frontier models cannot fail over quickly because they require pre-loaded GPU racks.
Vendor diversity is not redundancy if providers share the same physical infrastructure.
Tools mentioned
Techniques
- Nvidia rack configured as a single massive GPU via shared memory
- Pre-loaded GPU rack model serving that prevents rapid failover
Stop scrolling. Start reading smarter.
Receive the day's most important AI & engineering updates in one concise email. No spam.
Transcript (captions)
At 26 past 6 in the morning, Pacific time, Anthropic's status page turned orange. 4 minutes later, Grok's status page followed. A different company with no connection anyone could see. By a
quarter to 8, ChatGPT was failing for some people. Three status pages, three rivals, one Thursday. Then 6 hours passed before anyone explained the overlap. That gap is where this starts.
So, here's the map. By the end, you'll know what broke and why the backup you think you have doesn't exist. Look at what stopped first. If you write software, you're inside one of these
tools most of the day. Anthropic measured how its own coding agent gets used. The average user has it running 20 hours a week. Downdetector logged over 35,000 reports for ChatGPT alone. Claude
and Grok cleared a thousand each. The internet did what it does. The jokes landed on one idea. People had just lost the thing they think with. That was funny for about a day until you notice
half your working week routes through somebody else's building. So, what takes down three companies that are actively trying to beat each other inside about 90 minutes? When services fail together,
the reflex is to look for a shared dependency, one wire underneath all of them. A content delivery network, which just means the layer that hands you the pages, or one cloud region sitting under
all three. Cloudflare sits under a big slice of the web, so it got named first. Its spokesperson didn't hedge. Cloudflare is not experiencing any significant service disruptions at this
time. Our services are operating normally, and any reporting that deviates from this is incorrect. Then Azure. Here's how you check that one yourself, and it costs you about 10
minutes. Microsoft publishes a public incident review for every broad failure, and it keeps 5 years of them on one page. So, open it and read down. The newest review is dated the 23rd of July,
6 weeks before the outage. For the 3rd of September, there is no entry at all. That's two suspects cleared, and yet a lot of the coverage said an Azure region on the East Coast about 90 minutes. The
status pages say just under 3 hours for Claude and roughly 3 and 1/2 for Grok. They were wrong about the cause and wrong about the length by more than half. And some coverage said Gemini went
down, too, which Google has never confirmed. It takes an hour to read four status pages. The headline takes 20 minutes. That leaves the question I actually want answered. If you're paying
two providers so you're never stuck, how would you even find out they're in the same building? At 12:38 Pacific, 6 hours after the first failure, one paragraph appeared from a company whose name
wasn't in any of the headlines. Here it is, word for word. "We are sorry for the issues you may have experienced with Grok following an outage at our Memphis compute center this morning."
And then the line most people scrolled straight past. "We'd also like to apologize to our impacted compute partners." That word, partners plural, is the whole story, because Grok's
parent isn't only a chatbot company. SpaceX bought the company behind Grok earlier this year, and the account posting that apology is SpaceX AI. It sells compute, which means it rents out
buildings full of chips to other labs. On the 6th of May, Anthropic leased one of those buildings, all of it, Colossus 1 in Memphis, over 220,000 chips under one roof. TechCrunch reported the bill
at about 1.25 billion dollars a month. That site went up in 122 days, and by the middle of this year, the Memphis campus was reported at around a gigawatt. That is roughly a mid-size
city's electricity pointed at one workload. It's a strange lease to sign, though. In February, Elon Musk had posted that Anthropic hates Western civilization. Three months later, he
took its money and handed over the keys. So, here's where we are. Claude's frontier capacity and Croc both lean on the same Memphis campus run by the same company. Two brands, one landlord.
ChatGPT's failure that morning is still unexplained. Open AI called it a routing error and it has not been tied to Memphis. Now, the part that explains the morning. The
obvious question is why they couldn't just fail over to another machine. And the answer starts with something that sounds like a detail. A frontier model doesn't run on a GPU. One Nvidia rack
wires 72 of them plus 36 processors into a single domain that shares 13.4 terabytes of memory. Nvidia's own page says it acts as a single massive GPU. Think of a
fridge-sized chip. The rack is the unit. Here's what actually happens in inside the rack. The weights sit in that shared memory and they stay there. A serving replica isn't a process you restart
somewhere else in a second. It's a rack that was loaded hours ago like a plane already fueled at the gate. So, you can't pour a big model into spare capacity during an incident. The
capacity has to have been holding it already. So, it gets bought, wired, and loaded months ahead in whole buildings. That's why the lease existed at all. So, say you send one request that morning.
First, it lands on a router in front of the API. That router can't conjure a rack, so it sheds you onto whatever pools are left. Then, those pools were sized for their own load, not yours.
Your request queues then fails. You can watch the shape of it on Anthropic's own page. 10:00 to 7:00 Pacific, it posts an exhaustive list of the broken models, Mythos, Fable, and Opus, more than one
version of each. At 8:25, only Opus 4.8 and Opus 5 are still failing. By then, everything smaller is back to baseline, and the impact ends at 9:16. So, the two biggest models took nearly another hour
on the same hardware in the same incident. A small model needs a fraction of a rack and squeezes into the leftovers. A big one needs a whole domain free at once. And the reason
there's no spare Memphis sitting idle is arithmetic. You double the capacity for failover, and you double the bill for half the utilization. At $1.25 billion a month, nobody buys that and leaves it
switched off. Now, price what that morning interrupted. Sonar found that 42% of committed code is now written by AI JetBrains. Ask working developers how much they still write themselves, and
one in five write literally zero code without it. Anthropic's own study puts the split like this. You still decide what to build, roughly 7 to 3. The agent decides how to build it, roughly 8 to 2.
That's the dependency in one line. It reaches company accounts, too. Salesforce's deputy CFO told investors they unleashed Claude in their R&D cycle, and that it's part of the reason
they didn't raise margin guidance for the year. And this wasn't one bad Thursday. Anthropic logged 25 incidents in August and six more in the first 3 days of September.
OpenAI's own scorecard puts ChatGPT's uptime for the quarter at a number that works out to about 8 hours of downtime. So, here's where I land. Vendor diversity isn't redundancy. If your
fallback for Claude is ChatGPT, you don't have a fallback. You've got a second phone number that might ring in the same room. I'd have signed that lease, too, by the way. In April, the
company said out loud that growth was straining its infrastructure. Dario Amodei's version, they planned for 10 times a year and got 80. That is a company buying under real scarcity and
for most of us 3 hours once beats anything we could build. But the only real hedge is separation you've checked yourself or weights on hardware you own. And next month Google moves 110,000
chips into that same complex. So how many logos does it take before three companies is really one address?