The Outage That Proved AI Has No Plan B

summarized

TLDR

The September 3 outage that knocked out Claude, Grok, and ChatGPT was caused by a single failure at SpaceX AI's Memphis compute center, not independent incidents. Anthropic leases the entire Colossus 1 facility in Memphis for $1.25 billion per month, and Grok's parent company SpaceX AI runs the same campus. The real story is that frontier models cannot be quickly failed over because they require pre-loaded GPU racks, making vendor diversity an illusion when providers share physical infrastructure.

Key points

Three major AI services experienced outages within 90 minutes on September 3, 2024.

The cause was a failure at SpaceX AI's Memphis compute center housing 220,000 chips.

Anthropic leased the entire Colossus 1 facility from SpaceX AI for $1.25 billion per month.

Frontier models cannot fail over quickly because they require pre-loaded GPU racks.

Vendor diversity is not redundancy if providers share the same physical infrastructure.

Tools mentioned

Techniques

  • Nvidia rack configured as a single massive GPU via shared memory
  • Pre-loaded GPU rack model serving that prevents rapid failover
Transcript (captions)

0:00 At 26 past 6 in the morning, Pacific time, Anthropic's status page turned orange. 4 minutes later, Grok's status page followed. A different company with no connection anyone could see. By a

0:13 quarter to 8, ChatGPT was failing for some people. Three status pages, three rivals, one Thursday. Then 6 hours passed before anyone explained the overlap. That gap is where this starts.

0:26 So, here's the map. By the end, you'll know what broke and why the backup you think you have doesn't exist. Look at what stopped first. If you write software, you're inside one of these

0:37 tools most of the day. Anthropic measured how its own coding agent gets used. The average user has it running 20 hours a week. Downdetector logged over 35,000 reports for ChatGPT alone. Claude

0:50 and Grok cleared a thousand each. The internet did what it does. The jokes landed on one idea. People had just lost the thing they think with. That was funny for about a day until you notice

1:02 half your working week routes through somebody else's building. So, what takes down three companies that are actively trying to beat each other inside about 90 minutes? When services fail together,

1:13 the reflex is to look for a shared dependency, one wire underneath all of them. A content delivery network, which just means the layer that hands you the pages, or one cloud region sitting under

1:24 all three. Cloudflare sits under a big slice of the web, so it got named first. Its spokesperson didn't hedge. Cloudflare is not experiencing any significant service disruptions at this

1:36 time. Our services are operating normally, and any reporting that deviates from this is incorrect. Then Azure. Here's how you check that one yourself, and it costs you about 10

1:46 minutes. Microsoft publishes a public incident review for every broad failure, and it keeps 5 years of them on one page. So, open it and read down. The newest review is dated the 23rd of July,

1:59 6 weeks before the outage. For the 3rd of September, there is no entry at all. That's two suspects cleared, and yet a lot of the coverage said an Azure region on the East Coast about 90 minutes. The

2:11 status pages say just under 3 hours for Claude and roughly 3 and 1/2 for Grok. They were wrong about the cause and wrong about the length by more than half. And some coverage said Gemini went

2:23 down, too, which Google has never confirmed. It takes an hour to read four status pages. The headline takes 20 minutes. That leaves the question I actually want answered. If you're paying

2:35 two providers so you're never stuck, how would you even find out they're in the same building? At 12:38 Pacific, 6 hours after the first failure, one paragraph appeared from a company whose name

2:46 wasn't in any of the headlines. Here it is, word for word. "We are sorry for the issues you may have experienced with Grok following an outage at our Memphis compute center this morning."

2:56 And then the line most people scrolled straight past. "We'd also like to apologize to our impacted compute partners." That word, partners plural, is the whole story, because Grok's

3:07 parent isn't only a chatbot company. SpaceX bought the company behind Grok earlier this year, and the account posting that apology is SpaceX AI. It sells compute, which means it rents out

3:19 buildings full of chips to other labs. On the 6th of May, Anthropic leased one of those buildings, all of it, Colossus 1 in Memphis, over 220,000 chips under one roof. TechCrunch reported the bill

3:34 at about 1.25 billion dollars a month. That site went up in 122 days, and by the middle of this year, the Memphis campus was reported at around a gigawatt. That is roughly a mid-size

3:46 city's electricity pointed at one workload. It's a strange lease to sign, though. In February, Elon Musk had posted that Anthropic hates Western civilization. Three months later, he

3:58 took its money and handed over the keys. So, here's where we are. Claude's frontier capacity and Croc both lean on the same Memphis campus run by the same company. Two brands, one landlord.

4:11 ChatGPT's failure that morning is still unexplained. Open AI called it a routing error and it has not been tied to Memphis. Now, the part that explains the morning. The

4:21 obvious question is why they couldn't just fail over to another machine. And the answer starts with something that sounds like a detail. A frontier model doesn't run on a GPU. One Nvidia rack

4:32 wires 72 of them plus 36 processors into a single domain that shares 13.4 terabytes of memory. Nvidia's own page says it acts as a single massive GPU. Think of a

4:45 fridge-sized chip. The rack is the unit. Here's what actually happens in inside the rack. The weights sit in that shared memory and they stay there. A serving replica isn't a process you restart

4:56 somewhere else in a second. It's a rack that was loaded hours ago like a plane already fueled at the gate. So, you can't pour a big model into spare capacity during an incident. The

5:07 capacity has to have been holding it already. So, it gets bought, wired, and loaded months ahead in whole buildings. That's why the lease existed at all. So, say you send one request that morning.

5:19 First, it lands on a router in front of the API. That router can't conjure a rack, so it sheds you onto whatever pools are left. Then, those pools were sized for their own load, not yours.

5:31 Your request queues then fails. You can watch the shape of it on Anthropic's own page. 10:00 to 7:00 Pacific, it posts an exhaustive list of the broken models, Mythos, Fable, and Opus, more than one

5:45 version of each. At 8:25, only Opus 4.8 and Opus 5 are still failing. By then, everything smaller is back to baseline, and the impact ends at 9:16. So, the two biggest models took nearly another hour

5:59 on the same hardware in the same incident. A small model needs a fraction of a rack and squeezes into the leftovers. A big one needs a whole domain free at once. And the reason

6:10 there's no spare Memphis sitting idle is arithmetic. You double the capacity for failover, and you double the bill for half the utilization. At $1.25 billion a month, nobody buys that and leaves it

6:23 switched off. Now, price what that morning interrupted. Sonar found that 42% of committed code is now written by AI JetBrains. Ask working developers how much they still write themselves, and

6:35 one in five write literally zero code without it. Anthropic's own study puts the split like this. You still decide what to build, roughly 7 to 3. The agent decides how to build it, roughly 8 to 2.

6:49 That's the dependency in one line. It reaches company accounts, too. Salesforce's deputy CFO told investors they unleashed Claude in their R&D cycle, and that it's part of the reason

7:01 they didn't raise margin guidance for the year. And this wasn't one bad Thursday. Anthropic logged 25 incidents in August and six more in the first 3 days of September.

7:12 OpenAI's own scorecard puts ChatGPT's uptime for the quarter at a number that works out to about 8 hours of downtime. So, here's where I land. Vendor diversity isn't redundancy. If your

7:24 fallback for Claude is ChatGPT, you don't have a fallback. You've got a second phone number that might ring in the same room. I'd have signed that lease, too, by the way. In April, the

7:34 company said out loud that growth was straining its infrastructure. Dario Amodei's version, they planned for 10 times a year and got 80. That is a company buying under real scarcity and

7:46 for most of us 3 hours once beats anything we could build. But the only real hedge is separation you've checked yourself or weights on hardware you own. And next month Google moves 110,000

7:59 chips into that same complex. So how many logos does it take before three companies is really one address?

Frontier News · by Hyperjump Technology