OpenAI System Design: Why They Ran Python at 20 Million Requests/Sec

0:00 Every time you open chat GPT, before you get a single word back, something has to go and find you. Your settings, your chat history, your memory, hundreds of separate lookups, and the slowest one is

0:11 the one you feel. For most of Chat GPT's life, that job ran in Python. At peak, 20 million requests every second. This week, OpenAI published how they pulled that off. In the same post, they said

0:22 they've replaced it with Rust. So, here's what we're doing for the next 20 minutes. We'll follow one system from the day Chat GPT launched to this week and watch it break four separate times.

0:33 By the end, you'll know why OpenAI ran their slowest language at their hottest layer and why that was the right call. You'll also get the four design decisions underneath it, which work at

0:42 whatever scale you're building at. The system is called Habitat, and it's OpenAI's online storage platform. Today, it takes more than 70 million requests a second across almost 40 regions holding

0:53 over 500 pabytes. It didn't start anywhere near there. It started as one Python file. Let's go back. Chat GPT launched on November 30th, 2022 as a free research preview. A million people

1:06 signed up in 5 days. 100 million in about 2 months. The fastest consumer product the internet had ever seen. For comparison, Tik Tok took 9 months to reach 100 million. Instagram took over 2

1:18 years. None of that growth is the interesting part, though. Growth is a graph. What matters is what growth does to a database underneath it. 11 months later, at OpenAI's first dev day in

1:28 November 2023, the numbers went up on stage. 100 million weekly users, 2 million developers, about 300 million tokens a minute through the API. At that same event, OpenAI shipped GPTs, the

1:41 custom chat bots you could build yourself. Those needed somewhere to put user data. That is the exact moment Habitat was born. It wasn't a service. It wasn't a database. Habitat was a

1:52 Python client library that product engineers imported talking to a single database behind it. The idea is a good one. Product engineers shouldn't have to think about database management. So the

2:02 library handled the boring parts. What kind of data this is, where it lives, whether the request is allowed, how it gets encrypted. You call a function, you get your data, that's the whole

2:12 contract. And it spread fast inside OpenAI without anyone mandating it, which tells you it was solving a real problem. Then the growth kept coming. Open AAI's own number for it is blunt.

2:22 More than 10 times year-over-year for three years straight. Most infrastructure teams build for 10 times and hope it holds for a few years. Open AI hit 10 times annually. 3 years

2:32 running. So before Habitat was ever the problem, something else broke first. That something was Postgress. For years, the database actually holding chat GPT was Postgress, not a fleet of them. One,

2:45 in January, an open AI engineer named Bohan Jang published the details and the architecture is strange for the scale involved. A single primary Postgress instance takes every right for the whole

2:56 product. Around it sit nearly 50 red replicas spread across regions. Together they serve millions of queries per second for 800 million users. No sharding. Sharding means splitting data

3:07 across many independent databases. So each one holds a slice. Postgress at OpenAI has never been sharted. The reason is about effort not technology. Sharting it would mean rewriting

3:18 hundreds of application endpoints and that's months of work maybe years. So instead they defended the single primary. To understand how you need to know how it kept falling over every

3:28 Postgress outage they had followed one shape. Something upstream spikes. Say the cache layer fails and every read lands on the database at once. Load climbs queries slow down. Requests start

3:39 timing out. Then clients retry which adds load which makes it slower which causes more retries. That's a death spiral and once it starts removing the original cause doesn't stop it. Hold on

3:51 to that idea because it comes back later in a much stranger form. There's also a specific reason Postgress struggles under heavy rights and it explains their whole migration plan. Postgress uses M

4:02 VCC multi- version concurrency control. Readers don't block writers because every update makes a new version of the row. The catch is the word every. Change one field in one row and Postgress

4:14 copies the entire row to make that version. Update a single yes or no flag on a wide row and you've written the whole row again. That's right. Amplification reads pay too. Your query

4:24 walks past all the dead old versions to find the current one and the indexes track every version until cleanup catches up. So their plan follows directly. Keep reads on Postgress where

4:34 it's excellent. Move write heavy work somewhere that shards. And they went further. New tables are banned. You cannot add a table to OpenAI's Postgress deployment anymore. Then they went

4:44 hunting for expensive queries. They found one that joined 12 tables together. And spikes in that single query had caused multiple high severity outages on their own. Most of those bad

4:54 queries came out of the ARM, the layer that turns your code objects into SQL for you. It writes the query, so the query goes unread. Connections were the next wall. An Azure Postgress instance

5:05 caps out at 5,000 connections and they'd already had incidents where a connection storm ate every one of them. Their fix was PG bouncer, a proxy that reuses a small pool of real connections for a

5:16 large number of clients. Average connection time dropped from 50 milliseconds to 5. That leaves caching and schema changes. And these two are the clever ones. Caching got a smarter

5:26 fix. When a popular key expires, every request that wants it misses at the same instant and stampedes the database together. So they added a lock. The first request that misses goes and

5:38 fetches the data. Everyone else waits for that one answer instead of piling on. Schema changes got the strictest rule of all. Only lightweight changes are allowed and there's a 5-second

5:48 timeout on them. 5 seconds. If your migration can't finish in 5 seconds, it isn't running in production. Here's what all that discipline bought. low double-digit millisecond latency at the

5:59 99th percentile, 59s of availability, and in 12 months, exactly one SE0, which is their highest severity incident. It happened in March 2025 when OpenAI shipped image generation inside GPT40,

6:13 and the internet turned itself into Studio Gibli, a million new users in a single hour. Over a 100 million new signups in one week, right traffic jumped more than 10 times. That's the

6:24 one thing that broke a single Postgress primary in a year. A meme. Step back for a second because you've now got a database you can't shard and can't add tables to. Where does all the new

6:34 product data actually go? That's Habitat's answer. And underneath it is Azure Cosmos D B which shards natively. But by the middle of 2025, Habitat had hit its own wall, not a performance

6:46 wall, a deployment wall. Habitat was a library, which means a copy of it ran inside every service that used it. Dozens of services, each with its own copy. Then, OpenAI wanted to move their

6:57 most critical data onto regionally distributed database accounts. So, one region failing wouldn't take everything down. That change needed new routing logic in the library. So, the roll out

7:07 went like this, and it's painful to read. Write the routing logic, hide it behind a feature flag, ship it to every client, wait for every team to deploy days. Then they realized they should

7:18 shadow the traffic first to check the new sharding logic was even correct. Another couple of days, then a bug turned up in that logic, fix it, and it's another couple of days to get the

7:28 fix everywhere. Finally, they were ready to flip the flag. At that moment, one team rolled back their service for a completely unrelated reason to a version with the old buggy client in it, which

7:38 caused the exact outage the entire project existed to prevent. That's the moment Habitat stops being a library. When your control logic ships inside other teams binaries, you don't control

7:48 it. You negotiate for it. So they pulled it out into a standalone service. One deployment, one place to observe, one place to improve. There's a second reason that matters more every year,

7:59 which is that a service is a choke point. You can enforce things at access control, audit logging, limiting what can reach raw storage at all. With agents now reading and writing user

8:09 data, one door beats 30. Right? So now the hard question. You're building a service that will eat tens of millions of requests a second. What do you write it in? They wrote it in Python on

8:20 purpose, knowing it was wrong. The post is unusually direct about this. They call it a strategic incursion of technical debt. Running Python as a service costs more than running it as a

8:30 library because you add a network hop, more CPU, more memory, higher latency. They wrote that the inefficiencies of Python would not be acceptable at a 100 times their scale, which made an

8:41 eventual rewrite almost certain. So they knew on day one that they were building something they'd have to throw away. They did it anyway because the real bottleneck wasn't CP you. It was product

8:53 engineers waiting on a platform. Speed of shipping beat cost of running. Then they made a bet I haven't seen another infrastructure team put in writing. They bet that by the time the rewrite was

9:02 unavoidable, their own coding models would be good enough to do it. Park that one. It pays off near the end. Choosing Python at that scale isn't a free pass, though. You still have to make it fast

9:13 enough that users don't feel it. Their own line on this is the best sentence in the whole post. When the average user request causes hundreds of database calls, the slowest database call is the

9:23 one the user feels. So, the entire game is tail latency, not the average. the worst few requests out of every hundred. Which brings us to the first thing that broke and it's the one every Python team

9:34 eventually meets. Python has a global interpreter lock. The gill only one thread runs Python code at a time per process. Async IO works around that for waiting. While one request waits on the

9:45 network, another can use the CPU. That's concurrency, but it is not parallelism. If two requests both want to do actual CPU work, one waits. There is no second lane. And Habitat does a lot of CPU

9:58 work, routing, compression, encryption, check sums, health checking downstreams, shadowing requests, hedging them. So picture a single thread with a queue of jobs. Storage comes back with your

10:09 answer in 2 milliseconds. And that answer just sits there. The core routine that would parse it has to wait its turn on the CPU. The database was fast. The waiting room was full. That's exactly

10:20 what they saw in traces of their slowest requests. downstream storage responding quickly and the request stalled waiting to be rescheduled. Here's how they measured it and you can steal this one

10:30 today. They schedule a tiny background task to run at a known time, then record how late it actually ran. That gap between when it should have run and when it did is your event loop delay. It's a

10:41 direct read on how starved your loop is. What they found was jitter of hundreds of milliseconds and in edge cases several seconds. Now, the worked example because one specific cause of that was a

10:53 config file. They use Statsig for feature flags, the experimentation platform OpenAI later bought for $1.1 billion in September 2025. By default, Statig pulls for fresh config every 60

11:06 seconds. No jitter, so every process pulls at the same moment. And the config it pulled contained every production rule for every service at OpenAI in one large JSON blob. Meanwhile, an unrelated

11:18 decision had them running up to eight Python processes per pod to push CPU usage higher. So, multiply it out. Once a minute, all eight workers on a pod stop serving traffic at the same instant

11:29 to parse the same giant file. Every pod, every minute. Request already in flight just sit there. They found it by profiling CPU on the live service, not in staging. The fix was three boring

11:41 changes. Ship a smaller config. Pull less often. add jitter so the processes don't move as one. The structural fix was the opposite of what you'd expect. Give each process fewer concurrent

11:52 requests, then run far more processes. If one thread can't be shared, well, stop trying to share it, which solves the CPU problem and creates the next one because now you have an enormous number

12:02 of processes and every one of them opens connections. This is where the story gets weird and it's my favorite part. They had a burst of traffic from one client that overloaded part of the

12:12 service. normal enough, they stopped that client and the service didn't recover. A subset of processes stayed degraded, then got worse, taking on more and more requests until someone

12:22 restarted them by hand. Think about how strange that is. The cause was gone and the effect was growing. Something was feeding it. That's a metastable failure. The trigger sets off a feedback loop and

12:33 the loop keeps the system down long after the trigger leaves. Here's the mechanism in four steps. Say your fleet is three server processes and one of them is already overloaded. One, clients

12:44 pull their connections and reuse them rather than opening new ones. Two, Python's Osync HTTP client picks the most recently returned connection first. Last in, first out. Three, a slow

12:55 overloaded process takes longer to answer, so its connections come back to the pool last. Last back means picked first. Four, the pool hands the next request to the slowest process in the

13:05 fleet, which makes it slower, which makes it return even later, which makes it get picked again. The pool was sorting by which server was worst and rewarding it. A sensible default became

13:15 a machine for concentrating load on whatever was already dying. Before they fixed it, some processes carried five to 10 times the concurrent request of the average. The fix is one word, first in,

13:26 first out. Take the connection that's been idle longest instead of the freshest one. That breaks the loop and it evened out their steadystate load as a bonus. Now, here's the detail I love.

13:36 Open AAI's post links out to the research paper that named this failure class. That paper is from 2021 and its first author is a systems engineer called Nathan Bronson. Bronson is now a

13:47 member of technical staff at OpenAI. He also spent 9 years at Facebook and we're about to run into his other paper. One more thing before we leave the fan out. All those processes hammer everything

13:58 downstream too which is the thundering herd problem. Their answer is envoy a proxy sitting in front of the traffic. It takes many HTTP1 connections from the Python processes and multipplexes them

14:09 into far fewer HTTP2 connections. Same requests a fraction of the sockets because it's one place. It's also where rate limits and circuit breakers actually work. So that's three problems

14:20 patched. But patching a language only takes you so far. And it isn't the real reason Python survived here. The real reason is that Habitat does less on purpose. This is the design decision I'd

14:30 argue is the most valuable thing in the post. Habitat doesn't let you write SQL at all. It exposes a small NoSQL API and nothing else. Their reasoning is a sentence worth writing down. It is cheap

14:42 and easy to write SQL queries that are expensive and hard to run. That's the cost imbalance. 2 minutes of your typing can cost a database an hour of CPU and nothing in the language warns you.

14:52 Remember that 12 table join that kept taking down Postgress. Somebody wrote that in about a minute before Habitat existed. They reviewed every query and schema change by hand which works with

15:01 one team and stops working the moment you have 40. So they remove the ability instead of policing it. No unbounded queries, no arbitrary joins. The shape they chose is objects and edges. You

15:13 define your object types. You define the edges between them. And you can read an object or read its direct edges. Think of it as a graph. You're allowed to touch one hop at a time. Say you want a

15:22 user's conversations. You ask for the user object, then you ask for that user's direct edges. You cannot ask for a chain of five hops in one query. They say where it came from, too. It's

15:33 inspired by Tao, which is Facebook's data store for the social graph, published in 2013. Which is where that name comes back. The first author of the Tao paper is Nathan Bronson, same

15:43 engineer now at OpenAI, whose failure paper they cited in act. Tao's own paper gives one of its three motivations as this. In a look aside cache, the control logic runs on clients that don't talk to

15:55 each other, which multiplies failure modes. That is the same sentence as OpenAI's reason for killing the Habitat client library 12 years apart. The lesson didn't get relearned, it got

16:05 rehired. The partitioning follows the same logic. An object and its edges live together in one partition. So reading them is one trip. But an edge pointing at a distant object gets no such

16:16 treatment. Follow it. And you might be hitting a different database account in a different region. That's not an oversight. Making traversals visibly expensive is the whole point because it

16:26 pushes that cost into your application where you can see it. For teams that really do need complex queries, there's an escape hatch. Habitat streams its changes out to a separate analytic

16:36 system called Rockset, and you query that copy instead, which Open AI also bought in June 2024. Their reads and their rights now live in two systems they own, kept in sync, deliberately

16:47 kept apart. So, a question for you at the halfway mark. Look at your own stack. How much of your database load comes from queries nobody would have written if the system had simply refused

16:57 them? Okay, four problem solved. One language patched to its limit. Now the part the title promised. By the time they moved, Habitat was the second largest service at OpenAI by core count

17:08 and fourth by proxy footprint. Python had run out of road. At its peak, it had served more than 20 million requests a second, which is a real achievement and also a warning sign. In the second

17:18 quarter of this year, they rewrote the entire service in Rust. Two engineers, two engineers plus codeex and GPT 5.5. and GBT 5.5 shipped on April 23rd in the same quarter as the rewrite. That's the

17:32 bet from 2025 paying out. They deferred it believing their models would make it cheap, and the model they used was weeks old when they used it. The numbers are what you'd hope for, six times more CPU

17:43 efficient, 15 times more memory efficient, meaningfully lower average, and tail latency. As of this week, Rust is serving 95% of production requests. Let me correct the framing now,

17:54 including my own title. This was not deleted and it wasn't done in the dark. Open AAI announced it in a blog post and on their developer account on X. Python is still serving traffic today and they

18:05 expect to deprecate it fully in the coming weeks. It's also one service, not OpenAI's back end. Python is still everywhere else in that company. The actual sequence is better than the

18:14 dramatic one. Anyway, a team ran a language past its limit for years on purpose, then left on their own schedule. So, what happens next? A second post is coming covering

18:24 multi-tenency, read performance, and how they scaled their Cosmos DB partnership. Postgress is still uncharted, still one primary, still nearly 50 replicas. They're testing cascading replication

18:36 with Azure, where replicas feed other replicas to get past a hundred of them. And they're hiring for Habitat in Seattle, describing it as a full stack, one sizefits- database platform, which

18:47 tells you the ambition is bigger than storage. So here's my verdict and it cuts against what the title implies. The Rust rewrite is not the lesson. The lesson is the years of Python before it

18:57 and OpenAI won that argument. Deliberate dated technical debt beats premature optimization for any team whose load might grow by an order of magnitude. That's the call I defend. Three

19:08 receipts. They shipped products instead of tuning run times. They got to 20 million requests a second anyway. And the repayment cost two engineers in one quarter. The concession is real though.

19:19 and you should hear it. This only works if you write the debt down and name the trigger that repays it. They knew the exact condition, which was a hund time scale. They knew the exact instrument,

19:29 which was their own models. That isn't procrastination. That's a scheduled payment. Skip that part and you get a frozen Python service and a team that can't say why. And if you're not growing

19:39 10 times a year, most of this doesn't apply to you. Your single Postgress instance is fine. That's the steelman and I'll take it. But the four decisions underneath do apply at any size. Take

19:50 the control logic out of your clients because anything you can't deploy on your own is not yours. Make expensive operations impossible rather than discouraged because reviewing queries

20:00 stop scaling before your database does. Measure the thing under the thing. Event loop delay not CPU percent. The gap between expected and actual is where your tail latency hides. and write your

20:13 debt down with a trigger on it because undocumented debt is the kind that doesn't get repaid. Which leaves one question I don't know the answer to, so I'll hand it to you. Open AI's bet was

20:23 that their own models would pay off their debt later and that bet just worked for two engineers on one service. If a rewrite costs a quarter instead of a year now, does technical debt stop

20:33 being a compromise? Does it become the correct default? I don't think the industry has worked that through yet. If you've got a view on it, put it in the comments. If you learned something new

20:42 here, subscribe for your daily dose.