Scale Your API to 1 million requests Per Second

0:00 This month, a public benchmark posted a number. One machine, one web server, just over 3 million requests a second. Not a cluster, not a region, a single box with 64 cores running one process

0:12 doing all of it by itself. So, a million a second is about a third of one machine. On paper, this video is over before the intro finishes. And yet, the service you maintain probably starts

0:22 wobbling somewhere in the low thousands on more hardware than that. That gap has almost nothing to do with your language, your framework, or the rewrite you keep postponing until next quarter. It is

0:33 four things that number leaves out, plus a fifth thing your cloud provider charges you for by default. Five walls, in order of what they cost you. So, this is the number, and everything standing

0:43 between you and it. The board is called HTTP Arena. Same machine, same load generator, every entry, all of it published in the open. It exists because the benchmark this industry quoted for

0:55 more than a decade was archived 5 months ago. For scale, Cloudflare's own network averages 81 million requests a second. A million is a bit over 1% of that. TechEmpower archived its framework

1:06 benchmarks in March. The announcement gave no reason for it and named no successor. Its last results are round 23, from February of last year. The replacement runs 30 profiles where the

1:17 old one led with one. 157 entries, 30 profiles, one Ryzen Threadripper, every score the best of three runs. And that is the whole trick, because the number people quote comes from the first

1:29 profile. Start at the top. The leading entry serves just over 3 million requests a second on the baseline profile. And baseline, in the project's own words, is the primary throughput

1:39 test. Persistent connections, one request at a time, a small response the server already holds. No parsing, no lookup, no decision. One column over sits the pipeline profile at 19 million,

1:51 which is a number no browser and no mobile client is going to hand you. Ask that same entry to serialize 50 items into JSON and it lands at 1 million 90,000. The body is no longer a constant

2:03 and building it cost about 2/3 of the headline. Wrap that in TLS and it settles at 850,000. Encryption takes about a fifth of what was left, which makes it the cheapest

2:13 wall in this entire video. Then ask it one question. A single pulled asynchronous Postgres query per request. 275,000. Same box, same entry, same run. 11 times

2:25 below the headline. And that 11 is a ratio between two profiles of one run, not a decline over time. Nothing got slower. You added a round trip and the round trip owns the ceiling now. The

2:36 same board hints at the fixed one column over. On the create, read, update and delete profile, those entries score more than twice their database number and the reason is printed in their own

2:46 description. That implementation keeps an in-process cache and invalidates it on right, which is where the argument you keep having at work falls apart. On the plain profile, the entry at the top

2:56 of that board beats Flask by 31 times and Django by 100. Put a database behind both and that lead against Flask collapses from 31 times to 6 and 1/2. The rewrite bought a wide gap at doing

3:08 nothing and most of it evaporated the moment there was work. The shape repeats down the whole board. The Rust entry falls from 2.7 million to 149,000

3:18 on the same journey. A factor of 18 paid to a database it did not write. So, the framework was not your ceiling. It sets how cheaply you can do nothing and nothing is not what you are shipping.

3:29 The 11 times from one round trip, hold on to that because every wall left is a variation of it. The board runs a profile that rarely gets quoted and it is the one that looks like real traffic.

3:40 Short-lived, identical work except the client hangs up and reconnects after every 10 requests. The leader gives up about a fifth to those handshakes. One of the raw engines falls from 3.7

3:51 million to under a million, nearly 3/4 of its throughput gone. Same server, same responses, same work, most of it now spent on setup and teardown rather than on answering anybody. Which sets up

4:03 the cleanest experiment on the whole board because two entries there are the same server twice. Same handlers, same language, same machine, same round. One runs on E poll. The other runs the newer

4:15 kernel interface, IOU ring. On short-lived connections, the newer interface is 2 and 1/2 times faster. That is a measured controlled win and it is exactly what you would predict

4:25 because that profile is almost pure system calls. But on the database profile, the gap is 3 and 1/2 percent. On the static profile, the newer interface is 19%

4:34 slower. And on the mixed workload with 16 CPUs, it is 15% slower still. So, the kernel rewrite pays where the system calls are and it does not pay anywhere else. It is a fix for one

4:45 specific wall, sold to you as a fix for the number. Now, the figure that should end most of these arguments. On that same machine, 1 and 1/2 million requests a second over TLS,

4:56 mean latency 19 milliseconds, P99 983 milliseconds, almost a full second at the exact throughput being chased. And both entries land in the same place there, so neither interface caused it.

5:09 That is simply what the load costs on that box. So, which number did you actually promise anyone? The one on the slide or the one your slowest customer sits

5:17 through? Mark Brooker, a distinguished engineer at Amazon Web Services who spent years building their compute, storage, and database services wrote about the price of that tail last month.

5:27 His framing is the useful one because it treats latency as a capacity problem rather than a customer experience one. For his work distribution, the requests at or above the 99th percentile carry

5:38 about half of the mean latency and everything at or above the median carries about 99% of it. In real services, he writes, that top percentile share is often past half and

5:49 sometimes past three quarters. Tails, in his words, tend to be disproportionately expensive to serve and so disproportionately important to focus on as we think about optimization. Little's

5:59 law makes that concrete. Concurrency is arrival rate times latency. So, on the encrypted profile, roughly 29,000 requests sit in flight at any moment. On the static profile, at a slightly higher

6:10 rate, about 600 do. 47 times the concurrency between those two profiles at lower throughput. Every one of those in-flight requests is a threat, a buffer, a connection slot, and a lock

6:22 you are paying rent on, which means the sharper version of the question is not how fast one process can go, but what happens in the space between your processes. One team published exactly

6:31 that this year with graphs. Zalando's product Red API serves over a million requests a second across about 25 European markets, and raw throughput was not their problem. Connor Gallagher, a

6:43 senior principal engineer there, wrote it up in June. A single batch request fanned out to as many as 100 parallel internal calls, and every one of those hops crossed the shared cluster proxy.

6:53 His post puts it plainly, for a single batch of 100 requests, their API had 100 times the exposure to that proxy. The proxy itself was fast, a couple of 100 microseconds per hop against a typical

7:04 product request of about a millisecond, but a couple of 100 microseconds taken 100 times is no longer a rounding error. Worse, during incidents, the team could not be certain whether a latency spike

7:15 came from the proxy or from their own code because the routing sat in the hot path of every request and they did not operate it. The repair started with visibility. Once they logged the exact

7:26 destination, the pod address they were calling, and the node underneath it, a pattern surfaced that shared metrics had hidden. Individual nodes freezing for about 2 seconds at a time. Therefore,

7:36 they took the proxy out of the internal path. Routing moved inside the calling process, a consistent hashring, 100 virtual nodes per endpoint, a 30-second

7:45 fade in for new pods, and bounded load sized with Little's Law, so no single pod gets buried. Consistent hashing is what makes that safe at their scale. Adding or removing an endpoint,

7:56 Gallagher writes, remaps only about 1/n of the keys, which minimizes cache churn, so the pod local caches survive a deployment instead of starting cold. The proxy fleet went from over 50 pods

8:08 to eight, and that fleet's bill went from about $450 a day to about 110. Spreading load by how busy each pod actually was, rather than by how many requests it had been sent, then took

8:19 roughly another quarter off the back-end fleet. They did not buy the million a second by adding a hop. They made it predictable and cheaper by deleting one. Which leaves the wall that has nothing

8:30 to do with your code at all. The default throttle on Amazon's API Gateway is 10,000 requests a second per account per region. That is 1% of the target, and in 13 regions, the default is 2,500,

8:42 a quarter of 1%. So, you need a hundredfold quota increase before you write a line. There is a burst allowance on top, a token bucket holding 5,000, and Amazon's

8:52 own documentation says that one is set by the service team and is not a quota a customer can request changes to. Then there is price. The REST tier bills $3.50 per million

9:02 requests. So, at a million requests a second, that is $3.50 every second. The tier step down as volume climbs, which lands a full month at $6 million. The cheaper HTTP tier lands at 2.3

9:15 million before a single byte of data transfer. So, here is the actual answer, taken from the profile on that board that looks most like your service. Mixed

9:24 endpoints, a database in the path, and the server capped at four CPUs and 16 GB of memory. The best entries there land between 32 and 68,000 requests a second. 1 million

9:36 divided by that is 15 to 32 replicas, 60 to 128 virtual CPUs. That is a rack, not a research project. The boring horizontal build wins, and it is not close. Small stateless replicas behind a

9:50 layer 4 balancer, one data access per request, and a cache in front of that access. The 5% where that is the wrong call is uniform, cacheable traffic, ad bids, telemetry, plane lookups, and

10:03 there the tuned single box is right. And the same board proves it. I would still take the boring stack at twice the machine count because the person paged at 3:00 in the morning can reason about

10:12 it, and a hand-tuned kernel path is a thing you have to keep being good at forever. Gallagher, who actually built the exotic version and shipped it, ends his own post this way. Should you build

10:22 your own? For almost everyone, the answer is don't. So, here is a dated bet. A year from now, the top entry on that four CPU profile is still under 100,000 requests a second because the

10:33 ceiling there is the round trip and not the run time, which leaves the question worth arguing about. If the number was never the hard part, what has your architecture actually been optimizing

10:43 all this time?