Rate Limits Explained: Claude, OpenAI & Gemini

0:00 One line in a log, 429, too many requests. It is the politest error on the entire internet. Nothing crashed, nothing is broken. A server just told your code in plain language to slow down

0:12 for a moment. Your code did not slow down. It caught the error, waited a beat, and sent the exact same request again. And this is rarely one request. Eight agents in a fleet all hit the same

0:23 wall inside the very same second. So, the second wave arrives as one clump instead of a spread. The wall says no eight more times immediately. Now, every one of those failures retries, too. The

0:34 traffic pushing you over the limit is your own reaction to it. Amazon put a number on this shape, five layers of services, three retries each, and the bottom sees 243 times the load. That is

0:45 how a 429 becomes an outage. Nothing broke. A speed bump got driven into until it became a wall. The fix is three ideas, and all three are older than you are expecting. A

0:56 bucket, a doubling, and a coin flip. Token buckets, exponential backoff, and jitter. Every serious API runs all three on both sides of the wire. None of them were

1:07 invented for AI. They were invented for radio packets bouncing between Hawaiian islands. That was 1970. 56 years later, they are still the answer because the failure has not changed.

1:18 So, what is actually sitting behind the limit you keep hitting? And why does waiting the wrong way make it worse? Start with the thing doing the rejecting because most code is written against a

1:26 counter that does not exist. There is no scoreboard that empties at the top of the minute. There is something simpler and much less forgiving. And once you can see it, the error messages stop

1:36 being mysterious. Picture an arcade with a single token machine. It drops one token into a bowl at a fixed rate, say one a second, and the bowl holds 60. Walk up when the bowl is full, and you

1:47 can spend all 60 at once. That is your burst. After that, you get exactly what the machine drips. That is a token bucket, and it takes two numbers to describe, not one. The refill rate,

1:58 which is your actual sustained throughput, and the bucket size, which is how much you are allowed to save up. People quote the first and get bitten by the second. Anthropic states it outright

2:08 on the page. The API uses a token bucket algorithm, and your capacity is continuously replenished up to your maximum limit, rather than being reset at fixed intervals, which kills the

2:18 mental model most retry code is built on. The same page spells out the consequence, and it is worth reading twice. A rate of 60 requests per minute might be enforced as one request per

2:29 second. So, the script that fires all 60 at once, once a minute, is not under the limit. It is over it, then idle, and you are rarely fighting one bucket on Claude's API 3 run at the same time.

2:41 Request per minute, input tokens per minute, output tokens per minute. Trip any one, and the response is the same status code, so the code alone does not tell you which wall you hit. OpenAI

2:52 counts five things instead. Request a minute, request a day, tokens a minute, tokens a day, and images a minute. Gemini counts three and applies them per project rather than per key. So, the

3:03 second API key somebody on your team created buys you nothing. GitHub does not count requests at all on its secondary limit. It counts points, where a read is one and a write is five. It

3:13 counts concurrency at 100 in flight, and it counts 90 seconds of CPU time per 60 seconds of real time. Six different rule books, which sounds hopeless until you notice they all tell you the answer is

3:25 sitting in the response your error handler threw away. These services attach headers saying what the limit was, what is left, and when it refills. And one header matters more than the

3:35 rest. Retry-After. It is 14 years old, RFC 6585, April 2012, written by Mark Nottingham and Roy Fielding. Its example server is almost sweet about

3:47 it. I only allow 50 requests per hour to to website per logged-in user. Try again soon. And the number in that example is 3,600. 1 hour in seconds handed to you for

3:58 free. Anthropic's version of the same idea is the one to tape to your monitor. The number of seconds to wait until you can retry. Earlier retries will fail. Not may fail, will. That number is not a

4:10 suggestion about queue depth. It is the time your bucket needs to hold one more token. It is a bouncer handing you a card with a time on it. Show up before then and you go back to the end of the

4:19 line. There is one precondition that only surfaces after it has cost you. Retrying is safe only if the request is safe to repeat. Amazon's own guidance is blunt. API

4:30 ES with side effects are not safe to retry unless they provide idempotency. A read costs nothing twice. A charge costs your customer twice. So, obey the header and you are fine. Except the header is

4:40 optional and its format is a free-for-all. RFC 9110 from 2022 lets retry after be either a count of seconds or a full date. Both legal, both shipped. Your parser has to handle both.

4:54 The other headers are worse. Anthropic sends reset times as RFC 3339 timestamps. OpenAI sends go duration strings. GitHub sends epoch seconds. Amazon has its own header in

5:06 milliseconds. Five encodings of come back later and the standard meant to unify them is still a draft. Number 11, dated May. So, sometimes nothing tells you and you fall back on a default that

5:17 is 56 years old and was built for a problem that looks nothing like an API 1970. The University of Hawaii where Norman Abramson had a problem cable solve and radio does not. He wired

5:28 campuses across the Hawaiian Islands together over a single shared radio frequency. Two terminals transmit at the same moment and both packets are destroyed. Both retransmit. Both collide

5:39 again. The channel works perfectly and no data crosses it. The fix was embarrassing in its simplicity. Before retransmitting, wait a random amount of time. 1976, Robert Metcalfe and David

5:51 Boggs publish Ethernet and add the second half. Each collision doubles the window you pick your random wait from. One collision, wait up to two slots. Two, up to four. Three, up to eight.

6:02 Binary exponential back-off, and it is still in the silicon, which is the same thing as knocking on a door. Knock, wait, knock, wait longer. You do not knock harder, and you do not

6:13 knock faster, because the reason the door did not open has nothing to do with your enthusiasm. Few people thought this mattered outside a cable until October 1986, when the internet did it at

6:24 national scale. Throughput between Lawrence Berkeley Laboratory and UC Berkeley, 400 yards apart in two hops on the network, fell from 32 kilobits a second to 40 bits a second. 40. 800

6:36 times slower over 400 yards. Van Jacobson called it a sudden factor of thousand drop in bandwidth, and spent close to two years working out why, because the hardware was fine the whole

6:46 time. What had collapsed was the software behavior on top of it. The paper he published with Michael Karels in 1988 is the one the internet still runs on, and

6:56 it contains this. For a transport endpoint in a network of unknown topology with a constantly changing population of competing conversations, only one scheme has any hope of working,

7:06 exponential back-off. Only one has any hope. That is a strong claim from 1988, and it is still the algorithm everything ships with. So, why does a fleet of perfectly

7:16 well-behaved clients still land as a spike? Because back-off fixes how long you wait. It does nothing about when the whole fleet waits together. A thousand clients get throttled in the same

7:25 millisecond. Each one backs off 50 milliseconds. Each one comes back in the same millisecond. You did not spread the load, you moved the spike, then doubled it, and moved it again in lockstep. It

7:36 is a cinema emptying. A thousand people through one door in the same second is a crush. The same thousand across 90 seconds is a walk. Nothing about the crowd changed, only

7:46 the arrival times did, and arrival times are the whole problem. Amazon says it in one sentence. If all the failed calls back off to the same time, they cause contention or overload again when they

7:56 are retried. Our solution is jitter. And jitter is one line. Do not wait the back off. Wait a random amount of time between zero and the back off. Mark Brooker published the arithmetic in

8:07 March 2015. Full jitter. Sleep equals a random number between zero and the capped exponential. With a hundred contending clients, that one change cut the calls needed to clear the whole set

8:18 by more than half. Not a tuning wind, a different regime. Eleven years later, that formula is not a blog post. It is the ship default in the AWS SDK, written the same way. Delay equals random zero

8:30 to one times the smaller of 20 seconds and the base delay doubled per retry. Same shape, from a suggestion to a library default. And the base delay depends on what

8:40 actually broke. A connection reset waits 50 milliseconds. A throttle waits a thousand, 20 times longer, because those are different messages. A reset says something glitched. A 429 says the

8:51 answer is no, and it stays no until capacity comes back, which leaves a question worth sitting with. How much of your provider's capacity right now is you retrying yourself? Because a polite

9:02 retry is still a retry. Back off and jitter change the shape of the curve. Neither reduces the total amount of work you are asking for. Google's answer, from the SRE book, is not a better

9:12 curve. It is a cap. Three attempts per request, and no client may let retries rise above 10% of its own traffic. Under that, retries are a rounding error. Over it, you are a load generator with a

9:24 service-shaped hobby. They go further. Each client tracks two numbers over the last two minutes, requested attempted and requested back end accepted. While things are healthy, those match.

9:35 As the back end starts refusing, accepts fall behind, and once requests are twice accepts, the client rejects its own calls before they leave the process. Amazon does the same job with a token

9:45 bucket, only pointed inward. The bucket does not limit requests, it limits retries. The budget starts at 500 tokens. A retry after a server error costs 14. A retry

9:56 after a throttle costs five. A request that succeeds first time puts one back. At zero, retrying stops. 500 divided by 14 is about 35. 35 failed retries and your client stops asking on its own.

10:09 Amazon publishes the crossover, too. Past roughly 22% sustained failures, the budget drains faster than success refills it. And their stated reason is the whole argument. Continued retrying

10:20 can delay recovery for everyone. For everyone, not for you, because a retry storm does not only make an outage worse, it can produce an outage that no longer has a cause. In 2021, four

10:32 engineers led by Nathan Bronson gave this a name. Meta-stable failure. A system that will not recover even after the trigger is gone. Their first example is retries, and the arithmetic fits in

10:42 your head. A database answers in under 100 milliseconds while it is under 300 queries a second. Your app runs at 280, comfortable. Then a switch blips for 10 seconds and

10:54 everything queued arrives at once. Each request with its retry behind it. 280 becomes 560. The database slows. Slow means timeouts. Timeouts mean retries, and 560 stays 560. The switch is fine

11:08 now. The trigger is gone. The system does not come back because the load is now made of its own failures. And here's the number no dashboard shows you. Recovery does not begin at 300, it

11:19 begins under 150. The paper calls that hidden capacity, the load you can climb back from as opposed to the load you can serve. A lift rated for 10 people that can only restart with three inside. You

11:31 paid for this distinction last October, AWS US East 1, where a DNS race condition emptied a DynamoDB endpoint just before midnight. The event did not close until the following afternoon, but

11:43 fixing the DNS record did not end the outage. The subsystem holding leases on every physical server had queued more work than it could finish, and AWS named the state in its own report congestive

11:54 collapse. The cure was throttling their own internal traffic and restarting hosts to drain the queues. 4 months earlier, Google's global outage report had been blunter still. A binary crash

12:05 looped in every region at once, and the write-up names the missing piece. Service control did not have the appropriate randomized exponential back-off implemented. US

12:15 Central 1 took 2 hours and 40 minutes to come back, not from the bug, but from the herd of its own restarts. So, the verdict, and it is not the one the title implies, back-off and jitter are table

12:25 stakes, not a strategy. Cap how many requests you allow in flight at once. Obey retry after exactly, then budget your retries, in that order. A retry is a bet that the failure was temporary,

12:37 and a 429 is the server telling you it was not. For transient faults, resets, and server errors, retry fast and let jitter do its job. AWS makes all of this the default for every SDK in November,

12:50 which is the largest API operator on Earth telling you your retry loop was a capacity decision all along. Which leaves one question I cannot answer for you. What is your hidden

13:00 capacity? Not the limit on your dashboard, the load you can climb back from.