DeepSeek V4.1 Flash: The Architecture That Crushed Cache Costs by 437×

0:00 If you run a coding agent, most of your bill isn't the code it writes. The same context sent again every turn. You pay to reread what you already sent. Price at work on the big American models, and

0:12 that one line item is about 2/3 of the total. Deepseek just shipped a model that charges 3/10en of a cent per million tokens for it. That isn't a discount. They rebuilt attention until

0:24 the line item almost stopped existing. By the end, you'll know exactly how. in four steps and you'll know where it breaks. Here's the number it all turns on. Bytes of memory per token of

0:35 context. On DeepSk's first model, one token cost 389,000 bytes. On the one they shipped this month, that same token cost 890. Same company, four generations, 437

0:50 times smaller. And here's what shouldn't work. The new model is nearly twice the size of the one it replaces. Bigger, cheaper, faster, better. One of those four is supposed to be a trade. They

1:02 shipped it on the 10th of September and the post is modest about it. Smarter, faster, more efficient. 552 billion parameters, weights on the table, MIT license benchmarks are what most of the

1:15 coverage led with. So, let's clear those early. On aentic coding, it ties Claude Opus 5. On the hardest terminal benchmark, it loses to Claude Opus 5 by 20 points. Now the invoice because this

1:28 stops being abstract the moment you see it. A developer posted his own token log from a single long coding task on a medium codebase. 447 turns. Across those turns, the agent sent about a million

1:41 tokens of fresh input. It also read 36.5 million tokens back out of cash. That's the same context returning turn after turn about 36 times over. priced on GPT six Astra the session bills $55 and

1:58 twothirds of it is the cash line alone run the identical log on the new deepseek and it bills 36 so that's where the money goes which leaves the question what is that line item and how do you

2:10 shrink it while making the model bigger start with what you're actually paying for a model reading your prompt doesn't keep the words it keeps what each layer thought about each word that's the key

2:21 value cache and we'll just call it the cash. Think of it as the notes on the desk that stop you rereading the book. Notes take space and they take work to write. So, one line item hides two

2:33 costs, reading your prompt and keeping the bites afterwards. Deepseek went after both and the move sounds wrong the first time you hear it. 40 layers cut in half. The bottom 20 are the encoder. The

2:46 top 20 are the decoder. A single seam in the middle. When a prompt arrives, only the encoder's 20 layers actually run over all of it. The decoder, the top half, doesn't read your prompt at all.

2:58 So, where does the top half get its notes? Its global keys and values are never computed from each decoder layer's own hidden state. They're projected once, straight off the encoder's final

3:09 hidden state through a separate weight matrix per decoder layer. One pass, 20 copies, no second read. So, the arithmetic falls out. Reading a prompt costs half a network instead of a whole

3:21 network. The report puts preo complexity at order n* l over two and calls that nearly having preful computation. Writing isn't the same job. A token the model generates has to pass through all

3:34 40 layers for real. So decode activates 16 billion parameters against prefalls 8. Every provider charges less for input but only because preful batches. Well, here the network itself is lopsided.

3:48 Once the seam exists, one thing is left over. Every layer keeps its own sliding window, 128 tokens wide, built from its own hidden state. Rebuilding the decoder's windows exactly means

4:01 replaying that window for 20 layers. Thousands of tokens of work for windows the next turn will slide past anyway. Bounded replay says no. It replays only that last window, rebuilds them

4:13 approximately, and accepts that the answer isn't bit identical. 20 times less work, and the paper says so plainly. Let's walk one request through it. Say your agent dumps 200,000 tokens

4:26 of repository into the prompt. The encoder's 20 layers run across all of it at 8 billion active parameters a token. The decoder's 20 layers touch only the last 128. The cache that lands is 178

4:40 megabytes. The reply then streams out at 16 billion active a token. Next turn, the whole repository comes back, hits the cache, and costs 30,000 of a cent. On GPT6 Astra, that same read is 20

4:54 cents. But having the layers that read a prompt removes no bytes at all. So what shrank the bytes? There are only three places a cache like this can shrink, and the report names all three. the size of

5:06 an entry, the number of entries along the sequence, and the number of layers that keep one at all. Every earlier trick in this field picked one. This picks all three, and they multiply.

5:18 Entry size goes first. 64 query heads share a single latent and the whole layer has a single key value head. The sequence dimension is next, and it's the simplest idea here. Across the encoder,

5:31 every two tokens get compressed into one cache entry. half as many entries because neighboring tokens in a repository dump aren't independent. Then the layer dimension which does the heavy

5:42 lifting. Walk the shipped config and only four of the 40 layers own a cache at all. Layers 2, 8, 14, and 20. Every other layer keeps nothing. You don't have to take that from a paper. It's in

5:55 a comment on the attention class in their code. Compress ratio greater than zero does not mean the layer compresses its own cache. Only the source layers do. The rest read that same cache. So

6:06 what do the other 36 do? Each gets a borrowing mode permanently at training time. A reindex layer takes the shared keys and rescores them with its own query. So it picks its own short list. A

6:19 reuse layer doesn't even do that. It takes the previous layer short list and just attends to it. No scoring. 11 kernels a token instead of 15. Neither mode writes a bite of cache. most of the

6:32 stack reading four layers notes. After that precision, the surviving cache is stored four bits to a number with one 8-bit scale across every 16 channels quantized after the rotary embedding

6:46 against the previous generation's 8-bit cache that halfs it again. Add all four together and you get the number the paper leads with 890 bytes a token in the fast memory on the card. So a

6:58 million tokens of context is 890 megabytes. A single card could hold about 90 of those and still have room for the weights. Run the same sum on their first model and it stops being

7:10 funny. At that first rate, a million tokens of context would have needed 389 GB, five cards worth of memory before a single weight loads. So those are the four generations, each step smaller than

7:23 the last, stacked across 34 months. Now decide something before we go on. Four layers out of 40 hold every key this model will look at and the other 36 read somebody else's notes. Would you ship

7:36 that? Either something clever chooses which keys survive or this model is worse at long context than its benchmarks say. Something has to pick which entries each query actually reads.

7:48 That job belongs to a small side attention of 32 heads which scores the candidates and keeps the top 512. The paper calls it the indexer. And here's the problem it creates. Scoring every

8:00 visible entry is linear in context length. And it happens again at every layer that indexes. Fix attention and the chooser becomes the bottleneck. So how it works in the decoder? The first

8:11 full mode layer layer 20 scans the whole visible context once. Then a second cheaper pass over the same scores in blocks of eight positions keeping each blocks maximum. From there, it keeps its

8:24 top 24,048 blocks. At eight positions a block, that's a pool of just over 16,000 candidate positions, and the pool gets handed down the stack. After that, every deeper indexing layer scores only those

8:38 16,000, not the million. So, each deeper indexer's cost stops growing with context length and goes flat. And it's trained that way, not bolted on. The same restriction applies during

8:50 post-training and at inference. Put a query at the far end of a million token context. Layer 20 scores every visible entry once, then hands down its pool. The indexing layers above it each score

9:03 16,000 instead, 61 times smaller, four times over, which is where the catch lives. So, let's say it now. That pool is one layer's opinion. If the entry a deeper layer needed wasn't in layer 20's

9:16 top blocks, no layer below it can get that entry back. The bottleneck moved. It didn't vanish. Before we price any of this, there's one more thing they moved. Two modules sit at layers 1 and 14, and

9:29 they aren't matrix multiplies at all. Their lookup tables. Deep See calls the pair engram. How it works. For each token, the model hashes the last two tokens, the last three tokens, and the

9:42 last four tokens separately through eight hash heads each. Each of those 24 hashes indexes a table of roughly 16 million rows. The table sizes are picked as distinct

9:53 primes. So a collision in one head doesn't line up with a collision in another. Do the multiplication and the size stops being a mystery. All those tables across both modules come to 196

10:06 billion parameters. More than a third of the models weights and none of them multiply anything. But a token only ever touches 24 of those rows. So the parameters are enormous and the work per

10:18 token is a table lookup. Memorization gets decoupled from computation. And once the addressing is deterministic, the tables don't need to be on the card at all. They live in host memory. And

10:30 background transfers prefetch the rows the model is about to want, overlapping the very first block of the network. stored eight bits to a parameter, that lock comes to about 200 gigabytes,

10:41 sitting in ordinary system memory while the cards hold the backbone. And they are not the only lab that move memory off the card. Late in August, Alibaba shipped a Quen flash model with 51

10:54 billion engram parameters running in system memory instead of card memory. Two labs weeks apart, the same move. Watch which problem a lab spends its architecture budget on. Anthropic and

11:06 Open AI spend theirs on capability because capability is the product. Deepseek spent two generations on bytes per token because it sells cheap inference and then has to serve it. So

11:18 that's four mechanisms and every one of them is an approximation. Half a network reads your prompt. Bounded replay rebuilds Windows approximately. Most of the layers borrow keys they did not

11:29 score. The indexer throws most of the context away. Which brings us to the part the launch coverage skipped. Deepseek's own limitation section names two failure modes. Selection errors in

11:41 the sparse attention and approximate state reconstruction in bounded replay. Both of those in their words may still cause capability degradation in untested boundary cases. The deployment section

11:52 is blunter. The replayed prefix state is approximate. So the cache depends on where the cache hit landed. Same prompt, different cash state, slightly different numbers. That's their sentence, not

12:04 acitics. Now, the scoreboard properly because the wins here are real and narrow. On terminal bench 2.1, it leads Claude Opus 5 by a point and a half. On the software engineering benchmark by a

12:17 fifth of a point. Those are the winds, and they are narrow enough that a different harness could flip them. Then the other direction, which is not narrow at all. On terminal bench 4.0, zero. It

12:29 finishes 20 points behind Claude Opus 5 on humanity's last exam, 19 12 points behind. Those are the two biggest gaps on the whole board. And Deepseek says why parody on the averages, they write,

12:42 does not imply that the model matches the frontier capabilities of leading closed source systems. There's a cost no leaderboard shows either. Artificial Analysis measured it the day after

12:53 launch at around 200 tokens a second and called it very verbose. It writes nearly twice the output of the median model they test and a cost that isn't about the model at all from the 14th of

13:05 September. Request to the flagship route into this one instead at this one's prices and the flagship is being phased out. If you tuned prompts against the old one, you were not consulted. Simon

13:17 Willis's line about that on Hacker News was about trust rather than quality. Providers, he wrote, should lean towards not swapping out models on their paying customers. The answer that survives came

13:29 out of that same thread. The weights are on hugging face under an MIT license. A team that needs a frozen model can freeze it for as long as it likes, which is a choice nobody on a closed endpoint

13:41 has. and the older complaints about this family misbehaving encoding harnesses mostly traced to third-party gateways serving lower quality quantisations with open weights. The same model served by

13:53 two hosts is two different products. So here's the call. This model wins the access that costs money and loses the access that needs expert knowledge. If your agent rereads a long context all

14:05 day, it's the pick and it isn't close. If your hard problems are science-shaped, pay anthropic. Three receipts. The first is that software engineering benchmark where it lands a

14:16 fifth of a point ahead of Claude Opus 5 on a third party harness for 76 times less money on the workload we started with. The second receipt is the one I trust most because it isn't a benchmark

14:28 at all. 890 bytes a token, down from four times that a few months earlier. You can check it against the config file they shipped. The third is what the company did rather than what it said. It

14:41 routed its own flagship into the cheap tier at cheap tier prices and put it on the way out. A lab that retires its flagship in favor of the cheap one believes the cheap one. The suspicion

14:53 here belongs on an axis, not on a company. Leaderboards rank models by score and print price in a footnote, which is how a fifth of a point for a 76th of the money reads as a tie.

15:05 Deepseek's own paper has the sharpest line against that standard evaluation infrastructure becomes more susceptible to model gaming. One bet before we go, so this is falsifiable. Deepseek says a

15:17 bigger version of this architecture is coming. My call is that it ships by the end of March 2027 and it does not close that terminal bench gap. The cache is down to 890 bytes a token and four

15:30 generations of architecture got it there. So here's the one I can't call. Is there a fifth compression still in there?