The Secret GPU Kernel Behind DeepSeek V4.1 Flash

0:00 890 bytes. That's everything DeepSk's newest model keeps to remember one token of your prompt. Their first model kept 389,000. Same company, same job, under 3 years

0:13 apart, 437 times smaller per token, and it shipped this week. If you rent a long context window, that's the line item you see only as a total. And it sets your bill every time you send a prompt. The

0:27 weights are open and so is the kernel doing the work underneath them. The win here isn't what this model remembers. It's what it refuses to look at. By the end, you'll know exactly where that

0:38 number comes from. And who says the idea was theirs first. Every token you send leaves keys and values behind. Think of them as notes on the desk. So the next token doesn't have to reread the whole

0:50 book. That pile of notes is the cash. Normally, every layer keeps its own copy. So your bill is layers times tokens. Three years back that bill ran 389 kilobytes for every single token.

1:04 Then version 3.2 cut it eightfold and the first flash cut it almost 14 times again. So that's the number and it's already the story. Now let's get to the machine underneath it. Open the

1:16 reference code Deepseek published alongside the model and we can watch the whole trick happen in one line. It scores every position you've sent. Then it calls a function named top K to keep

1:26 the best few hundred. Be precise about what that kernel is, though. It isn't where the bytes get saved. It's what makes saving them affordable. Because the design that saves them runs this

1:37 selection for every token it writes. So here's the shape of the thing. The model keeps your entire context in memory, right down to your last token. It simply refuses to read most of it. And that

1:48 refusal is the product. A small side network called the indexer does the choosing. Think of it as a scout running ahead of the expensive attention. It scores every position against the token

2:00 being written, then points at the handful worth reading. Then it keeps 512 of them. That isn't a rule of thumb either. The model's own config file carries a line reading index top k 512

2:13 and that line is the whole policy. At a million tokens, 512 positions is 5 hundreds of 1%. The rest is still sitting in memory, and you pay for every bite of it, unread. Eight of the 40

2:28 layers run that selection. Each one picks its own 512 using its own query. So, the layers really do disagree about which parts of your prompt matter. That disagreement is what makes it work,

2:40 which means the model leans on one unglamorous operation. Find the biggest 512 numbers in a list of a million over and over for every token it writes. So, how fast can that actually go? Start

2:53 with what top ks. It does no floating point math at all. It compares numbers and it moves them. So, the only ceiling it can hit is memory bandwidth. Pytorch's built-in version doesn't

3:05 behave that way. On this shape, it sits flat at roughly a quarter of a terabyte a second, whether you hand it 16,000 scores or a million. Flat is the tell. It's leaving the memory idle. Deep

3:18 select starts somewhere else entirely. It holds one number in the fast memory on the chip, a threshold set to minus infinity. Beside it sits a small tray for candidates. That is the entire

3:30 state. From there, it chops your row into blocks of 1,024. Then it shuffles the order it will visit them in. Hold on to that shuffle because it turns out to be the clever part. Then

3:42 every one of your elements gets exactly one question bigger than the line into the tray. Smaller, dropped, and gone for good. One comparison running at memory speed. Once the tray fills past its

3:55 mark, a real sort finally runs, but only on the tray, never on your data. It keeps the best 512, then raises the line to the smallest of those, and the line only ever goes up. So, every block after

4:08 that faces a harder bar than the block before it. The tray fills more slowly, and the sorts get rarer the further in you go. Meanwhile, every one of your million scores was read exactly once in

4:20 order in contiguous blocks. That's why the benchmark reports bandwidth instead of operations. There's no arithmetic in there worth counting. So here's the question worth stopping on. Out of a

4:32 million numbers, how many do you think ever reach a sorting step at all? Pick a number in your head before I tell you. So say you send one query at a million token context and you want the top 512.

4:45 Their own proof puts a ceiling on how much of that ever gets sorted and the bound is on screen. 1 and a half times the buffer size times the harmonic number of the block count. Work it

4:55 through and you get 29,000. Under 3% of your data ever reaches a sort. The other 97% meets one comparison and disappears. Now the shuffle pays off because the visiting order is drawn without ever

5:10 looking at your data. No input can be built to make it slow. That's what turns a good average into a guarantee. The rest is instruction level craft. One pass builds a 32-bit mask, one bit per

5:22 element, so the right can walk it directly instead of counting first. And some integer ads get done in floating point because the integer units are already busy comparing. There's a second

5:33 path for when the batch is too small to fill the chip. One row gets split across several processors. Each takes a local top K, then ships its winners to a single leader that merges them. It has

5:45 hard edges, too. It compiles for two Blackwell targets and nothing else with the older ampier and hopper lines commented out in the build file. Top K above 4,000 is unsupported and one bad

5:57 value aborts the whole kernel. Five commits, 9 hours, two names on the citation block. So that's the kernel. Now for what it buys you, the model is 40 layers and they aren't all doing the

6:09 same job. 20 of them form a causal encoder that reads your prompt and 20 form a decoder that writes the answer back. It's a big model on disk and a small one at runtime. Only 8 billion of

6:21 its parameters fire while it reads you, 16 billion while it writes. Each layer gets one of three jobs, fixed when the model is built. A full layer computes its own cache and picks fresh. A reindex

6:35 layer borrows that cache but rescores it. A reuse layer borrows the pix 2 and just attends. Four of the 40 keep a global cache. That isn't a design principle you have to take on faith.

6:46 It's a list sitting in the config file. Layers 2, 8, 14, and 20. Every other layer keeps nothing global at all. 36 of them read the keys and values those four already stored, which is how the bill

6:59 stops being layers times tokens. Three of the four owners sit in the encoder where two tokens fold into a single cached entry. So across 40 layers, the model stores two and a half entries per

7:10 token instead of 40. Sharing the cache saves storage. It doesn't save the scoring though, and that's a separate problem. Any layer that wants its own picks still has to score your whole

7:22 context to get them. So layer 20 does the one full sweep. It scores every reachable position and keeps its own 512. And while it has all those scores in hand, it keeps something extra on the

7:34 way past. It groups positions into blocks of eight and gives each block the best score any position inside it managed. Then it keeps the 2,00 blocks. The newest tokens are pinned in. So

7:46 recency can't be outvoted by an old block. That's 16,000 candidates under 2% of your context. And every indexer above layer 20 scores only inside that pool. From there, the cost of each one stops

8:00 growing with context length at all. which is why the colonel's own test file sweeps top 512, top 1,000, and a vocabulary of 129,000. That benchmark is this model's config

8:14 written down. Then precision finishes the job. The main cache is squeezed into four bit numbers with one shared scale for every 16 channels, about 4 and 1/2 bits each. They can drop the safety

8:26 scale on top of that because those values can't get big enough to need it. After normalization, the largest they ever saw in training was around 10 against a format that reaches nearly

8:37 3,000. The sliding window cache stays at 8 bits. That one's sensitive to rounding. And the report says so out loud instead of shipping it in silence. And that window stopped being written to

8:48 storage at all, which is the other half of the story, an eighth of the disc the previous flash needed. So let's add it up. The published total is 890 bytes a token against 3.5,000 for the previous

9:02 flash and 389,000 for their first model. Hold a full million token context and that's 890 megabytes, about the size of a long podcast. The same conversation on that

9:14 first model would have been 389 GB. That's more high bandwidth memory than an 8 chip node has in total for one conversation. So, we're two things in. It's smaller and it isn't obviously

9:28 worse. Here's the part that isn't a claim at all. Because a price list is a company telling on itself. On Deep Seek's own page, a cash input token on Flash cost you 3/10en of a cent per

9:40 million. You pay 50 times more when it misses. So, everything you've just watched exists to make the cheap path the likely one and to make the prefix you keep sending nearly free. Against

9:51 their big model, you're paying 15 cents a million instead of 66, five times the concurrency limit and vision support that the expensive model doesn't have at all. On Terminal Bench 2.1, it comes out

10:03 ahead of Opus 5 and GPT 5.6. It loses Terminal Bench 4.0 by 20 points, and DeepSeek publishes that loss themselves, first in their own report. Artificial analysis measured it from the outside at

10:18 just over 200 tokens a second, but it spent nearly twice the median number of output tokens getting its score. Cheap per token and talkative. Then something more interesting happened. The older

10:30 flash was retired into this model. The big one was supposed to follow on the 14th. Their docs now say it stays in response to user demand. 24 hours and 17 minutes after that release commit.

10:44 Somebody opened issue 13 on the repository. The title asks for a citation, not money, not a takedown, a citation. So, let's put the dates side by side. Back on the 13th of July, a

10:57 paper called light top K went public as a preprint. Five authors, three universities, and a kernel built for exactly this job. Its abstract describes sampling a small subset to estimate a

11:09 threshold, then filtering online and writing back only the promising candidates while still returning the exact answer. 2 days later, that was a pull request in the VLM serving engine.

11:21 A week after that, the same idea landed in SG Lang, credited to its author by name. Two more projects picked it up before September was a week old. They also say they wrote to Deepseek about it

11:34 back in August. There's no public record of that either way, so take it as their account of events, not as an established fact. 59 days after that preprint, Deep Select shipped. Its design document says

11:47 Deepseek designed a new top K algorithm, and it cites nobody at all. So, I went and read the kernel myself. The published pseudo code starts the threshold at minus infinity, exactly the

11:59 way the write up describes it. The shipped code doesn't. It loads a 32 kilobyte window first, takes an initial top K out of that and begins the real scan already holding a working

12:10 threshold. Now the other side because we should take it seriously. Threshold filtering is old. The document says its own pseudo code is simplified. The proof, the

12:21 instruction level work and the multi-chip path are deep selects own and the claimants themselves say the two implementations start differently. There's a generous reading here as well.

12:32 Two teams chasing the same bottleneck with the same hardware manual open will land near each other. That's what a shared constraint does to independent engineers. The attention isn't symmetric

12:43 though. 18 stars in 14 weeks for the papers repository against nearly 300 for deep select in a single day. And people really are reading this code. Inside the first day, one outsider filed four bugs

12:57 and four fixes, including an integer overflow in the index offsets and vector writes running past the end of the tensor. The issue is still open with one participant on it and no reply from

13:10 anybody at Deepseek. The engineering itself isn't in question. An outside team took the riskiest approximation in the whole design, ran a maths benchmark with it switched off and then on, and

13:22 scored 453 out of 480 both times. So, here's where I land. If you run agents on long context, and that's most agent work. Now, this one wins, and it isn't close. The gap at the top end is real,

13:38 and it might be where your work lives. On the hardest science work, it's 20 points behind Opus 5. Deepseek says so first in their own report, and their customers push back hard enough to keep

13:49 the big model alive. I take it at twice the price because a price is answerable by next Tuesday and 890 bytes isn't. A lab that got very good at deciding what to throw away threw away one line. So

14:02 here's my bet, and you can hold me to it. If that design document still has no prior work section by the end of October, the omission was a choice and not an oversight. An open license with

14:14 no citation against a free paper whose own repository has no license file at all. Which of those two did more for the people who actually have to run these models this year?