Enjoying this issue?
Get tomorrow's AI & engineering digest in your inbox — hand-picked, summarized, and always spam-free.
TLDR
Standard VRAM calculators systematically overestimate cache memory for modern open-weight models because they assume all layers use full attention. For models like Qwen 3.827B, Gemma 4 12B, and GPT OSS 120B, three architectural trends — gated delta nets, sliding windows, and very small key-value heads — cut actual cache requirements by factors of 4 to 6, especially at long contexts. The config file's `layer_types` field is the single most reliable guide, and cache quantization to Q8 saves memory with negligible accuracy loss, while Q4 on keys can cause catastrophic failures that perplexity metrics miss.
Key points
- Standard VRAM calculators assume every layer keeps a per-token cache, leading to a 4x overestimate for models like Qwen 3.827B, where 48 of 64 layers use gated delta net blocks (fixed state, not growing cache).
- For Qwen 3.827B at 131,000 tokens: real cache is ~62.5 MB per token (64 KB × 16 full-attention layers × 4 KV heads × 256 head dim); formula predicts 256 KB per token (64 layers × 4 KV heads × 256 head dim).
- Gemma 4 12B uses 5 sliding-window layers (window 1,024) per full-attention layer, so at 128K tokens real cache is ~8 GB vs. formula’s 48 GB — error grows with context length.
- GPT OSS 120B alternates sliding (window 128) and full attention layers, with head dimension 64, yielding 4.5 GB cache for 131K tokens vs. formula’s ~50+ GB.
- Quantizing GPT OSS 120B saves almost no weight memory: weights ship as MXFP4 in most layers, so the weight file changes by only ~1.3% from 2-bit to 8-bit.
- Cache quantization to Q8 saves 47% memory with negligible accuracy: in 4000+ comparisons, at most 4/500 answers changed; Q4 saves 72% but caused one older model to flip 375/500 (keys side).
- Q4 cache quantization incurs a speed penalty: 3% at 8K tokens, 35% at 64K tokens, making Q8 the recommended default.
- AutoGPTQ and AutoAWQ are archived (last commits April/May 2025); maintained path is GPTQ-Model and llama.cpp v3 (but AMD cards lack RockM and NVFP4 support).
Tools mentioned
Techniques
- gated delta net cache (fixed-size state overwritten per token)
- sliding window attention (cache size bounded by window length)
- key-value head dimension reduction (e.g., 64 vs. 256)
- MXFP4 weight format (most weights already 4-bit before quantization)
- cache quantization at Q8 and Q4 levels
Stop scrolling. Start reading smarter.
Receive the day's most important AI & engineering updates in one concise email. No spam.
Transcript (captions)
Type a model name into a GPU memory calculator and it gives you a number. That number decides what you download. Ask it about a 27 billion parameter model at full context and it says the
cache alone wants 64 GB. It wants 16. Off by a factor of four. Using the formula that every calculator on the internet runs. That is not one broken tool. Every calculator you have pasted a
model name into computes the same thing the same way. They all assume every layer in the model keeps a cache. For most models you would download this year, that is false. So the number on
your screen is not describing your model. It is describing a transformer from 2023. Modern open weights escape that formula in three separate ways, and each one changes what your card can
hold, which means the practical version is short. Your card runs a bigger model at a longer context than you were told. Three escapes, one flat line that should stop you quantizing a model, and one
saving that looks free and is not. Every figure comes out of the model's own config files, so you can open them in a browser and check me while I talk. Start with the formula because the formula is
the correct part of this story. Nothing in it is actually wrong. Cache size equals 2 * layers * key value heads time head dimension time bytes per value time tokens. Two because you store a key and
a value. Head dimension time key value heads because that is the width of what one layer has to remember. and layers because every layer remembers that last word is where it breaks. This is the
actual code from one of the calculators people paste model names into hosted on hugging face. Four lines of arithmetic and it multiplies the width of one layer by numbum hidden layers times context.
Num hidden layers all of them. That field is the model total depth, not its attention depth. And until about a year ago, those were the same number. So the tools had no reason to tell them apart.
The entire calculator is four files. Search every line and you will not find sliding window or full attention interval or layer types anywhere in it. And layer types is the field that has
been sitting in these config files the whole time naming exactly which layers keep a cache and which ones keep nothing. So open one Quinn 3.827B released this month. 64 layers and the
file lists all 64 by name. The pattern reads linear attention linear attention linear attention full attention. Then it repeats 16 times over. Which means 48 of those 64 layers never run the kind of
attention that keeps a cache. They run a gated delta net block. And that block holds one fixed size state that gets overwritten as you talk instead of a list that grows. If 3/4 of the layers
keep nothing, what is holding your conversation together? The delta rule. Each new token corrects the state rather than appending to it. The way you update a running total instead of keeping every
receipt. It costs memory, but the cost is fixed. It does not grow when the chat does. So, the 16 real attention layers are the only ones paying per token. Four key value heads. Head dimension 256, a
key and a value, two bytes a piece, 4 kilobytes per layer per token. 16 layers of that is 64 kilobytes a token. Take a,000 tokens of conversation and the whole cache is 62 1.5 megabytes. The
formula every calculator runs says 250. Same model, same context, same config file four times over and the gap is one field the tool does not read. Scale it to the full context this model ships
with and it is 16 GB of cache against 64 on a 24 GB card. That is the difference between fits and does not. That is the most extreme escape. But the second one turns up in a model far more people are
actually running. Gemma 4, the 12B, 48 layers, and the config lists five sliding attention layers, then one full attention layer eight times through. A sliding layer only ever remembers the
last thousand tokens. The window is written in the file sliding window 1,024. Push past it and the oldest token drops out of that layer completely. So at
8,000 tokens of context, the eight global layers hold all 8,000 and the 40 local ones hold about a,000 each. Add it up and the cache is 832 megabytes. The formula says 3 GB 3.69 times over and
that ratio is exact because both sides come out of the same four numbers in the same file. And this is the part that inverts what people assume. The error does not stay at 3 1/2. Ask for 128,000
tokens and the real cache is 8 and a bit gigabytes while the formula screams 48 nearly six times. The longer the context you ask for, the more wrong the number in front of you gets, which is backwards
from how anyone expects an estimate to fail. Escape 3 produces the figure I still find hard to look at. GPT OSS 12B from OpenAI. 36 layers alternating perfectly. Sliding full, sliding, full.
Half the layers are windowed. And this window is not a,000 tokens. It is 128. Then the model does something else on top. Head dimension 64 against Gemma 256. Eight key value heads, two
kilobytes per layer per token, a quarter of what Gemma pays for the same job. Put those together and the entire 131,000 token context on 120 billion parameter model costs 4 1/2 GB of cash. A 7
billion parameter llama back in 2023 burned two gigabytes of cache to hold 4,000 tokens. This is 120 billion parameters holding 131,000 for four and a half. So why is anyone still picking a
download from a formula that stopped describing real models? Watch what that buys using file sizes read out of the repositories in bytes not rounded off somebody blog post. 8 GB of card Quinn
3.5 4B at 8bit is 4 and a half gigabytes of weights. Full 8bit precision on a card people apologize for owning. 16 GB. Gemma 4, the 12B also at 8 bit, 12.67 GB, not a 4-bit compromise to make it
fit. The real weights 24 is where it gets interesting. Quen 3.827B at 4bit is 16.46 GB, leaving about 7 1/2 GB once the runtime has taken its buffers. 7 and 12 GB of cache at 64
kilobytes a token is roughly 120,000 tokens of context on one consumer card from one download. The calculator handed the same card in the same model budgets 256 kilobytes a token and tells you
30,000 same hardware a quarter of the answer and 64 GB of memory runs GPT OSS 120B at 63.39 GB of weights which leaves about 5 GB. Its entire context needs four and a half tight but it lands which
surfaces something in that file listing that took a second read to believe the same model quantized down the entire ladder. 2 bit 62.57 GB 4bit 62.77 6bit 63.28
bit 63.39 from 2bit all the way to 8 bit the ladder moves 1 and a3%. The bars refuse to get shorter. Quantizing this model is a rounding error. The reason is one line
of its config. The weights already ship in MXFP4 and everything except attention. The router and the embeddings is 4bit before you touch it. So when you download the 4-bit build to save space,
you are re-encoding numbers that were already 4bit. 600 megabytes off 63 GB plus whatever the re-encoding cost you in quality. If a model still does not fit, there is a second lever and it is
the one most people reach for first. Quantize the cache itself. This one you can derive instead of benchmarking. In GGML, a block of 32 cache values at 16 bit takes 64 bytes. At Q8 it takes 34.
At Q4 18, 8 1/2 bits and 4 1/2 bits. So Q8 saves 47% of your cache and Q4 saves 72. You do not need a benchmark for that. It falls out of the file format. On the 24 GBTE box, Q8 takes you from
about 120,000 tokens to about 230,000, nearly the model full native window on one card. And measured with perplexity, the quality cost is nothing. Long context runs put the whole spread under
a hundredth of a point all the way down to 4bit, which is where a thread on the Llama CPP repository gets uncomfortable because a contributor went and measured the thing perplexity does not see. 500
reasoning questions, deterministic decoding, fixed seed, four models, one question. Against a full precision cache, how often does the chosen answer actually change? At Q8, at most four
answers in 500 moved, and on two of the four models, none moved at all. Across 1600 deep context comparisons, out to 41,000 tokens, zero flips. At Q4, three of the four models barely twitched. Two
answers, three answers, 34. The fourth was an older Quen, the 2.5 generation, and it changed 375 of 500. 92% accuracy down to 24. Below guessing, with the cache as the only thing that moved, the
same thread isolates it. Four bit on the keys alone reproduces the whole collapse. 4bit on the values alone changed one answer out of 500. Keys are the side you cannot cheapen. Speed is
not free either. A run across four context lengths measured 4-bit cache costing 3% at 8,000 tokens and 35% at 64,000. So the cache setting that saves the most memory takes the most speed
away exactly where you needed the memory. That is a trade, not an upgrade. And the person who measured it landed on Q8. The last thing worth checking is the advice wrapped around all of this
because it has aged the same way the formula did. Auto GPTQ the pip install in a thousand tutorials is archived. Last commit April of 2025. Auto AWQ archived one month after that. The
maintained path is GPTQ model and it is still taking commits. Xlama v2 is not archived. Whatever you have read, it is dormant. Its last commit landed in March while version 3 was still being pushed 5
and 1/2 months later. But if you run an AMD card, read version 3 redmi before you switch. Under what is missing on the to-do list, oneline RockM support and NVFP 4. The newest 4-bit format needs
Blackwell silicon. A 5090 or a B200 have the tensor cores for it. A 4090 does not and gets the memory saving only. Then there is the claim people repeat hardest that IQ quants require an importance
matrix. It is 2/3 wrong and Llama CPP own source settles it in one function. Six types refuse to build without one and they are the very low bit ones three bit and under. IQ4 IQ3S no requirement
at all. A comment in that same file says the K quants you actually download do not need it either. Go looking for the study that measures how much it helps. One model, one bit rate with and without
and you come back empty. The guides quote a range. Not one of them cites a run. So where do I land? The winner is not a calculator, a rule of thumb, or a post about what fits on a 24 gig card.
It is the config file. 4 kilobytes of plain text in your browser in 10 seconds. Open it, find layer types, count how many of them say full attention, and multiply by that number
alone. If you have 8 to 24 GB, that one habit is worth more than any tool you can install. The concession is real. run a fully dense model, every layer attending, and the old formula is exact.
Those calculators were built correctly for the world they were built in. The villain is not a company. It is advice that outlived its architecture. Nobody lied. A formula that was right in 2023
got copied into a 100 tools and the models moved out from under it while the number stayed on screen looking authoritative. three receipts, 62 1/2 against 250, a quant ladder that moves
1% from top to bottom, and a cash saving that dropped one model below random guessing, while Perplexity reported everything was fine. Which leaves the question I would actually ask about your
own setup. What else in it is a number that nobody has reerived since the thing it describes change shape?