They Changed One Equation. 2-Bit AI Got Smarter.

0:00 One number in a paper from October stopped me. When a model is squeezed block by block, the worst blocks error is often over 10,000 times the best blocks. I'd pictured two bit damage as

0:10 the same small error everywhere. If you're fitting a 27 billion parameter model onto an 8 GB GPU, that matters. So, the answer is no. At 2 bits, it isn't lossless. And part of the damage

0:22 looks like uneven tuning that the paper shows you can reduce. Say you pick a 2-bit Quinn 3.5 27B for that card. The weights alone work out to 27 billion* 2 bits divided by 8. That's about 6.75 GB.

0:37 Your card is nearly full before the model has said a single word. You've probably heard the rule. Inventive HQ's benchmark of GGUF files, a common local model format, calls the Q2K type

0:49 nominally 2bit, clearly degraded. Its advice is to use it only when nothing else fits. I wanted to know why because rounding error alone doesn't seem to explain it. In this paper though, two

1:01 bit weights on Quen 3.527B go from perplexity 37.51 to 22.20 with only the loss changed. Perplexity is how surprised the model is by real text. So lower is better. How much of

1:15 the gap can that one change close? By the end, you'll know why tuning gets uneven and what the loss swap changes. You'll also know whether a 2-bit 27B is worth running on 8 GB. Start with what

1:27 two bits means. Each weight in the original model takes 16 bits. At two bits, a weight can only take one of four values. Numbers get coarser and the error adds up as the signal moves

1:38 through the layers. The paper writes W2A16 for two bit weights with 16 bit activations, the numbers that flow between layers. On llama 2 7B at that setting, plane round to nearest, which

1:51 snaps every weight to the closest allowed value, scores a perplexity near 38,000. Full precision for that model is 5.47. GPTQ, a smarter rounding method, still

2:02 scores about 7,700. That's an improvement in the way a smaller fire is an improvement. So, the popular rule is fair for simple methods at two bits. Learning based methods try

2:12 harder. Omniquant, the method behind the paper's headline results, quantizes one transformer block at a time. A block is one repeating layer of the model. Before it moves on, it trains a few small extra

2:24 parameters. Two of those parameters matter here. A learnable clipping threshold cuts off extreme weight values. A learnable transformation takes the few extreme values in the

2:33 activations and shifts that difficulty into the weights. Both get tuned for the block in front of them. What is the tuning aiming at? Each quantized block's output should match the full precision

2:43 blocks output. Call the gap between the two the reconstruction loss. A small sample of real text measures that gap. I compared the new paper's Omniquant number with Omniquant's own paper. I

2:54 expected some drift, but 36.92 against 37.37 is close. Each block has its own loss, and they aren't similar. Across Llama 2, 7B, Llama 3.1, 8B, and 3 Quinn 3.5 models. The largest per block

3:10 loss often exceeds the smallest by more than 10,000 times. Output magnitudes generally grow with depth, the authors say. They add that errors from earlier quantized blocks accumulate into later

3:21 ones. Figure 1 is a 4-bit setup. Omniquant on Quen 3.59B at W4 A4. The paper's two-bit results are where the loss swap pays off most. A big loss means a block that's far off, so more

3:34 work for it is fine. But does a small loss mean a block needs less work? To see why that matters, follow how the optimizer learns. Learning runs on the gradient, a signal that says which way

3:45 to move and how hard. Usually, the loss is mean squared error, MSE. At the block's output, its gradient grows with the square root of the loss. That's equation five. Take two blocks. Block

3:57 A's loss is 10,000 times block B's. Take the square root and that works out to 100. So, block A gets a push roughly 100 times harder. The size of the gradient is that push. Block B's loss is smaller,

4:09 so its gradient is about 100 times smaller. It gets a weaker push even if the block still needs the work. The author's principle is that tuning strength shouldn't depend on loss scale.

4:19 Fix it by taking the square root of the loss itself. That gives RMSSE root mean squared error, which is equation six in the paper. In equation seven, the gradient size is 1 over the square root

4:30 of n, where n is the number of values in the block's output. The loss term is gone. RMSSE's gradient is MSE's gradient divided by twice the square root of the loss. And that division cancels the

4:41 square root MSE had. Every block gets the same push, whether its loss is huge or tiny. The paper's name for this is implicit gradient normalization. One switch sounds simple, but the paper

4:53 tests four versions. They differ in which axis the error is normalized over samples, channels, tokens, or elements. On Quen 3.59B at W2A16, Omniquant with MSE averages

5:06 45.44% accuracy. The sample and token versions reach 47.61 and 47.63. Element wise reaches 52.76

5:16 and channel-wise reaches 54.78. Element-wise RMSSE equals L1 loss. So, one of the four variants is an old loss wearing a new name. So, the swap hides a choice. The paper calls feature grouping

5:29 a real design dimension. Channel one on Quen 3.59B. The best variant varies by model family and the paper doesn't explain why channel wins here. On Quen 3.527B with

5:41 omniquant at 2-bit weights, MSE gives perplexity 37.51 and average accuracy 60.45%. RMSSE gives 22.20 and 70.96%. Those are the paper's headline numbers.

5:54 That accuracy averages six tests including PIQA, ARC, and BU Q. Full precision scores 6.90 and 77.10%. Subtract and the MSE accuracy gap is 16.65 points. RMSSE closes 10.51 of

6:10 them. Divide one by the other and that's about 63%. Six points remain. Perplexity says something similar. The gap from 37.51 down to 6.90 is about 30 points and 22.2 20 closes roughly half of it.

6:26 Call 22.20 about three times the original 6.90. Accuracy closes almost 2/3 and neither gets all the way there. Llama 2 7B looks rougher. Perplexity goes from 36.92

6:39 to 16.06 and accuracy goes from 43.98 to 49.97%. Full precision is 5.47 and 70.45%. Count what's left and the accuracy gap is about 20 points. Gains shrink as

6:54 precision rises. On Quen 3.527B at 4bit weights, perplexity goes from 7.17 to 7.09. At 3bit weights, it goes from 8.42 to 8.26. In the author's words, gains grow with more aggressive

7:10 quantization. RMSSE accuracy at four and three bits, 78.37 and 78.13 lands slightly above the full precision 77.10.

7:21 That's noise level, so don't read it as better than the original, making the file cost almost nothing extra. Table 4 reports quantization runtime from 1.19% lower to 2.82% higher and peak GPU

7:34 memory up 0.00%. Those costs apply to making the file. Stronger methods gain too by smaller margins. Flat quant at W4A4 on Quen 3.59B goes from perplexity 10.26 to 9.15

7:49 and accuracy from 72.23 to 74.19%. The paper's reasoning and coding results are for the 9B at 4bit weights. GSM 8K goes from 95.15 to 95.98. Math 500 from 94.20 to 94.80 80 and MBPP

8:09 from 57.14 to 62.96. Human eval slipped from 75.61 to 75.00 the one benchmark that voted against it. The authors state a limit under W2A16G 128 a two-bit setting with groups of 128

8:28 weights. RMSSE alone cannot fully recover severely degraded performance. The 27B headline run uses plain W2A16. So treat this as a warning, not a measurement of that run. I read the

8:40 abstract of a separate April paper that names two failure modes at two bits. In signal degradation, computation patterns survive, but cumulative error hurts precision. In computation collapse, key

8:52 components fail and the signal is destroyed in early layers. Training free repairs can address the first. The second needs structural changes. Nothing in that abstract mentions RMSSE, so it

9:02 can't say which failure the leftover gap is. My guess is that RMSSE helps with the first kind since it evens out tuning. A collapse component would need a different fix. Unsloth says its

9:12 dynamic 2.0 files give sensitive layers higher precision and tolerant layers lower, which is a separate idea from the paper's loss change. I went through the author's GitHub repository. It readme

9:24 says code and checkpoints are coming soon. Coming soon doesn't come with a download link. For now, you can read the results, but you can't reproduce them. Let's go back to your 8 GB card and the

9:34 2-bit file. The paper's headline result for the 27B is 70.96% against 77.10 for full precision. Your 6.75 GB is the minimum for the weights alone. So about 1.25 GB is left at most. Real files run

9:50 a bit bigger because they store scale values and keep some parts at higher precision. A 2-bit file you download today might not use this loss at all. The paper's 4-bit rows on the 27B sit

10:01 close to full precision, but 27 billion weights at 4 bits come to about 13.5 GB, too big for your card. My call for an 8 GB card is the 9B at 4 bits. Call it about 4.5 GB for 9 billion weights at 4

10:16 bits before extras. On the 9B at 4bit weights, the paper reports GSM 8K95.98 and math 594.80 80 reasoning scores that differ from the six test average. No score compares the 9B with the 27B at

10:32 two bits. So that call is my judgment alone. If you need the 27B skills and can accept the gap, wait for the checkpoints, then test them on your own task. The paper's perplexity and six

10:43 multiple choice tests don't show whether a two-bit model keeps long reasoning, coding, or agent skills. None of those numbers comes from an 8 GB GPU either. So two-bit quantization isn't lossless.

10:54 But the authors argue part of the damage came from tuning that followed loss size. Swap in RMSSE and each block gets an equal push. In the paper's 27B result, that recovers about 63% of the

11:06 accuracy gap at almost no cost to make the file. Perplexity is still about three times full precision at 2 bits. At 4 bits, it's 7.09 against 6.90. Closed, but not identical. What closes the last

11:20 six accuracy points and the three-fold perplexity gap? A better loss, mixed precision, and a change to the model structure are the three candidates I'd like to hear which one you'd bet