Qwen 3.8 27B in 8.4 GB. But Don't Download That One

0:00 Two files, same model, same 8.4 gigabytes on disk. Both load in llama.cpp, no patch, and no custom kernel. They're both plain GGUF, the format your tools

0:13 already read. On a three benchmark average, one scores 86, its twin scores 78, 7.7 points apart from two downloads you'd swear were the same file. The model is the new Qwen 3.8 27 billion

0:29 parameters out in August. At full precision, that's 54 gigabytes, and it reads images and video, not just text. Shrunk to 8.4, it fits on a 16 gigabyte gaming card with room for context. So,

0:44 compression works. What's strange is that two people hit the same size, and one did it far better. By the end, you'll know what's inside those two files, why they differ, and which one to

0:55 download. Start with the rounding. Quantization means every weight in the model gets stored in fewer bits than it deserves, and that's the whole idea. Think of a photo squeezed down for the

1:06 web. You throw away detail, and hope the eye skips past it. Same trade here, except what you're throwing away is the model's judgment. At four bits per weight, that's fine. Below four, it

1:18 stops being fine, and it stops suddenly. Researchers call it the error wall, and the simple formats have been stuck behind it. So, the field split. One camp kept the simple format, and ate the

1:30 accuracy loss. The other built clever formats that pack weights in groups, and won the accuracy back. Those clever formats have a problem, though. They're so fiddly to decode that the memory you

1:42 save never turns into speed. You get a much smaller file that runs barely faster. Dan Alister, our group in Austria, asked a different question. These are the people behind GPTQ, still

1:54 one of the two most deployed rounding methods. What if the format was never the problem and we were just choosing badly inside it? Rounding is a choice and a choice has no gradient. It's a

2:05 staircase, not a ramp. You round up or you round down and there's nothing to differentiate, which is why every method before this one optimized something else instead. So, they made the choice soft.

2:17 Take one real weight. At two bits, it has four allowed values, so give it a score for each one. A random draw turns those scores into a blend and the weight floats between values instead of landing

2:29 on one. Now, you can train it. First, run the layer and measure how far the output drifted. Then, push those scores downhill and cool the whole thing. A

2:39 temperature setting falls toward zero, the blend sharpens and your weight snaps onto a single value. Nothing about the format changed. Same values, same scales, same file layout. All that moved

2:51 is which value each weight landed on, usually by a single step. Multiply one step by 27 billion weights and you have most of that gap. That fixes one tensor, which is just one weight matrix. Our

3:04 model has 851 of them. 10 precisions are on the menu, any tensor can take any of them and the finished file still has to land on a size you picked in advance. Two things so far.

3:16 The rounding inside each tensor got better and now something has to decide how many bits each tensor deserves. Normally, you score every tensor on its own and solve a knapsack. It's fast and

3:28 it breaks when you squeeze hard because layers interact and a score computed in isolation can't see that. A second paper goes geometric. Write each tensor's choice as soft

3:38 probabilities and the budget stops being a rule you fight. Say you set it at two and a half bits a weight. That's now a surface and every legal allocation is already standing on it. After that, you

3:50 just walk on it like following a ridge line. Flatten the gradient so it can't push you off. Take one ordinary step, then let a bisection search pull you back on. At that point, the file size

4:02 isn't approximately two and a half bits, it's exactly that at every step. On a smaller model of theirs, that search finished in 117 minutes. The evolutionary method it replaced needed

4:15 11 to 14 hours and still scored slightly worse, which is a lovely theory. So, before we look at what it chose, answer this one yourself. If you had to hand extra bits to one

4:26 part of a network, which part would you protect? Almost everyone says the first layer and the last. Until the end of August, you couldn't check because the lab hadn't published the assignment.

4:37 Then a user posted their own benchmarks in the discussion tab and asked for it. The lab uploaded it the same day. What arrived is a plain text file, one line per tensor, each with the exact

4:49 precision it got. 402 of them were picked by the search, spanning 10 different precisions inside one file. Look at what it did with the token embedding table, which is just the

5:00 lookup that turns your words into vectors. It got the second lowest precision in the whole file. The tool that builds every one of these files already has a rule for this exact

5:10 case. Inside the code, at this compression, llama.cpp puts the embedding on a two-bit type and the output head on a five-bit one. You can override that by hand and good repos do.

5:24 Gradient descent went lower on both and much lower on the embedding. So, where did those bits go instead? Let's split the network into quarters and look. In the first quarter, 39 of 100 weight

5:36 tensors sit on a one-bit type. In the last quarter, exactly one does. So, it half agreed with you. It protected the end of the network hard and it flatly refused to protect the beginning. No

5:48 engineer wrote that rule, and none could have. It fell out of running the real model and watching the loss move. So, that's how the file gets built. Now, the part that download page won't tell you.

6:00 The 8.4 GB file is the headline, and it isn't the one to run. Against full precision, it gives up 5.8 points, almost all of it on code. Spend another 800 MB on the 9.3 GB build, and the gap

6:15 drops to 2.3 points. The competition math score goes back to a perfect 100. Benjamin Marie, an independent researcher who put 950 prompts through 15 of these files, landed in the same

6:27 place. Probably the best you can get at that size, and skip the smallest one if you're writing code. One caution on the numbers, including mine. On the easy five task suite, these files score above

6:38 the model they came from. That doesn't mean compression improved anything. It means the suite stopped telling them apart. Weights are only half the bill. Just 16

6:48 of the 64 blocks run real attention, so the context cache here is unusually cheap. At 128,000 tokens, it still wants about 8 GB on top. So, here's my verdict. The 9.3 GB

7:01 build wins. For anyone with a 12 or 16 GB card who wants a 27 billion parameter model that also sees. The concession is real. Unsloth's repo has 95 times the downloads and a file for every card ever

7:15 made. This lab shipped four of them in late August, and vision costs another gigabyte on whichever one you pick. I'd still take it because the villain here isn't a company. It's a constant sitting

7:27 where a measurement belongs. Somebody reasonable decided years ago that the embedding and the output head deserve extra bits, and everything downstream inherited that call without re-deriving

7:39 it. One search on one model says that call is wrong about this model, which leaves a question I can't answer for you. Every quantized file already sitting on your drive had its bit width

7:50 chosen by hand. How many points are they leaving on the table for free?