Claude Just Shrank Qwen 6x. Then Something Weird Happened

0:00 If you've squeezed a 27 billion parameter model onto a small graphics card, you probably downloaded a specialist file packed to about two and a half bits per weight.

0:09 A paper submitted on November 4th says Claude Sonnet 5.5 working alone for 8 hours built a better scoring one in almost the same space. So the file on your drive may be second best on

0:20 quality. The 27B Qwen 3.8 is the model and in the paper's appendix table Sonnet's version beats the expert made public Qwen on perplexity where lower is better and on accuracy. Both files fit

0:33 in 8 gigabytes of weights. A third number in the same table points the other way since Sonnet's version decodes 23.4 tokens a second to the expert file's 75.5.

0:44 Divide one by the other and the winner runs at about a third of the speed and what the paper says about why is its most useful part. The GGUF guide on the BMD Pat blog gives the usual advice and

0:56 it's sensible. Start with a standard Qwen like Q4KM when memory's tight, test it on your real task and keep the smallest file that passes. Behind that advice sits the assumption

1:07 that people make the files worth testing and Compression Bench by Jong and colleagues with Baidi Chen as last author checks whether agents can do it alone.

1:16 By the end you'll know how the test worked, why the winner is slow, and whether it changes the file you pick. Six agents took part, Claude Sonnet and Opus 5.5 plus GLM, Kimi, MiniMax, and

1:28 DeepSeek from other labs. All six drove Cloud Code at maximum reasoning effort. Each got 1H 200 NVL, a data center card with 141 gigabytes of memory plus an 8-hour clock. They also ran version 0.32

1:43 of the VLLM inference engine. Everything else follows from the size limit because the starting point is huge. Qwen 3.8 ships in BF16 which is 16 bits per weight, so 27 billion weights

1:55 at 2 bytes each works out to about 54 gigabytes. At the tight setting budget is eight gigabytes, which Ista-Dash Labs expert file fills at two and a half bits per

2:05 weight. Each weight keeps call it a sixth of the bits it started with. Beyond the file itself, the limit counts the weight sitting in GPU memory, including the

2:14 scales and codebooks a compressed format needs. Any full-size copy you unpack and keep counts as well. As a model reads your prompt and writes its answer, it fills memory called the KV cache, and

2:25 the limit leaves that out. So eight gigabytes here is a weight budget, which doesn't mean the model fits an eight gigabyte card. BMD Pads Guide warns about this too,

2:35 saying file size is only the start of the memory bill. You'd expect an agent to run an existing quantizer and hand back whatever came out, but the task asked for more. Each

2:45 one worked offline with read-only weights, starter code, papers, and calibration data, and it could mix any of the five compression families the paper lists from quantization to

2:55 pruning. In return, it had to write the code that runs whatever format it invented. While an agent searched, the only score it saw was a perplexity ratio, a measure of how much more

3:05 surprised the compressed model is than the full one by text that the full model wrote. Every check came out of the same eight hours, so an agent that asked for its

3:13 score after every tweak would spend a good part of its day being graded instead of getting better. Then came a gate they never saw set before the search started and made of

3:22 150 held-out questions. 50 were grade school math problems from GSM8K. 50 were coding tasks from HumanEval, and 50 were trivia questions from TriviaQA. To pass, the compressed model had to

3:35 reach 80% of the full model's accuracy. Across three source models with two size limits each, 29 of 36 submissions passed, including all 18 at the looser limits. Besides the dense Quinn at eight

3:48 and 12 gigabytes, the set had two larger models from Quinn and Nemetron at 32 and 48. Both are mixture of experts designs which use only a slice of their weights for each token. For the headline, go to

4:00 the tight setting where lower perplexity is better. Uncompressed, the model scores 3.22 and gets 74% right. Sonnet's version scores 3.36 and 68.67%. Then the expert G S Q R C O file scores

4:15 3.76 and 65.33. Its package is also a little smaller, 7.86 GB to Sonnet's 7.96. Opus came in just behind Sonnet at 66% and DeepSeek cleared the gate at 61.33

4:32 since 80% of 74 comes to about 59. G L M, Kimi and MiniMax missed at this size. Plain G P T Q, the standard method that gives every weight its own short code, scored 4.67%,

4:45 a very compact way to store a model that stopped answering questions. Give the agents four more GB and the gap widens with the public release at 66%. Opus reaches 78.67,

4:57 Sonnet 76.67 and G L M 80.67. Hold those scores loosely because with 150 questions in total, one question moves the overall score by about two-thirds of a point. Count it out and Sonnet's lead

5:10 over the expert file at 8 GB is five questions. G L M's 80.67 at 12 also sits above the full model 74 and that's most likely the wobble of a small test set. Pooling all six tasks gives a steadier

5:25 number and picking the best agent for each one left perplexity 3.4% above the full models. Accuracy landed at 77.33% against 79.56.

5:36 Five public releases were in the comparison and the best agent model B each one. How they got there is the part worth borrowing because in 21 of 24 revisions

5:46 that barely changed the size, agents lowered perplexity by improving the quantizer or moving precision around. Kimi, for example, went from 3.74 to 3.13 by adjusting only two quantizer

5:58 settings, clipping and damping. Local winds didn't always survive though because testing the complete model reversed 17 of 79 local rankings. Opus spent 1.7 hours exploring a one-bit tier

6:11 for the expert subnetworks in a mixture of experts model. Its local tests liked the design, but the full model check preferred the old one, so agents can lose an afternoon to a benchmark too.

6:22 Perplexity misled them as well, since small changes in it flipped some answers to right and others to wrong. At 8 GB, the GLM and MiniMax models that failed had kept pruning designs which cut parts

6:34 of the model out entirely, and they wrote repeated or cut off human eval code. Their own local generation tests had already flagged it. Section 4.4 of the paper spells out what

6:43 Sonnet built, and the first thing it did was replace that standard method. Instead of plain GPT Q, it rotated the weights with a had a mar transform and packed them with trellis codes chosen by

6:55 a Viterbi search. Sonnet added a correction step the paper calls error compensation and gave more bits to the parts where errors hurt most. Sonnet's exact bit split in trellis

7:05 settings aren't published, so the steps that follow describe how methods in this family usually work from Q tip and earlier paper on them. First comes a random had a mar transform which mixes a

7:15 weight matrix's values together so the few far bigger than the rest get spread out. Afterwards, the weights look roughly like a bell curve and Q tip reports no outliers left, which matters

7:25 because a code tuned for one predictable shape then fits every matrix. Packing comes second and a trellis code stores a long run of weights as one path through a grid of allowed values. Picture

7:36 describing a walk by its turns instead of listing every footstep, so each weight costs only the few bits that pick the next turn. Where the comparison breaks is accuracy

7:45 because a walk retold by its turns comes back exact while the path only lands close to each real weight. Choosing that path is the third step and the Viterbi algorithm finds the route

7:56 through the grid that lands closest to the real weights. Its running time grows only in step with the length of the run, which keeps the search practical. Now back to the 23

8:05 tokens a second, the number that pointed the other way. When the model runs, Sonnet's format rebuilds weight tiles from the stored trellis codes as each matrix multiplication needs them. So

8:15 every new token rebuilds those tiles all over again. Keeping an unpacked copy around won't dodge it because a full-size copy kept in memory counts against the 8 GB bytes. At 12 GB bytes,

8:26 the paper's text gives an even wider gap, 16.7 tokens a second against 74.6 measured one request at a time. Section 4.4's lesson is that a format and its kernels, the GPU code that does the math

8:40 on it, have to be designed together. Nowwhere does the paper break down where Sonnet's decode time goes, so the fix isn't known. Over in Q-Tip, the authors report a decode-friendly trellis

8:51 reaching over 80% of peak memory bandwidth one request at a time. Trellis formats then can probably run fast when the kernel is built for them. Back on your own machine, remember that

9:01 every run used an H 200 NVL with 4.8 terabytes a second of memory bandwidth for 8 hours. None of these speeds come from a consumer card, and the 8 GB byte budget also leaves out the cache your

9:13 context needs. There's also nothing to install yet because the compression bench repository ships the harness and the results CSV files with no agent-made checkpoints.

9:23 For now, the best 8 GB byte quen in the paper exists for you as a row in a spreadsheet. If you run local models, the better pick today is still a public expert file like

9:33 GS QRC Os two and a half bit IQ 2 XS because you can download it. In the paper's H 200 test, it also decoded 75.5 tokens a second to Sonnet's 23.4 on hardware you probably don't

9:46 have. Quantization tool builders should read this paper closely since in its tests, agents beat human-made quants on quality per byte and haven't matched them on speed. Your file from the start

9:57 of this video still earns its place. Whatever quant you end up with, judge it on quality, speed, and memory together on your own task. In this paper, the best score was also the slower of the

10:07 two by a wide margin. Would you trust an agent to quantize the model you run every day if it scored a little better and ran at about a third of the speed?