Enjoying this issue?
Get tomorrow's AI & engineering digest in your inbox — hand-picked, summarized, and always spam-free.
TLDR
Claude Sonnet 3.5 produced a 2.5-bit quantized Qwen 27B model that fits in 8GB and beats a human-made quant on quality (68.67% vs 65.33% accuracy), but runs at 23.4 tokens/second versus the human quant's 75.5 — roughly 3x slower — because its rotated trellis codes require rebuilding tiles during matrix multiplication, negating the memory savings.
Key points
Sonnet's quant achieved 68.67% accuracy on a 150-question test, beating the human quant's 65.33%.
The human quant runs at 75.5 tokens/second, while Sonnet's runs at 23.4 tokens/second.
Sonnet used rotated trellis codes instead of plain GPTQ rounding for weight compression.
The 8GB size limit averages about 2.5 bits per weight for the 27B parameter model.
Plain GPTQ in the same 8GB scored only 4.67% accuracy, effectively failing the test.
Tools mentioned
Techniques
- rotated trellis codes
- GPTQ
Stop scrolling. Start reading smarter.
Receive the day's most important AI & engineering updates in one concise email. No spam.
Transcript (captions)
Claude's on it 5.5. Squeeze a 27 billion parameter quant into 8 gibibytes and beat a published human-made quant on quality. Then it's version decoded at 23.4 tokens a second. The human quant
runs at 75.5. That's compression bench, a new paper from Baidu Chen's group, and every number here is theirs. Six agents, the two quants inside Claude code, each got
a model, a size limit, 1 H200 and 8 hours. Each one tries a design, checks its perplexity, how surprised the shrunken model is by text the full model wrote, and revises.
It ships the weights and the code that runs them. A hidden 150 question test follows, and the result needs 80% of the full model score. By our maths, 27 billion weights at 16 bits are about 54
gigabytes. So 8 gibibytes averages about 2 and 1/2 bits a weight. That's weights only, not a promise about an 8 gigabyte card. The human quant sits right at that average. How'd Sonnet win then? Plain
GPTQ rounds each weight to a nearby grid value. Sonnet swapped that for rotated trellis codes, which the paper names but doesn't unpack. It spent extra bits on the embeddings and output head, the
layers that turn tokens into numbers and back. The rest it stores as codes that get rebuilt into tiles while the matrix math runs. Plain GPTQ in that same 8 gibibytes scored 4.67%.
Count it out, seven right answers out of 150. Gate failed. Technically, it fit. In the paper's table one, the full model scores perplexity 3.221 and 74%. Sonnet, 3.363 and 68.67%.
Aista Dos Labs release, 3.764 and 65.33%. Lower perplexity is better, so Sonnet wins on both counts. That accuracy lead works out to five questions out of 150.
So Reddit is close. You'd expect fewer bits to mean more speed, since every token reads less memory. The human quant fits that at 75.5 tokens a second against the full models 61.4.
Sonnets sits at 23.4 because rebuilding those tiles mid multiplication eats the memory savings and then some. On one data center H200, the best agents now beat human quants on quality. Sonnets
version at least pays for it. Call it a third of the human quants speed. Would you trust an agent to quantize the model you run every day?