Qwen 3.8 27B is 3X Faster With ONE Setting

0:00 Two numbers, same graphics card, same model file, same prompt, 60 tokens a second and 167. Nothing changed between them. No new hardware, no smaller quant, no quality given up anywhere. One flag

0:13 did all of it and it already shipped inside the model file sitting on your desk. It is called MTP multi-token prediction. And in almost every tool that runs this model, it is off. QN put

0:24 these weights out on the 14th of August 2026 with an MTP head trained in. So, the speed is free, nothing extra to download, nothing traded away. The output comes back identical, which is

0:36 where it gets awkward because the number in your terminal is probably nowhere near it. Read the measurements posted on release day and that three times traces back to one row of one reporter's table.

0:47 The reporter who measured 167 on agent turns measured 93 on pros. Same card, same file, same flag, nearly double decided only by what you asked it to do. There is also a dial on this thing and

1:00 most published recipes set it short of where it pays best. So, what is this flag actually doing? Who turned it off and what does it buy on your hardware? Start with what it fixes. Generating

1:11 text is a queue with one server, one token, one full pass through 27 billion parameters, then start over for the next word. And that pass is not limited by arithmetic. It is limited by memory.

1:23 Your card spends most of every pass dragging weights across a bus so it can multiply each one exactly once. So, a card doing 80 tokens a second is not thinking 80 times.

1:33 It is reading the same 18 GB off memory 80 times for 80 single words. Speculative decoding goes after the waste instead of the work. Something cheap guesses the next few tokens ahead.

1:45 Then the real model checks all of them in one pass for roughly what a single token used to cost on its own. Guess right and four tokens arrive for the price of one. Guess wrong and the bad

1:55 ones get thrown out and you land exactly where you started. A wrong guess costs you time. It cannot cost you anything else. That last part is the one to hold on to. The full model verifies every

2:05 drafted token before it ships. So, speculation is incapable of changing the answer, which means switching this on costs you nothing in quality. Usually, the guesser is a second model you have

2:15 to find, download, and keep resident in memory. Qwen trained one into the file instead. Ollama has 12 tags live for this model. Three of them marked MTP, and the 4-bit tag weighs 18 GB with it

2:28 and 18 without. One gotcha before any of it works. This is a dense model, so the flag half the internet reaches for, the one that pushes experts onto your processor, does nothing here at all.

2:39 There are no experts to push. It is plain layer offload plus the speculative flag. So, it is free, it is lossless, and it is already sitting on your disk. Which makes the real question, why so

2:50 few terminals are showing the fast number. Part of the answer is dull. Four runtimes support this. Four of them spell it differently, and not one ships it switched on. In llama.cpp,

3:01 it is a spec type. In vllm, it hides inside a JSON blob nested in a serving config. In SG Lang, it is a speculative algorithm. In Ollama, you do not set a flag at all. You pull a different tag.

3:14 Four names for one capability, but the sharper answer is a rename. On the 13th of May, 2026, 3 months before this model existed, the llama.cpp project changed how that flag is

3:25 spelled. Multi-token prediction itself landed in the main branch 3 days after that. The old spelling was not removed with an error. It is still accepted and quietly ignored. So, generation keeps

3:36 running and speculation simply stops. Your throughput collapses, your command line still reads correct, and the law gives you no reason to look. One write-up measured the damage on a 4090,

3:46 about 140 tokens a second falling back to about 70. Half the throughput gone from a setting you are certain you configured because one word inside it went stale. And every guide written

3:56 before that Wednesday still carries the dead spelling. So, the reason this feels like a secret is not that the capability is obscure. The capability shipped. The instructions rotted. Fix the word,

4:07 switch it on, and you meet the dial. The flag takes a number. How many tokens to guess ahead? On the day this model shipped, a developer swept that number from two up to eight on a workstation

4:17 Blackwell card and published the whole curve. At two guesses ahead, 81% of drafts survived the check. At five, 56%. At eight, 40. Guess deeper and you are wrong more often, which is exactly what

4:30 anyone would predict. But, throughput refuses to follow acceptance down. It climbs to 115 tokens a second at five steps, then falls back. There is a peak in the middle, and most recipes online

4:42 ship two or three, which leaves as much as 27% on the table. The number the community copies is not the number the hardware wants. Then, a second reporter found something stranger.

4:52 His draft acceptance had collapsed from around 85% on the previous release to around 65 on this one. And the cause turned out to have nothing to do with the model. It was temperature.

5:02 He was running the sampler at 1.0 because that is what this model's own authors recommend. A hotter sampler flattens the next token distribution, which makes the draft head guess wrong

5:12 more often. He swept it downward and watched acceptance climb. So, the setting that decides your speed is not the speed setting. It is the quality setting, three lines earlier in

5:21 the config, which puts a real choice in front of you before any of the rest of this matters. Would you turn that temperature down against the recommendation to buy the throughput

5:30 because the ceiling on Nvidia is real, and it sits well above anything a flag can reach? SG Lang published 206.1 tokens a second on 15090 on release day, and a reader replied underneath claiming

5:42 216 on Windows. For the small desktop machine, they quoted 38.28. An independent operator benchmarked that same box for a full day and got 34 in real use, 38 averaged, 46 at peak. About

5:56 as cleanly as a vendor number ever reconciles. But, read what produced it. That headline is not multi-token prediction. The credit belongs to a 4-bit format called NVFP4,

6:07 running alongside a completely separate drafter, and the 4-bit format only exists on Blackwell silicon. So, on a 3090 or a 4090, that number is not slower. It is unreachable. The tensor

6:18 cores that run 4-bit natively are simply not on the die. Somebody asked exactly that under the announcement, and nobody there contradicted them. On AMD, the interesting part is not a number, it is

6:29 a choice. AMD's own release day post quotes up to 51.8 tokens a second on a workstation Radeon, and every figure in it was measured on Vulcan, not on ROCm, not on the stack AMD builds, staffs, and

6:42 markets. Independent sweeps put Vulcan 20 to 30% ahead on generation for these cards, with ROCm ahead on prompt processing. But, a vendor publishing its own launch numbers on somebody else's

6:53 back end says more than any blog post about driver maturity ever could. And a 16 GB Radeon owner posted both halves of the real trade. At 128,000 tokens of context, 32 a second. Half the context,

7:06 switch the flag on, 51. That is nearly 60% more speed bought with half his context window. The download is free. The memory is not. The people who merged this measured about 2 and 1/2 extra GB

7:19 held resident for the draft head. On Apple, the port is real, and somebody checked it properly. A 48 GB laptop went from 8.3 tokens a second at 8-bit to 20.3 with a ported

7:31 drafter. 2.45 times on average, three times on max, and under two on chat. That same run turned up something better than the multiplier. 8-bit with drafting beat 4-bit without

7:43 it, 20 against 14.6. Better answers and more speed in one run, which is a trade quantization, does not usually offer anybody. A desktop Apple owner reported the biggest jump in

7:54 any of these threads, 10 tokens a second on llama.cpp, P 40 to 70 on a metal engine with handwritten kernels for this architecture. The flag did not do that, the kernels did. But the loudest reply

8:07 under that post was unimpressed, and its objection is the sharpest thing in the whole pile. He asked whether the stack does prefix caching for this hybrid architecture at

8:15 all, because an agent session rereads its whole history on every single turn. If it does not, he argued, three times becomes half a time the moment a session gets long, and his claim is that no

8:26 server on the market yet has both prefix caching and speculation for this architecture. That is an argument rather than a benchmark. It is also the right

8:34 question. Two shiny things arrived this month that do not transfer either. 17 days before this model shipped, a second speculation method landed in llama.cpp with strong numbers on smaller models.

8:47 A developer converted the official version for this one, ran it on a 5090, and reported no gain at all, sometimes a loss. Draft acceptance, 29%. And a fork advertising 30 to 50% more

8:59 throughput turns out, in its own published table, to mean a different model family entirely. For a dense 27B like this one, that same table claims 5 to 7%. This model is not on its

9:11 supported list, so strip it back. Where does a free, lossless, already downloaded speedup buy you nothing? On the machine that needs it most. A 12 GB laptop owner published both runs. 4-bit

9:22 with the flag on gave 4 and 1/2 tokens a second. 2-bit with the flag off gave 12.8. 2.8 times faster from the crude thing the clever thing was meant to replace. His model did not fit on the

9:34 card, so his real bottleneck was the cable out to system memory. Speculation hides memory latency. It cannot hide a transfer that has not started yet. So, here is the call. Turn it on today on

9:45 every machine where the model already fits because it is lossless and it is sitting in the file you downloaded. But, buy your video memory for the quant, not for the flag. Multi-token prediction

9:55 multiplies the speed you already have. It cannot create one. The receipts are three curves. Acceptance falling from 81% to 40. One card getting 60, 93, and 167 off the

10:08 same file depending only on the job. And a laptop where switching the flag off one, 12.8 against four and a half. What I am arguing with is not a company. It is the single number. Every figure here

10:20 is true and every one of them is a ceiling somebody measured with the conditions cropped off the end. So, if a flag this free has been shipping since May and still arrives

10:28 switched off everywhere, how much of the compute being bought right now is paying for a default that shipped in the off position?