Qwen 3.8 27B Starts Answering 1.78× Faster With One Trick

0:00 Paste a long document into a local model and watch the cursor blink. On one test setup, a 27 billion parameter model squeezed to one bit sits there for 12.27 seconds before the first word shows up.

0:12 8,000 tokens in and nothing out yet. Long enough to wonder if it's thinking or just gone. A paper posted September 22nd by researchers at ISTA and Nvidia cuts that to 6.90 seconds. Same machine

0:26 and the same one bit weights still generating the answer. And the model score on MMLU Pro, a tough multiple-choice reasoning test, jumps from 29% to 61.5.

0:37 Look at that second number again. Whatever writes the answer wasn't retrained. Same checkpoint, untouched. So, where did 32 points of accuracy come from?

0:45 If you run models on your own machine, the quantization advice you've probably heard treats the model as one job. Shrink the weights so it fits and generates faster. But that first weight

0:55 is a different job with a different bottleneck. Squeeze down to one or two bits and you speed up the writing without fixing the weight and you pay for it in accuracy. Short answer,

1:05 reading your prompt and writing the answer hit different walls in your hardware. And yes, sometimes this trick makes things slower. Picture one cook with two jobs. Job one, read your

1:15 prompt, all of it at once. We call that prefill. Job two, write the answer one token at a time, which is decode. Same cook, same kitchen, very different days. Reading is a busy prep station. Our cook

1:29 grabs an ingredient from the fridge once and uses it on thousands of plates lined up down the counter. In model terms, each weight comes out of memory once and gets reused across every token in your

1:39 prompt. So, the trip to the fridge barely registers. What sets the pace is the chopping, the actual arithmetic. According to the paper, long prefill is generally compute-bound. More math per

1:51 second means a shorter wait for your first word. Writing is a sandwich counter that serves one order at a time. For every new token, the cook fetches every ingredient, uses each one exactly

2:01 once, and starts over. At batch size one, each weight contributes a single multiply add, so the chopping's trivial and the walking dominates. Your weight before the first word is mostly the

2:12 reading job. You'll see it called time to first token, and it's the number this paper goes after. Tiny weights help writing for exactly this reason. A one-bit weight is a

2:21 lighter bag to carry, so every trip is shorter. All that trinket advice is really advice for this counter. Over at the prep station, the cook wants something else, cheaper chopping. Low

2:32 precision arithmetic, like NVFP4, a four-bit number format NVIDIA's newest chips can do math on directly, makes the chopping itself faster. Boil the paper down and you get one line. Low precision

2:44 math speeds up reading, compact weights speed up writing. You can see the imbalance on the test machine. NVIDIA's DGX Spark has 128 GB of unified memory moving at 273 GB a second next to

2:58 up to a petaflop of four-bit compute with sparsity. A giant kitchen with a narrow hallway to the fridge. Try serving both jobs with one set of weights. Weight only one-bit formats

3:08 like IQ1_S shrink the bags, but they're aimed at the carrying, not the chopping. Go the other way, NVFP4 everywhere, and every writing trip hauls a heavier bag. Either

3:19 way, one job gets the wrong tool, so the author stopped choosing. They call it disaggregated quantization. Each half of the job gets its own format. Keep the tiny one-bit copy in memory for writing.

3:31 That's the decoder. In the headline test, it's Unsloth's IQ1_S checkpoint, left exactly as it was. Then train a second copy in NVFP4 that only does reading, called the pre-filler. Two

3:43 copies of a 27 billion parameter model sounds like it undoes the whole point of squeezing the first one. Except the released pre-filler, all 12.8 GiB of it, never moves into device memory. Instead,

3:55 it streams from the SSD one block of layers at a time into two small buffers. While one block works through your prompt, the next is loading into the other buffer. Then they swap. In the

4:06 paper, this is offloaded disaggregated prefill, or ODP. And only the big matrix multiplies, the linear projections, get this treatment. In kitchen terms, the prep cook now has

4:18 two cutting boards. One's in use, and a runner is already stocking the other from the warehouse, the SSD. As long as the chopping takes longer than the running, the cook never stands around.

4:28 And those two boards don't take new counter space. According to the model card, they're carved out of memory already set aside for the output head, the last layer that scores possible next

4:38 words. So, they add zero device memory, no new allocation at all. They just borrowed space the output head was already holding, which is the most developer fix imaginable. What you pay

4:48 instead is extra reads from the SSD. The headline number comes from a very specific setup. Qwen 3.8 27B, Alibaba's dense open weights model from August, running in the authors' fork of

5:01 llama.cpp on a DGX Spark with an 8K token prompt. Time to first token drops from 12.27 seconds to 6.90, or 1.78 times faster. Across prompts from 4K to 32K tokens,

5:16 the speedup ran from 1.38 to 1.78 times. Each point is three runs after one warm-up with prompt caching off, so a cache isn't doing the work. Every figure here is the authors' own, not

5:29 independently reproduced. So, where did those 32 points come from? With the writer frozen, the gain has to come from the reading side. Put the NVF P4 pre-filler on the prompt, and the

5:40 one-bit model goes from 29.04 to 61.54% on MMLU Pro. On MMMU Pro, a separate test that mixes in images, it goes from 24.39

5:52 to 59.65. My take. At 1-bit, a lot of the damage was happening while the model read. Give it a clean read of your prompt, and the same clumsy writer does far better.

6:02 Careful though, writing doesn't shrug off low precision. In fact, the paper found the reverse. On decode-heavy benchmarks, squeezing only the writing loses two to four times more accuracy

6:13 than squeezing only the reading on most models, and up to seven times on Gemma 3 1B. So, the 1-bit writer is still the weak link. You keep it because it's fast at

6:22 the counter. All the pre-filler does is stop the reading from piling on more mistakes. And look at what the doubling is measured against. IQ1_S is about as squeezed as a model gets.

6:33 29% on MMLU Pro is a model that showed up to the exam without its glasses. At 2-bits, the paper reports much smaller gains, roughly 4.5 to 7.4 points, depending on the benchmark.

6:45 The catch lives in that SSD. Streaming costs time up front. On a short prompt, there isn't enough chopping to hide the running. So, the cook stands around waiting on the runner. Per the paper,

6:56 short prompts can stall on loading and come out slower. Where's the line? On DGX Spark, the paper says compute overtakes loading around 8K tokens for the Qwen 3 models. Its figure 1 shows

7:07 the 27B already ahead at 4K. Different lines though. One marks where loading gets fully hidden. The other where the whole setup already beats the 1-bit baseline. Below 4K, the measurements

7:19 stop, so I wouldn't assume it helps. If your usual prompt is fix this typo, this trick can be a very elaborate way to wait longer. Where it earns its keep is the long pastes. A full document, a pile

7:30 of logs, a big chunk of your codebase. Your drive matters too. Everything here leans on how fast the SSD can feed those buffers, and the paper doesn't say which drive it used, or how fast it reads. By

7:41 the paper's own logic, a slower drive would push that break even later. So, treat 1.78 as belonging to this one setup. You'll also need the right chip. The release pre-filler requires a

7:51 Blackwell GPU, either a GB10 like the Spark or an RTX 50 series card. As released, it won't help a Mac or an older Nvidia card. On the decoder side, it pairs with four unsloth checkpoints:

8:04 IQ1_S, IQ1_M, IQ2_XXS, and IQ2_S. The authors list the edges themselves.

8:14 They didn't evaluate high batch serving, and they didn't test multi-turn chats or agents. In a long conversation, the writer keeps running notes on the chat so far, called the KV cache. Meanwhile,

8:25 the pre-filler rebuilds its own set of notes, and the two could differ. They flag that as untested. Mixture of experts models are another edge. They say ODP doesn't carry over cleanly

8:35 there. What they're aiming at is local dense models like Qwen 3.8 27B, Apache 2.0 with a native context of 262,144 tokens. Plenty of room for the long prompts this trick needs. What about

8:51 typing speed once the words start? Streaming doesn't change the decode speed up, according to the paper. You've still got the same one-bit decoder writing. So, the speed win is all in the

9:00 wait before the first word. The accuracy win comes from the better reader. My verdict, if you run Qwen 3.8 27B on a Blackwell machine with one of those one- or two-bit unsloth decoders, and you

9:13 regularly paste in 8K tokens or more, this is worth trying with the author's fork. At one bit, that setup got the faster first word and a much smarter model in the same move. On a Mac, an

9:23 older GPU, or a mixture of experts model, skip it for now. Mostly short prompts, skip it, too. Unless you're at one bit and happy to risk some extra weight for a lot more accuracy. Still,

9:34 the idea travels. Your model has two jobs, and they don't have to share one set of weights. What I'd want to see is a chat case. Turn after turn, the reader and the writer each keep their own notes

9:45 on the conversation. Do those notes drift apart? So far, it's unmeasured, and that's the experiment I'd run.