Enjoying this issue?
Get tomorrow's AI & engineering digest in your inbox — hand-picked, summarized, and always spam-free.
TLDR
Qwen 27B's 381 tok/s on a single RTX 3090 is only achievable when the model is copying text already present in the prompt; free-form generation runs at about 70 tok/s. The speed depends entirely on how guessable the next tokens are, with prompt lookup speculative decoding providing a 2.4x boost for copying but minimal gain for creative writing. This makes the headline number a 'photocopier benchmark' rather than a general-purpose speed claim.
Key points
Qwen 27B achieves 381 tokens per second on one RTX 3090 only when copying from the prompt.
Free-form generation with the same setup runs at approximately 70 tokens per second.
The speed boost comes from speculative decoding using a small drafter that guesses tokens via prompt lookup.
The model is requantized with a quality cost, so its benchmark scores against Claude Opus 4.6 Max do not directly apply.
Code prompts achieve about 188 tokens per second, between copying and free-form speeds.
Techniques
- speculative decoding
- prompt lookup
- requantization
Stop scrolling. Start reading smarter.
Receive the day's most important AI & engineering updates in one concise email. No spam.
Transcript (captions)
Quinn's 27-billion-parameter model hit 381 tokens per second on one RTX 3090 with Hyper Quinn, but only while it was copying. Free-form answers in the same repo test run about 70. Alibaba's model
card lists a full-precision model ahead of Claude Opus 4.6 Max on 16 of 24 benchmarks. That's vendor reported, not rerun in the same harness. The 3090 build is requantized with a small
quality cost, so those scores don't strictly carry over. The speed is the part to check. The trick starts with a small drafter. It writes a block of guest tokens seven at
a time in one quick pass. The big model checks the whole block in a single pass. Every right guess is a token you didn't wait for. Wrong ones get thrown out, so the output doesn't change. Is the
drafter just brilliant at copying? Not really. The prompt does the work. With lookup on, it takes the last few tokens it just wrote, searches the prompt for the same words, and offers whatever came
next there as guesses. So the block can stretch to 15 tokens when the model copies. Check them all at once, and about 15 tokens come out per step. Hyper Quinn's optimizations doc
has the table. Copying from a 25,000 token prompt, greedy sampling, drafter only, no lookup, 4.72 tokens per step, 159 a second. Lookup and a 15-token block, 14.97,
381. Call it 2.4 times faster. Both speeds are the authors' own measurements. Free-form writing is the other end, about two tokens accepted per step, around 70 tokens a second. Plain
chat sits between 127 to 133 in the repo's numbers. Divide copying by free-form and it's about 5.5 times faster. A photocopier benchmark, basically. So speed depends on how
guessable the next tokens are. Text already in your prompt is the easiest to guess, which is why copying is fastest. Code is more predictable than prose, so code prompts hit about 188 in that
reproduction. Concurrency and context length move the number two. Froxy trains 381 as the authors best case. Always ask 381 tokens per second doing what?