Run Qwen 3.8 27B Locally: Opus 4.6 max Performance on Your GPU?

0:00 3 in the afternoon, UTC. On the 14th of August, Alibaba's QN team pushed a folder of files onto HuggingFace. 11 minutes later, someone on Hacker News wrote the sentence that put this video

0:12 on your screen. 27 billion dense parameters at Opus 4.6 level. Opus at home. That thread finished its first day past 1,000 points and 600 comments. Because if that sentence holds up, the

0:25 machine already sitting on your desk just changed job description. So, this video does one job. It works out what that sentence actually cost you. Measured in gigabytes. The model is QN

0:36 3.827B Apache 2.0. The weights are on your drive and they are yours. Dense, not a mixture of experts. Native vision built-in 262,000 tokens of context

0:49 stretchable to a million and a model card claiming it beats a Frontier commercial model on four separate benchmarks. It also loses on three of them. We are saying those out loud

0:58 because the losses are what make the wind believable. Then we walk down the hardware ladder together. 24 GB, 17, 16, 12, 8. One of those rungs is a trap. It is the one most people are standing on.

1:12 And the whole thing turns on 73 MGB. And at the bottom, the number that decides your speed turns out not to be the number printed on the box. Start with what QN published. SWEBench Pro is the

1:24 agentic coding test that has been breaking models all year. Real repositories, real failing tests, and an agent that has to actually fix them. QN 3.827B

1:35 scores 61.7. Opus 4.6 Max in the column immediately beside it scores 53.4, 8.3 percentage points in favor of the model you can download for nothing. Remember that number because we have not

1:48 yet asked the obvious question about who ran the test. OS World Verified is computer use a model driving an actual desktop and clicking actual menus. 84.3 against 72.7

2:00 11 12 points. If bench asks whether a model does what the instruction said rather than what it felt like doing 79.5 against 62.5 17 points live codebench version 6

2:13 competitive programming 90.3 against 88.8 Eight. That gap is a point and a half. Close, but the right side of the line. Two more before the losses because they are the two. The coverage skipped.

2:26 Driving an Android phone. 81.9 against 62. Multimodel software engineering. Reading a screenshot of a broken UI and fixing the code behind it. 38.6 against 27.1. Now, the losses, which are the

2:39 more interesting half. Terminal bench 2.1. The long terminal sessions where an agent has to keep its head for an hour. 73 against Opus at 78.2. GPQA diamond graduate level science 89.2 against 91.3

2:55 which is tight. And HLE the hardest multiddisciplinary exam anyone has assembled 30.8 against 40. That gap is 9.2 points and it is not close. On the deepest reasoning we know how to measure

3:08 a 27 billion parameter model is still a 27 billion parameter model. So the shape of it is this. On the work a coding agent does all day, the downloadable model is ahead. On the work a research

3:20 assistant does, it gives ground. The question is which one your Tuesday looks like, which changes the question. It stops being how good is it and becomes what does it cost to have it in the

3:30 room. The official weights ship at BF16, which is 54.66 GB on disk. That does not go onto a gaming card. So, the local answer is a quantized build. Quantization rewrites every weight at

3:43 lower precision. The file shrinks, the model becomes slightly less exact, and there is now a table telling you exactly how much less exact. Atomic Chat published theirs, measured against the

3:54 full model. Their 4-bit build picks the same Nex token as the original 95.59% of the time. Their one bit build 76.34. So 4bit is a rounding error, and one bit is a different model wearing the same

4:07 name. Keep both numbers in your pocket. We need them shortly. Top rung 24 gigabytes a 3090 a 490 a 590 a 7,900 xtx 6bit the file called Q6_k 22.88 88 GB. That is the best quality

4:26 anyone runs at home. And after you load the vision file alongside it, you still have around 2 GB free for context. Nothing to think about. One rung down. And this is the number for the

4:36 thumbnail. 17 GB. Standard 4bit is 17.11. Unslo's dynamic 4bit is 17.92. Unsllo's own guidance says plan for 17 to 19 GB of combined memory for the

4:50 4-bit build. QN's documentation points at the same tier. This is the recommended way to run this model. And this part is worth being precise about. That is the whole model. All 27 billion

5:02 parameters. The full 262,000 tokens of context. The vision 4bit is the same model stored coarser, not a smaller one. Then comes the rung that eats people. 16 GB. a 460Ti,

5:16 a 560Ti, a 570 Ti, a 970 XT. The most common card in the room. Look at the two numbers again. Standard 4bit is 17.11 GB. A 16 GB card actually holds 17.18.

5:32 It fits by 73 mgabytes. And the reason that is a trap rather than a win is that 73 megabytes is not headroom. The key value cache lives in VRAMm beside the weights. and one user measured 32,000

5:45 tokens of context costing two and a half gigabytes of it. Unsloth's dynamic 4bit does not even reach the trap. At 17.92 GB, it is about 700 megabytes larger than the card before the cash shows up

5:58 at all. So, if 73 megabytes decides whether a model loads on the card you already own, what exactly were you buying when you bought it? The 16 GB answer is to step down one notch and buy

6:09 the space back. IQ4_Xs at 15.71 GB or the threebit dynamic build at 11.91. Then shrink the cache itself. Cache type K at 8 bit, cache type V at 4bit. Two

6:24 flags on the llama.cpp command line that have the thing competing with your weights. Here is a real launch day command from a user on a 490. IQ4_nl weights. The vision file attached. Both

6:37 cache types quantized. Flash attention on 170,000 tokens of context and speculative decoding enabled. 70 to 80 tokens a second. 12 GB next. A 30060 or a 30080. The 3bit dynamic build at 11.91

6:54 leaves you under a gigabyte of room. It loads and context is where you pay for it. Atomic Chat's own guidance for a 12 GB card is a two-bit build and their table prices that add about eight points

7:05 of token agreement against 4-bit. On a 12 GB laptop card, a launch day tester measured 3 to 4 and 1/2 tokens a second, which is reading speed for a slow reader. Which brings us to the rung the

7:17 comments keep asking about 8 GB. The answer is no. And it is worth a few minutes because this is where the internet is most confidently wrong. The claim doing the rounds is that a 1-bit

7:28 build exists that gets a 27B model onto an 8 GB card. So, here are the actual floors. Read straight off the file listings the day after release from the five publishers whose quants most people

7:40 download. Unsllo's smallest file is 9.01 GB. Barttowski's smallest is 9.39. Mr. Raider Macker stops at 10.86. LM Studios Community Build does not go below 16.81. And the llama.cpp CPP projects own

7:56 repository stops at 18.97 which is larger than the card most of the people asking this question own. Four of those five do not go below 2bit at all. There is no AWQ build and no GPTQ build and

8:10 most of the 1-bit files people link to belong to a different model entirely. QN shipped two things in August 2026. This 27b under Apache 2.0 and a 2.4 4 trillion parameter mixture of experts

8:23 under a custom license with revenue thresholds in it. The extreme one-bit quants are for the big one. Now, one publisher does have a real one-bit build of the 27B. Atomic Chat, who make a

8:34 local chat client and also quantize their own files, ship a 1-bit build at 8.5 GB. In the units a graphics card actually uses, that is 7.91 GB, which clears an 8 GB card by 92 megabytes. 92

8:49 megabytes for the cache, the compute buffers, and whatever your desktop is already holding. It does not run. And at 76% token agreement, you would not want it to. The one bit build exists and it

9:01 still does not save you, which is when people reach for the SSD and llama.cpp will let you try. Note that this is not the same thing as swap. On one documented rig, letting Linux swap

9:12 handle it came in at about a third of a token a second, and it grinds the drive doing it. What llama.cpp CPP does instead is memory map the model file. The operating system pages weights in

9:23 from the drive as they are needed read only with nothing written back. So the drive is only ever being read. Two things kill it though. Passing no map switches the whole mechanism off. An

9:34 MLOCK cannot make an oversized model fit because all it does is pin pages that already fitted. But the real obstacle is architectural and it is the one thing about this model that everything else

9:44 follows from. It is dense. Every one of those 27 billion weights gets read for every single token. A mixture of experts reads a few percent of itself per token, which is the entire reason the CPU

9:57 offload flags people paste from guides exist. On a dense model, there is nothing to skip. So those flags have nothing to do. Do the arithmetic yourself. 17 GB of weights read once per

10:08 token. A fast Gen 4 drive sustains around 5 GB a second. That is 0.3 tokens a second as a ceiling if it is truly paging. Back in January 2025, someone ran a 671 billion parameter mixture of

10:22 experts off an NVME drive and got 1 to 2.4. There is no published measurement of a dense 27B running off a disc. So treat that 0.3 as a bound rather than a result. The rule underneath it is

10:35 simple. Once the majority of your weights are coming off the drive, you are under a token a second and it stops being a tool and becomes a batch job. For reference, a laptop CPU running this

10:45 model out of ordinary system RAM with no disc paging at all was clocked at 0.94 tokens a second. That is the fast version of too slow. So VRAM decides whether the model runs. It does not

10:58 decide how fast it runs. And that caught a lot of people out this week. QN trained this checkpoint with a multi-token prediction head baked into it. It is a small extra layer whose job

11:08 is to guess the next few tokens before the main model gets there. The main model then checks all the guesses in one pass. The correct ones are kept effectively for free. The wrong ones get

11:18 thrown away. Same weights, same output, far fewer round trips. Llama.cpp CPP merge support for it in May 2026. And Olama now ships MTP tags at exactly the same file size as the plain ones, 18 GB.

11:32 Either way, the head is already inside the checkpoint, so it cost nothing to carry. The gains are large. On the previous generation of this model, a single 5,090 went from 38 tokens a

11:44 second to 65, and a Stricks Halo desktop went from 7.4 to 18.1. on this one. On day one, a Stricks Halo owner reported roughly nine tokens a second without it and about 18 with. A second Stricks Halo

11:58 report put five bit at 10 and a half fresh, dropping to 9.4 once 32,000 tokens of context were loaded. And on the ES Lang team's own launchpost, a single 5,090 running the NVFP4 build

12:11 with speculation cleared 200 tokens a second. There is a catch and it is specific to this architecture. QN 3.8 8 uses hybrid attention and an open llama.cpp issue from May 2026 measured

12:24 draft acceptance on hybrid attention models at 35 to 37% on long tasks against a 70 to 90% baseline. 2/3 of the speculative work thrown away. Which means the same head that doubles your

12:38 speed on a short prompt can be doing close to nothing on a long one. Measure it at the context length you actually work at rather than at the one in the headline because those are rarely the

12:47 same number. Then four more things that cost people an evening. Vision is a separate file. The main GGUF has no eyes at all. The projector is a 931 megabyte download you pass in alongside the model

13:00 and skip it and what you have downloaded is a texton model. NVFP4 the fastest format on the list is Blackwell only. RTX 50 series, DGX Spark, B200 and up. On anything older, it is not slower. It

13:15 simply does not apply to you. On AMD, the backend with no marketing behind it is the quick one. Community measurements through 2026 put Llama.CP's Vulcan path roughly 30 to 40% ahead of Rockm on

13:28 token generation. Though on workstation cards, that result flips the other way. So measure your own card rather than believing either camp. And on a Mac, use MLX rather than the generic runner.

13:39 Because what decides Apple silicon is memory bandwidth. An older chip with a 400 GB per second bus beats a newer one with 273, which is the opposite of how the model numbers are ordered in the

13:50 shop, which is the right moment to say where all these speed numbers came from because that matters more than the numbers do. Every tokens per second figure in this video is a user

14:00 self-report from the first two days. different quants, different context lengths, different flags, different rooms. Treat them as a range, not a specification. And every benchmark

14:10 number came off QN's own model card, including the Opus column beside it. QN ran both sides of that table. One of the benchmarks in it is called QN Sebench. It is their in-house harness, averaged

14:22 over three runs with an 8-hour timeout. The competitor scored 63.8 on it. QN scored 79. That is normal industry practice. And the problem with it has nothing to do with dishonesty. The

14:34 problem is that as of this recording, nothing outside QN has reproduced those headline scores on a named third party harness. So the shape of the claim is solid and the exact decimals are

14:45 provisional. Here is the call anyway because knowing which part is provisional is enough to decide. If your local work is coding in agents driving software, this is the model. And the buy

14:55 is a 24 GB card running the dynamic 4-bit build with multi-token prediction switched on. 24 GB is not a magic number. It is the smallest card where the quant stops being a decision you

15:07 keep revisiting. Full quality at 6bit, room for the key value cache, room for the vision file, and both unsloth and QN pointing at that tier in their own documentation. and the sharper version

15:18 which is the actual lesson. A 16 gigabyte card properly tuned IQ4_XS with a quantized cache and MTP on will beat a 24 GB card running defaults. Buy VRAM for the quant buy speed from the

15:32 runtime. The 5% this is wrong for if your day is deep research reasoning. Opus is nine points ahead on HLE and five ahead on long terminal sessions and the subscription keeps that job. I would

15:44 still take the local one at twice the card price because the losses land on tests you do not run and the wins land on the ones you run before lunch. And the thing worth being annoyed at is not

15:54 a company. It is the habit of shopping for the number on the box. The same habit that made VRAM the only spec anyone quotes and left multi-token prediction which is free already in the

16:04 file and roughly doubles your throughput sitting in a footnote. So here is the question to leave with. If 27 billion parameters on one consumer card really is at this level and the file is Apache

16:16 licensed and already sitting on your drive, what exactly are you still renting an API