Qwen3.8-Flash-Next at 45 Tokens/sec on a 12GB GPU?

0:00 Here's your desk. A 12-gig RTX 5070, 64 gigs of RAM, and a model whose main file alone is 37.6 to 47 gigabytes, plus a separate 28.8 gigabyte table. Pour that into 12 gigs of VRAM and it doesn't come

0:14 close to fitting. Yet, the developers' own table shows 44.8 tokens a second at 128K context on exactly this setup. You'll find that number in Strata, an Engine 1 developer put on GitHub on

0:28 September 24th to run Qwen-3 8B Flash Next. A Reddit post about it calls it three times faster. My take up front, the layout's clever and it holds together, but the 3x is the shakiest

0:39 part of the story, and the reason is what's missing from the repo's own docs. To see how it could work, picture a small kitchen. Your 12 gigs of VRAM are the kitchen. Out back, there's a big

0:49 pantry, your 64 gigs of RAM with a second cook standing in it. And under the floor, there's a basement, your SSD. Strata gives every part of the model a room. Moi aussi,

1:00 how big is this menu anyway? Strata calls the model 125B. Hugging Face's page lists about 180B. The spec list on that same model card settles it. 125 billion parameters in

1:12 the core model, plus a 51 billion parameter engram table, and a 4 billion parameter draft layer called MTP, short for multi-token prediction. Add those up and you land at 180. So, both numbers

1:25 are right, which is the most annoying possible answer. 125 is the part doing the thinking. 180 is everything in the box. When you read a post, check which one it's counting, because those extras

1:37 end up in very different rooms of this kitchen. Inside, it's a mixture of experts model. Each of its 48 layers has 512 small expert networks. Multiply that out and you get 24,576

1:50 experts, which is exactly the count Strata works with. Each one's a recipe card on a very long menu. Why does a small kitchen stand a chance? For each token, a router in every layer

2:01 picks just 10 of those 512 experts, plus one shared expert that always runs. Across all 48 layers, that's 480 routed recipes per token out of 24,576. Count parameters instead, and you get

2:16 about 6 billion active per token out of 125. 24,000 dishes, and any single order touches about 2% of them. Even the Cheesecake Factory would call that excessive. Yours never needs the whole

2:28 menu out at once, just the right 2% fast. So, what goes on the counter? Stuff every order touches. The parts that mix information between tokens, attention layers, and a second

2:38 type called Delta net, routers, shared experts, and the output head that picks the final token, and the KB cache, which is the memory of your 128K conversation. All of it stays in VRAM. These are the

2:50 knives and pans you grab on every single token. Then there's a shelf of recipes kept close at hand. Strata sets aside about 6 GB of VRAM as an expert cache, holding roughly 4,500 of the 24,576

3:04 experts. So, that's under a fifth of the menu. When the router picks a dish that's on the shelf, the GPU cooks it right there. Out in the pantry, you've got the full

3:13 menu. All 24,576 experts are pinned in system RAM, even the ones with a copy on the shelf. Between pantry and kitchen, runs a hallway. PCIe 4.0 at about 26 GB a

3:26 second in Strata's diagram. Hauling expert weights down that hallway every token would keep the kitchen waiting. So, Strata sends a cook to the recipes instead. When the router picks a recipe

3:36 that isn't on the counter shelf, the second cook, your CPU, makes it right there in the pantry from RAM. Meanwhile, the GPU keeps working on its own part of the same token. Both plates land on the

3:47 same ticket. Your second cook isn't free, though. Strata runs a model in a few compressed versions called quants that store each weight in fewer bits. IQ3_XXS,

3:58 the one behind 44.8, squeezes weights to roughly three bits. The Q2 and IQ2 versions go down to about two. For IQ3_XXS, Stratta's table lists about 43 GB of

4:11 experts plus about six more in RAM and marks it slower with more work on the CPU. Treat that as ballpark since it doesn't line up exactly with the file sizes in Stratta's diagram. Either way,

4:23 a big chunk of your 64 gigs is spoken for before you've opened a browser tab. Down in the basement sits that 51 billion parameter N-gram table. An N-gram is just a short run of tokens in

4:34 a row and this table holds a learned entry for them. Call it a giant phrasebook. On disk, it's 28.8 GB and Stratta leaves it on the SSD. Only small pieces ever come up to the kitchen. Why

4:46 can a phrasebook live on a slower shelf? Because of how it gets used. Unsloth, who published compressed versions of models, store it at 4-bit in theirs. You don't run heavy math across

4:58 the whole thing. You flip to one row, read it, and close the book. And there aren't many lookups. Stratta's diagram shows 16 row reads per token from the NVMe drive issued a step ahead so the

5:09 pages are already on their way up before the kitchen asks for them. One more helper works the counter. The sous chef, which is that 4 billion parameter MTP layer. Guessing is its whole job. It

5:20 quickly pre-plates up to three tokens it thinks come next. Then the full model checks those guesses in one pass. Going in order, it keeps each one until it hits a guess it disagrees with, then

5:31 writes its own token at that spot. Stratta reports an average of 2.4 to 3.2 tokens per verification pass. So one expensive trip through the kitchen can send out two or three plates instead of

5:42 one, but one token through the whole thing. Counter tools fire on the GPU and each layers router picks its experts. Cached ones cook in the kitchen, the rest cook in the pantry both at once. 16

5:54 phrasebook rows arrive from the basement fetched a step early and the sous chefs guesses get checked in a single pass. So much for how it could work. Time to check the receipts. Strata publishes a

6:05 measured speed table for the RTX 5070 and it's the most useful thing in the repo because the numbers floating around online don't all agree with each other. At 128K context, the IQ3_XXS

6:18 version, the one we opened with, makes 44.8 tokens a second. Right above it, the 2-bit IQ2_XS row says 52 and the Q2_0 row, also 2-bit, says 65.1.

6:32 You may also have seen about 65 tokens a second quoted for Strata. Two cells in this table match it. The 2-bit Q2_0 row at 128K, 65.1 or IQ3_XXS

6:46 at a short 4K context, 65.6. So 65 is a real number. It's just the answer to a different question, which is how most spreadsheet arguments start. For a 128K conversation

6:58 on IQ3_XXS, quote 44.8. Then the 3x, as it's been passed around, the baseline is 15 tokens a second and 44.8 over 15 is about three.

7:11 But that 15 doesn't appear in the repo docs and nothing there says what software or settings produced it. Three times faster than an unknown setup tells you less than it sounds like. Every

7:21 speed in this table is the developers own and none has been checked independently. Numbers listed for other cards, like a 5060 Ti or a 3090, aren't measured at all. Strata's docs call them

7:33 estimates and say to treat them as plus or minus 20%. Quality is a separate question. Strata's table labels IQ3_XXS as best, which is the repo's own word. Speed tables measure tokens a second,

7:46 not how good the answers are. Keep that in mind before you reach for the faster two-bit rows. One more receipt, and it matters before you type git clone. Strata's repo is

7:56 only days old, and its license field is empty. Under the hood, it's built on Llama. CPP and GGML, which are MIT, but Strata's own code has no license. You can read it, you've got no permission to

8:08 reuse it. So, my call, if you've got a 12-gig card, 64 gigs of RAM, and you enjoy reading engine code, Strata's worth your evening. I think the kitchen layout's a smart one, but treat 44.8 as

8:21 the developers claim. Pick IQ3_XXS, knowing it leans on your CPU, and don't build anything on it until there's a license. Their duo's design explains how 44.8 could happen on the developer's

8:33 desk. The real test is yours. Once someone else benchmarks this layout with their own RAM and their own SSD, does that 44.8 survive the trip?