Run a 300B AI Model on a 128GB Mini PC (No NVIDIA Needed)

0:00 I worked out the weights for GLM 5.3 flash, a 320 billion parameter model at 16-bit precision, and they come to about 640 GB. Yandex Lab says that model runs on one mini PC with 128 GB of memory.

0:15 Divide one by the other and it's five times too big. If you've wanted a big model at home, that's the wall. Modem Guides, a hardware blog, estimates 128 GB only

0:26 just holds a model like this at one or two bits, and comfortable four-bit wants 192 to 256. That's an estimate, but it's a fair summary of the popular take. QiaoGen is a new engine built on Xilinx

0:38 3 for AMD Strix Halo chips. Its Redmi list measured speeds of 26 to 44 tokens a second for two big models, and a token is roughly a word or a piece of one. The packs weigh about 100 and 105 GB. The

0:51 blog's estimate is about whether the model fits. The repo's table is where speed shows up, so that's what I checked first. I read the repo's measured table, both model cards and the official model

1:01 descriptions. One half of the claim holds up better than I expected. The other half is thinner than the headline. So, the question is how 640 GB of model fits in 128 and still writes at a useful

1:13 pace. Four suspects. One, the model is smaller than advertised. Two, only a sliver of it is real. Three, it's too slow to use. Four, the squeeze wrecks it. By the end, you'll know which

1:26 suspect survived and whether a box that starts around $3,300 is worth trying. One question waits until last. After a squeeze this hard, how much of the original is left? Suspect one, the model

1:38 is smaller than advertised. The official descriptions say no. Xiaomi's page for the RL version of MiMO V2.6 flash lists 309 billion total. The Yandex pack is a different variant of that model, and I'm

1:50 assuming the same size. ZI lists GLM 5.3 flash at 320 billion. The repo says 300 billion class. So, the class does a little rounding. Put that in bytes. At 16 bits, each parameter takes two bytes.

2:05 That works out to about 618 GB for MiMo and 640 for GLM. The official weights ship at 8 bits. So, call it 320 GB for GLM. And that's still two and a half times too big. The model really is that

2:18 big. Something else has to shrink. First, make the giant fit. The first lever is how many bits each weight gets. Quantization stores every weight with fewer bits and keeps the network

2:29 structure. Picture a library where every book is reprinted as a pocket edition. Same titles, same shelves. The picture breaks at one point. A pocket edition drops pages while quantization keeps

2:40 every weight and rounds it. The published packs show the result. MiMo comes in at 105 GB with most layers at about two bits per weight. GLM comes in at 99.7.

2:50 Each runs separately on the machine, one model at a time. Divide file size by parameter count and you get about 2.7 bits per weight overall for MiMo and 2.5 for GLM. That's above the two-bit

3:02 headline because not every part gets the same squeeze. GLM's card says it started from a 2.05 pack and took expert layers 29 through 44 from a 3.05 pack. So, the blog's four-bit number holds up. A

3:16 public four-bit build of GLM 5.3 flash is listed at 153 GB on disk against about 643 in 16 bit and that still doesn't fit in 128. The Kyogen packs sit near 100. A 105 GB file has to sit

3:32 somewhere the GPU can reach. Where does that much memory come from? On an ordinary PC, 128 GB of RAM and a graphics card with its own small pile of memory are two separate worlds. The

3:44 model lives in one and gets hauled to the other. The Ryzen AI Max plus 395 shares one pool instead of two separate ones. Its CPU and integrated GPU draw on up to

3:55 128 GB of LPDDR5X at about 256 GB per second from AMD based spec listings. Remember that speed because it sets a ceiling later. Even on a shared pool, a slice gets assigned to

4:09 the GPU. AMD's FAQ says its Windows graphics memory setting can hand the GPU up to 96 GB. Kyojin's readme asked for a GFX1151 machine with 128 GB and the Linux Rock

4:22 and Path is the one tested. How memory gets assigned there isn't documented, so don't carry the Windows number over. Subtract the packs from 128 and call it 23 to 28 GB left over.

4:34 That has to cover the operating system and the cache, the running notes the model keeps for a conversation. The library fits, what's left is a closet, and the operating system and the cache

4:44 have to share it. It fits with a little room. Now the stranger part. Writing one token means running the model and 309 billion parameters sounds like a lot of running. Does every token really need

4:56 all of them? No, and that's the second lever. MIMO is a sparse mixture of experts with 256 routed experts and eight picked for each token. Only about 15 billion of its 309 billion parameters

5:09 work on any one token. GLM is the same idea with 18 billion active out of 320. In the library picture, the whole collection stays in the building. Each token sends the librarian to eight

5:20 shelves out of 256. Those other shelves aren't consulted, but they still pay rent. Every expert has to stay in memory because the next token might call for any of them. The

5:30 picture breaks at labels. Don't imagine a chemistry shelf because the sources don't say what each expert knows. Suspect two says only a sliver of it is real. 15 billion in a larger coat. That

5:41 doesn't hold. All 309 billion are stored and they cost you memory. What's small is the work done per token? Two suspects are out. Suspect three, can it actually run at a

5:53 useful speed? I expected a model this big to crawl. At full 16-bit precision, the arithmetic says it would. Writing a token means reading the active experts out of memory. Decoding usually runs

6:04 into memory speed before it runs into math speed. The repo doesn't say that about these two models, so what follows is my own estimate. Try it at 16 bits. 15 billion

6:14 active parameters at two bytes each is 30 gigabytes to read per token. Divide 256 by 30 and the ceiling is about eight or nine tokens a second. Sparse experts alone don't rescue it. Now squeeze them.

6:26 At about two to 2.7 bits, the same active parameters come to roughly four to five gigabytes per token. That puts a ceiling near 50 to 60 tokens a second. It leaves out the cache and the

6:37 attention weights, so call it a rough ceiling. Bytes read per token is the clue. Both tricks so far were shrinking it. The measurements come from one machine. A Ryzen AI Max Plus 395

6:49 with 128 gigabytes of LPDDR5X. They're the developers' own numbers, not independently reproduced. Mimo's rows are medians of six runs, 128 tokens each at temperature zero. Mimo's plain

7:03 decode, no speculation, writes 28.9 tokens a second. With speculation on, it's 32.1 for prose, 34.8 for chat, and 44.3 for code. Those are the figures behind the headline range. That's 1.11,

7:18 1.21, and 1.5 three times the plain speed. Every one of them sits under that 50 to 60 estimate. So, the table and the estimate at least point the same way. The ceiling is per token, and

7:30 speculation checks several at once, so treat it as a loose guide. Longer context costs something. A plain decode row at longer prompts reads 25 to 28. Speculation is how it climbs from 29 to

7:42 44. A small drafter guesses several tokens ahead, and the big model checks all the guesses together in one pass. Mimo uses a separate 4-bit drafter, 702 megabytes. GLM uses its own multi-token

7:55 prediction, MTP. Pros gains least. I'd guess code is easier to predict ahead, but that's a guess, not something the table says. GLM's 99.7 GB pack, with its speculation setting on, writes 26 to 30

8:09 tokens a second across the three readme rows at both temperature zero and temperature one. A lighter 82 GB GLM pack reports 32 to 39. I compared the readme speed table with

8:21 the Mimo card, and the table uses a setting the card advises against. The table is measured at temperature zero, which is greedy decoding, always taking the single most likely token. The card

8:32 says greedy decoding is not recommended, and the readme says greedy output under speculation isn't token identical to plain decode. A stopwatch is happy with greedy the model card isn't. So, treat

8:44 Mimo's 32 to 44 with speculation as greedy setting speeds, and GLM's temperature one rows as the closer match to how you'd actually run it. Reading your prompt, which the readme calls

8:54 prefill, runs at roughly 550 to 610 tokens a second in the table rows, and it's a separate number from writing. Early requests run slower, about 200 to 240 for GLM, until the engine tunes its

9:07 GPU code. What else could you do on this machine? The readme's llama.cpp row on a smaller 1.56-bit quant writes 16.74 tokens a second and drops to 7.19 at 64,000 context. Different format, so it

9:23 isn't like-for-like. So, suspect three is out. The speed claim is well documented by the developers on one machine. 26 to 44 tokens a second is a long way

9:33 from a crawl. That leaves the answer, three levers: bits per weight, sparse experts, memory bandwidth. For GLM, about two to three bits instead of 16 turns 640 GB into about 100. For Mimo,

9:47 eight of 256 experts means about 15 billion parameters of work per token. Then memory bandwidth, one shared pool at about 256 GB per second feeding them, which is where the earlier ceiling came

9:59 from. Speculation then squeezes a bit more out of the same reads. Take any one away and it breaks. Without the bits, you'd need 640 GB, which is a lot of money for a mini PC. Without the

10:11 experts, reading all 105 GB for every token would cap you near two and a half tokens a second by the same division. Without the fast shared pool, the ceiling drops with the memory speed.

10:22 That answers the first half of the opening question. Divide 640 by 100 and the files are about a sixth of their 16-bit size and each token reads only a sliver of them.

10:32 The second half bugged me more. After a squeeze this hard, what's left of the original model? That's suspect four. It's garbage at two bits. The evidence is agreement with the official

10:42 FP8 weights. FP8 means eight bits per weight, half the 16-bit size from the start, and it's the reference both cards use. Top one agreement is how often the squeeze model's most likely next token

10:53 matches the full model's. Mimo's card reports 91.99%. GLM's card is lower. It reports 89.31% on a 30-row sample, 90.11% on the 126 longest of 129 rows, and 88.74% on four

11:09 long rows. Mimo's French text agrees 87.37% of the time. Subtract those from 100 and the squeeze models pick a different top token about 8 to 11% of the time. That

11:20 measures disagreement on the next token, not how many answers are wrong. A different top token could be a synonym or a wrong fact, and this number can't say which. I compared the Mimo card's

11:30 125 held out rows with the readmes 129 and the counts don't match. It's a small gap, but those rows are the only quality numbers for these exact packs. What's missing is task scores for these exact

11:42 packs. GLM's card says it plainly, task scores of this exact pack were not run. Reasoning mode wasn't part of any task test and no same size GGUF quality comparison was run. The repo does have

11:54 task numbers, but they belong to an earlier smaller Nemo pack on different hardware, not the 105 GB one. So there's no evidence yet on whether this beats other two-bit approaches or only fits.

12:05 Next, what's open? The engine and serving scripts are public and so are the weights. The quantization pipeline that made the packs is not. Nemo's card says conversion tooling is not published

12:16 and that only GFX 1151 with ROCm is tested. The repo's convert script comes from upstream exllama 3 and whether it reproduces these packs isn't established. The repo itself is brand

12:28 new, created October 2nd, last pushed October 3rd under the MIT license. A repo this new has no track record yet and the star count, a handful at last check, shows little outside attention so

12:40 far. Now the money. One price aggregator lists 128 GB Ryzen AI Max plus 395 boxes from $3,324, a BAZ game M5 up to $4,349. Prices move, so check the day you buy.

12:56 The other box on the shelf is Nvidia's new DGX Spark, which is $4,999 for 64 GB. A 100 GB pack can't fit in 64 GB and no amount of optimism changes the arithmetic. The 128 GB Spark is listed

13:13 at $6,950 on secondary pages. Nemo's card says only AMD's GFX 1151 with ROCm is tested, so these packs aren't shown running on a Spark. My call, by viewer. If you

13:26 already own a 128 GB Strix Halo machine, I'd try it with your own prompts and sampled settings, not the greedy ones. If you'd buy a box for this, I'd wait for task scores.

13:38 If you need to reproduce the quantization, the pipeline behind these packs isn't published, and whether the upstream converter gets the same result isn't established.

13:46 So, back to 640 GB in a 128 GB machine. The speed claim is well documented on one machine by the developers. The quality claim is half measured, roughly 89 to 92% top token agreement, and no

14:00 task scores for these exact packs. When someone runs coding and reasoning benchmarks on them, does that gap turn into a small dip or a big one? If you try one, share what you find.