Is 24GB Enough for Local Al?

0:00 On the same laptop, a 40-billion-parameter model has a speed ceiling more than 10 times higher than a 27-billion one. But on a 24-GB machine, the faster model may not even fit. The

0:11 40-billion one is Flint at about 200 tokens a second on a MacBook Pro with an M5 Pro chip. It needs about 24 GB. The 27-billion one is the 27-billion Qwen 3.8 at about 18.

0:25 Both numbers are upper limits from arithmetic, not timed runs. Now, open LM Studio or Hugging Face and look at the labels. You see 27B, 2.5 billion active, Q4KM, 262K context, and MTP. None of

0:41 them tells you if a model fits in your 24 GB. They don't tell you why a bigger model can write faster. They also don't tell you why a chat gets slow after an hour.

0:50 Each of those labels feeds one short sum that you can do yourself. At the end, that sum will pick a model for your machine. Every speed sum here uses the same machine, a MacBook Pro with an M5

1:01 Pro chip. Apple's spec page lists its memory bandwidth at 307 GB a second. Start with the unit of work, the token. A model reads and writes text in tokens, which are pieces of words. OpenAI's rule

1:14 for English is about four characters or 3/4 of a word per token. The ratio changes with the language and the tokenizer. A 262K context is about 262,000 tokens, or roughly 196,000

1:28 English words. Next comes the parameter count. A parameter is one stored number in the model, and 27B means 27 billion of them. Quantization stores each parameter with

1:38 fewer bits. The file size follows one sum, parameters times bits per parameter divided by eight to get bytes. llama.cpp is a program that runs models on your

1:49 machine. It shows this sum in a table for the 8-billion Llama 3.1. At 16 bits, the file is 14.96 GB. At Q8_0, it's 7.95 and at Q4KM, it's 4.58. In the same

2:04 table, Q4KM averages 4.89 bits per parameter, not four. The table doesn't say where the extra bits go, so use 4.89 in every size sum. For Llama 8 billion * 4.89 / 8 is about 4.9 GB. The table

2:21 shows 4.58 because llama.cpp counts in binary GB. A binary GB is about 7% bigger than a normal one. For the 27 billion Qwen 3.8, the same sum gives about 16.5 GB. The same table also

2:37 measures speed. The 8 billion Llama 3.1 writes 29 tokens a second at 16 bits, 51 at Q80 and 72 at Q4KM. The file got more than three times smaller and writing got about two and a

2:51 half times faster. But reading the prompt didn't speed up. It went from 923 to 865 to 822 tokens a second. The readme doesn't name the machine, so trust the ratios, not the exact speeds.

3:05 Before you see why, one more thing uses memory, the KV cache. For every token in your chat, each attention layer keeps a key and a value, the K and the V. These are short list of numbers that later

3:17 tokens use to look back at earlier ones. A layer splits its keys and values into a few KV heads and the head dimension counts the numbers in each head. Per token, the cache costs two times the

3:28 attention layers times the KV heads times the head dimension times two bytes. The first two counts the K and the V and the last two is the bytes for each 16

3:37 bit number. Take the 32 billion Qwen 3. It's config lists 64 attention layers, eight KV heads and a head dimension of 128. That's 256 KB per token. So 128,000

3:52 tokens need 32 GB. That's more than 24 GB for the KV cache alone with no model loaded. The configs native limit is about 41,000 tokens. So, the 128,000 figure assumes YARN, a

4:06 method that stretches the context past that limit. The 27 billion Qwen 3.8 also has 64 layers, but only 16 of them keep a growing KB cache. The other 48 are gated Delta Net layers, which keep a

4:18 fixed-size state instead. With 4 KB heads and a head dimension of 256, those 16 layers cost 64 KB per token. At 128,000 tokens, that's 8 GB,

4:31 a quarter of the 32 billion Qwen 3. Neither model card states these cache sizes. They're 16-bit arithmetic from each model's published settings, and they leave out the Delta Net state

4:42 because the card doesn't give its size. Back in the speed table, writing and reading behave differently because a model works in two phases. First, it reads your whole prompt and processes

4:51 all those tokens in parallel. Then, it writes the reply, one token at a time. Engineers call the reading phase prefill, and the writing phase decode. For each new token in decode, the model

5:01 loads its parameters from memory again. In systems like this, decode speed usually depends on memory speed, not on math speed. Deep Seek runs the two phases on separate machines.

5:12 Its write-up, with statistics from February 2025 for V3 and R1, puts prefill on four nodes and decode on 18. Each H800 server node read about 73.7 thousand input tokens a second during

5:25 prefill. During decode, each node wrote only about 14.8 thousand. So, a node reads about five times faster than it writes. This is why memory bandwidth sets your writing speed.

5:36 Picture the model as a full glass and the memory bus, the connection between the memory and the chip, as a straw. For every token, the whole drink goes through the straw once, so the ceiling

5:46 is bandwidth divided by the bytes read per token. For the 4.9 GB Llama file, the M5 Pro ceiling is 307 divided by 4.9, about 62 tokens a second. The table measured 72,

5:58 so its machine was faster. Turn the sum around. 4.58 binary GB * 72 means that machine moved at least 354 GB a second. For the 27 billion Qwen 3.8 at 16.5 GB, the same sum gives a ceiling of about

6:15 18. These are upper limits because real decode also reads the KD cache and never reaches full bandwidth. The M5 Max at 460 or 614 GB a second raises every ceiling. The M6 at up to 170 lowers

6:31 them. That explains the speed table. The Q4 KM file sent fewer bytes through the straw. The straw picture has two weak points. First, prefill doesn't depend on the

6:41 straw. It sends many prompt tokens through one load of the parameters, so the math speed limits it. That's why reading barely changed in the table. Second, some models send only part of

6:51 the drink through the straw for each token. Those models answer the opening question about the 40 billion model. Recursive released Flint, a mixture of experts model, on the 21st of October,

7:01 2026, under Apache 2.0. Models like this usually hold many small groups of parameters called experts in their layers, and a router picks a few for each token.

7:12 Flint has 40 billion parameters in total, but each token uses only about 2.5 billion. On model pages, a label like active 2.5 billion usually marks that active count. All 40 billion must

7:24 sit in memory, and LM Studio says Flint needs about 24 GB. But each token reads only about 2.5 billion parameters, which is about 1.5 GB at 4.89 bits. 307 / 1.5 gives a

7:38 ceiling of about 200 tokens a second. That sum ignores expert routing and the shared layers that every token uses. Reports of over 125 tokens a second on an M4 Pro come from third parties, and

7:50 they aren't independently checked. Qwen 3.8 Flash Next shows the same split at a larger size. Its card lists 125 billion parameters with 6 billion activated from 512 experts. Each token uses 10 experts

8:04 that the router picks, plus one shared expert that every token uses. At 4.89 bits, the 125 billion alone need about 76 gigabytes, and the card adds 51 billion more parameters in a separate

8:17 N-gram embedding table. But, each token reads about 6 billion, so the M5 Pro ceiling is about 84 tokens a second. Storage follows the total, and speed follows the active count. One more

8:27 method makes each trip through the straw produce more text. A small draft model guesses several tokens ahead, then the big target model checks all the guesses in one pass. If the target keeps some

8:38 guesses, one load of its parameters gives several tokens. AMD's documentation for DeepSeek V4.1 DSpark says the draft proposes five tokens per cycle. The draft uses fewer experts than

8:50 the target. It uses 128 experts with the top three active, and the target uses 384 with the top six. The DSpark paper reports 60 to 85% faster generation per user in DeepSeek V4. DeepSeek's

9:04 published numbers don't say how many of the five guesses the target usually keeps. In an earlier report, DeepSeek V3 kept 85 to 90% of its second predicted tokens

9:15 for 1.8 times the speed. On FlashNext card, the line 4 billion MTP list parameters for multi-token prediction, the same guess ahead idea. This also explains the slow chat. Each new token

9:27 reads the parameters and the whole KV cache. Take the 32 billion Qwen 3 at Q4KM, about 19.6 gigabytes on the M5 Pro. With an empty chat, the ceiling is about 16 tokens a second. At 128,000

9:43 tokens, the 32 gigabyte cache joins every read, and the ceiling falls to about six. That's also about 54 gigabytes of memory, so a 24 gigabyte machine can't hold it. The 27 billion

9:55 Qwen 3.8 needs about 25 gigabytes at the same length, so it also needs more than 24. But, its ceiling only falls from about 18 to about 12 because its cache is only 8 gigabytes. So, the cache size

10:07 per token decides how well a long chat keeps its speed. For your own machine, multiply the total parameters by the bits per parameter, 4.89 for Q4KM. Divide that by eight to get the file

10:19 size in bytes. Add the KV cache for the context length that you actually use. If the total doesn't fit with room for your other apps, pick a smaller model or a shorter context. For speed, find the

10:31 bytes per token. Active parameters times bits divided by eight plus the KV cache. Divide your bandwidth by those bytes for your best case writing speed. On a 24 GB machine, the better choice is the 27

10:43 billion Qwen 3.8 at Q4KM. Flint alone needs about 24 GB, so it leaves almost no memory for your other apps. At about 32,000 tokens of context, the Qwen model needs about 16.5 + 2 GB with

10:58 a ceiling near 17 tokens a second. If you have 48 GB or more, try Flint first because its ceiling is about 200. Flash next needs well over 76 GB. The open question is how close Flint gets to that

11:11 200 on a real laptop.