The Free Fix That Doubled AI Speed on Old M1 Macs

0:00 Same 2021 laptop, same model. Quinn 3.8 27B goes from 18.9 tokens a second to 39.1. And what changed is who wrote the matrix maths. That model runs on Splash Inco's new engine for running models

0:15 locally on Apple Silicon. And the official requirement is one short line, Apple M3 or newer. So if you've got an M1 Max or an M2 Max with 64 gigs, you've already got more memory than Inco asks

0:28 for. The chip is what rules you out. So, is your Mac actually too old or was the software just never written for it? A free community port says it's the software. And the neatest part of its

0:39 fix fits in one line of bit maths. This line, a 4-bit number, Oregon ed into hex 6400. It turns a model weight into something the M1's matrix hardware can use without rounding it. Splash is

0:52 Inco's engine, open sourced in midepptember, and on hardware it supports, it's quick. On a 48 gig M5 Pro, Inco measured this model at 74 tokens a second on short prompts and 54

1:04 with 32,000 tokens of context. It also wants Mac OS 26.4 or later. A lot of that speed comes from speculative decoding. Picture two models at one desk. A tiny draft model, just five

1:16 layers and about 1.2 gigs, scribbles down seven tokens as a guess. The big 27B model reads all seven in one pass and keeps the ones it agrees with. Because the big model checks every guess

1:27 token, you get the same output distribution you'd get without the helper. Speculation changes how fast the answer arrives and nothing else. When the draft guesses well, one pass of the

1:38 big model can land several tokens. Hold on to that checking pass. Checking seven tokens at once is a stack of big matrix multiplications, so make those faster and every check gets cheaper. On the

1:49 22nd of September, a developer who goes by PaperNook tried the obvious thing. Splash checks which GPU family it's running on. M1 is Apple 7. M2 is Apple 8. And Splash wants something newer. He

2:02 lowered that gate, and Splash started up on his M1 Max. It ran in the loosest sense. To see what broke, you need the two jobs an engine does. Prefill is reading your prompt, every token at

2:13 once. Decode is writing the reply, a few tokens per step. Decode was the nastier problem. One kernel, linear tile simmed group, corrupted the output on Apple 7 without throwing an error. Speed dropped

2:25 to between 8.6 and 9.8 tokens a second and the text came back with grammar mistakes. Slow and wrong at the same time is a hard combination to sell. Prefill had a different problem. It

2:36 stayed slow and the developers read was that Apple's matrix library MPP probably has no dedicated matrix hardware to run on for this chip. He wrote all of this up as an issue on Inkco's repo. Inco

2:49 closed it as not planned, short and very clear about who wouldn't be writing the M1 kernels. So what's missing on the M1? This part is the port developers explanation, not Apples or Incos.

3:00 Stockplash hands its big matrix multiplications to that MPP library on Apple 7 and Apple 8. He says the library is emulated and it can hang the GPU outright. The second gap, the M1 has no

3:13 Bflat 16 arithmetic. Bflat 16 is a 16- bit format that's common for AI weights. Kernels built around it can't use it natively here, which is why the port needed its own. The numbers show how

3:24 much that costs. Apple rates the M1 Max GPU at 10.4 teraflops in 32-bit floats. On this model's matrix shapes, the stock MP tiles manage 2.4 to 3.5. That's roughly a/4 to a third of what Apple

3:39 rates it at. So most of the GPU was waiting while the maths took the long way round. The fix follows from that. Stop asking the M1 for arithmetic it can't do natively and give it the same

3:49 multiplications in number formats it can. 2 days later on the 24th the port had two new kernels for decode. A version of the old kernel that works in plain 32-bit floats since Bflat isn't

4:01 there. It's called SIM group F-32. For prefill a tile called mm A64. That's where the bit trick lives. So its payoff shows up when the model reads your prompt. MMA64 uses the M1's SIMroup

4:14 matrix multiply. A small group of GPU threads holds a block of the matrix in registers, the fastest storage the GPU has, and multiplies it together, never touching slower memory for that step.

4:25 Those units work with half precision numbers and 32-bit floats. The model's weights, though, are stored as 4bit integers. A 4bit weight is a whole number from 0 to 15. You need to turn

4:37 billions of them into half precision floats quickly without rounding a single one. That's the job of hex 6400. A half precision float is 16 bits. One for the sign, five for the exponent, 10 for the

4:50 fraction. Set those bits to hex 6400 and the value is exactly 1024. Why 1024? At that size, the gap between 1/ half precision number and the next is exactly one. The 10 fraction bits are

5:04 counting whole numbers and the bottom four of them make a slot for anything from 0 to 15. Take a weight of 9. In binary, that's 1 0 0 1 or it into hex 6400 and you get hex 6409 which the GPU

5:17 reads as exactly 1033. That's 1024 + 9 from 1 bit wise O with nothing rounded. Every weight becomes 1024 plus its value. And because that 1024 is the same every time, it can be taken back off

5:31 exactly. What's left is the weight itself unrounded in a format the matrix units handle natively. It's the kind of line a reviewer flags as a typo right up until they work out the float layout.

5:42 The activations, the numbers flowing through the model from layer to layer stay in full 32-bit floats. The developers claim is that every weight times activation product comes out exact

5:52 in 32-bit float. Nothing gets squashed into a smaller format to buy speed and it bought speed anyway. MMA64 runs at 7.3 to 7.4 teraflops on the same shapes where the stock tiles did 2.4 to 3.5.

6:07 Same GPU, two to three times the useful maths. And now about 70% of Apple's rated figure. Now bring back the two models at the desk. Every time the big model checks seven guesses, it's doing

6:18 matrix multiplications. So cheaper maths means cheaper checks. For writing the reply, that job falls to the FP32 decode kernel. MMA64 is the prefill tile, so its big payoff is in reading your

6:30 prompt, and you'll see that in a second. The developer measured the whole package, not each kernel on its own. And together, it roughly doubles the reply speed. The port showed up on r/local llm

6:42 under a headline of 39 tokens a second on a 2021 M1 Max. That number comes from the developer own benchmark, an M1 Max with 32 GPU cores and 64 gigs. Mac OS 26.7 to 27B at temperature zero. Each

6:57 run got a fresh server in ABBA order. So warm-up and ordering effects don't favor either side. Decode averaged over five prompts went from 18.9 tokens a second to 39.1. That's about 2.1 times. The

7:12 technical prompt went from 16.6 to 34.3. And at 32,000 tokens of context, 12.5 to 23.8. Prefill is where you feel it most. A 2048 token prompt went from 53 tokens a second to 142. A 3209

7:31 token prompt went from 48 to 110. At 48, the model spends about 11 minutes reading that prompt before it writes a word. At 110, it's under five. 11 minutes is long enough to regret pasting

7:44 in the whole repo. Then there's parallel work. With four requests running at once, total throughput went from 18.0 0 to 65.2 tokens a second. If you run local agents or batch jobs, I'd watch

7:56 that number more than the headline. Keep two caveats next to all of this. These are the developer own measurements, not an independent test. And the before is stock splash forced onto a chip it

8:06 doesn't support, which needed his fix for that broken decode tile just to produce correct text. The numbers also move with each release. The latest one lists 38.1 tokens a second for this

8:17 model on a 64 gig M1 Max. a touch under the 39.1 headline. Treat 2.1 times as the result for this benchmark rather than a promise for every prompt you'll ever send. For scale, inco M5 Pro figure

8:30 was 74 tokens a second on short prompts. Different machines, different people measuring, so this is rough. A new chip still wins. On these figures, it wins by about two times now instead of nearly

8:42 four. Speed only matters if the answers hold up. The developer's argument goes like this. Every multiplication is exact. And the one thing that changes is the order the numbers get added in.

8:52 Order does matter a little. In 32-bit floats, add 1 to 100 million, subtract 100 million, and you get zero. Subtract first, then add one, and you get one. Addition is associative in maths class

9:05 and negotiable in floating point. So, he measured the drift. across 2528 token positions against the previous M1 build. The model's top choice for the next token matched 99.92% of the time.

9:18 That's roughly two positions out of 2528 picking something different. He also measured KL divergence, which tracks how far the whole spread of probabilities moves across every option rather than

9:29 only the top pick. The average was 1.3 * 10 theus4nats which is tiny and he found no bias in the probability of the token it chose near identical then rather than bit identical and check the comparison

9:43 that's against the earlier M1 build not official splash on an M3 speculative decoding doesn't change the odds of what the model says the new kernels add a rounding size wobble on top what about

9:54 your chip the developer tested on an M1 Max only support for M1 Pro M1 Ultra and M2 rest on user reports and on those chips running the same code paths. One of those users went further. A developer

10:07 called JS Bonds AI built add-on kernels on top of the port. On an M2 Max, decode went from 51.2 to 62.5 tokens a second. On an M1 Pro, 19.4 to 23.6. That's their kernels against the port, not the port

10:24 against stock splash. And they report output bit identical to the port's kernels. a blogger at tinyapps.org ran it on a 2021 MacBook Pro with an M1 Pro and 32 gigs. It got 13.0 tokens a second

10:39 median sampled and 14.5 greedy. Another engine, TensorFold managed 9.7, so it wins, but it's a long way from 39. JS Bonds AI's M1 Pro did 19.4 before their add-on. On the smaller chips, the number

10:54 swings with the machine, the settings, and the prompt. If you want to try it, the newest release is 1.1.0 M1 from the 27th of September. [snorts] It adds gguf model support and splits

11:07 long prompts into chunks. Install is one curl command piped into ssh straight from the release page. Piping a stranger script into your shell is a long tradition. And like most traditions, it

11:17 goes better if you read it first. It installs as splash-m1 next to any homebrew copy of splash. The first pre-built release needed no Xcode, homebrew, or pip. So check the 1.1. Zero

11:29 notes before assuming the same. Memory decides whether this is for you at all. Encode lists 36 gigs or more for the full 27b packages with 48 recommended. JS Bonds AI puts the floor for this

11:41 model at 32. The port's 16 gig figure is only for a much smaller model called Turnary Bons. And the catches, it's an unofficial community port built by one developer who tested it on an M1 Max

11:53 only. Inco closed the first M1 issue as not planned, so don't count on help from Inkco. Releases have been landing days apart and the numbers shift with them. My call then on an M1 Max or M2 Max with

12:07 64 gigs, if you want this model running locally, I'd install Splash-M1 alongside what you use now and check it on your own prompts. On an M1 Pro with 32 gigs, try it, but expect somewhere between the

12:19 low teens and the low 20s, nowhere near 39. On 16 gigs, skip the 27B. And if it has to be supported software for work, stay with official splash on an M3 or newer. For M1 Max owners, the limit

12:33 turned out to be the software. What I'd like to know next is whether Inco brings an M1 path upstream now that someone shown how much speed that chip had left.