Enjoying this issue?
Get tomorrow's AI & engineering digest in your inbox — hand-picked, summarized, and always spam-free.
TLDR
Open BMB claims its 35B parameter Mini CPM V 4.7 runs on an iPhone in 3 GB of memory, but the actual iOS app requires 12 GB of RAM and streams experts from the SSD on demand. The model uses mixture-of-experts, Q3K quantization, and look-ahead expert scheduling to achieve this. The 3 GB figure is undefined and likely refers to the working set size, making the claim misleading.
Key points
Open BMB's Mini CPM V 4.7 selects 8 of 256 experts per layer in its MoE architecture.
The iOS app stores experts in 40 shard files and streams them from SSD on demand.
Look-ahead expert scheduling preloads predicted experts for the next layer to reduce stalls.
Open BMB reports over 13 tokens/second on iPhone 17 Pro Max, combining quantization, scheduling, and a draft model.
The iOS app requires 12 GB of RAM per the readme, contradicting the 3 GB memory claim.
Tools mentioned
Techniques
- mixture-of-experts
- Q3K quantization
- look-ahead expert scheduling
- speculative decoding
Stop scrolling. Start reading smarter.
Receive the day's most important AI & engineering updates in one concise email. No spam.
Transcript (captions)
Open BMB says it's 35 billion parameter mini CPM V 4.7 model runs on an iPhone in 3 GB of memory. A 35 billion parameter model should be tens of gigabytes. The files on that phone total
about 16.6 GB. Both numbers are Open BMB's own, so they'll have to sort this out with each other. Stored at 16 bits, the full model would be about 70 GB per a third-party read. You'd guess
compression, and that's part of it. The iOS build stores its experts in a compressed format called Q3K, which helps explain 70 becoming 16.6. But 16.6 GB still won't fit in 12 GB of RAM, and
Open BMB validated it on a 12 GB iPhone 17 Pro Max. So, where's the rest? On the SSD. Open BMB's iOS app read me says the routed experts sit in 40 shard files and stream in on demand instead of living in
RAM. Why can that work? It's a mixture of experts model. Each layer holds many small experts, and a router picks a few per token. Going by a third-party read of the config, that's eight of 256 per
layer. Different tokens pick different experts, so you can't know the files ahead of time. The catch is stalls. The model is a stack of layers run one after another. Layer N finishes, but the next
layer can't start until its experts arrive from the SSD, so the processor waits. Open BMB's fix, look-ahead expert scheduling, predicts the next layer's experts while layer N is still computing
and starts loading them early. Experts already in memory run first while the rest load alongside. The real router still decides the output, so a wrong guess costs time, not correctness. How
often the guess lands isn't in Open BMB's docs. Open BMB reports more than 13 tokens a second on the iPhone 17 Pro Max. That's its own combined figure, not independently reproduced. Alongside
look-ahead, it uses quantization and a small draft model that guesses tokens for the big one to check. Open DMB's Readme doesn't split that figure between the three. So, the 35 billion parameter
model never sat in 3 GB. Open DMB doesn't say what that figure counts, and its iOS app Readme recommends 12 GB of RAM. With less, the app keeps fewer experts in memory and runs considerably
slower. So, what is that 3 GB actually measuring?