Enjoying this issue?
Get tomorrow's AI & engineering digest in your inbox — hand-picked, summarized, and always spam-free.
TLDR
CPU can be faster than GPU for token generation in AI inference, with Xenolith Engine achieving 18.4 tokens per second on an Intel laptop versus 12 on the GPU for a mixture-of-experts model. The key insight is splitting prompt processing (GPU) and token generation (CPU) to exploit memory bandwidth differences, but results are limited to one setup and model type.
Key points
Xenolith Engine writes 18.39 tokens per second on CPU versus 11.96 on GPU.
The approach splits prompt processing to GPU and token generation to CPU.
llama.cpp CPU mode (14.66 tok/s) also outperforms its GPU mode (11.96).
The model is a mixture of experts reading only 3.8B of 25.2B weights per token.
Results are from one HP laptop with no independent verification.
Tools mentioned
Techniques
- compute-bound vs memory-bound inference
- split inference between CPU and GPU
Stop scrolling. Start reading smarter.
Receive the day's most important AI & engineering updates in one concise email. No spam.
Transcript (captions)
Zenolith readme clocks at writing AI answers at 18.4 tokens a second on one Intel laptop, while llama.cpp on the graphics chip manages 12. The faster one writes with the CPU.
Running a model is two jobs. Reading your prompt is one big batch of maths, which Uppsala University researchers call compute bound, and a GPU swallows that like a sorting machine eating a
whole page. Writing is different. Tokens, the word-size chunks the model writes, come out one at a time, and each one means fetching the weights it needs from memory again. That weight on memory
is what memory bound means. One courier running back to the warehouse for every word. The obvious move is giving the GPU both jobs. But on this laptop, the developer says the CPU can use more
memory bandwidth, and his explanation is for some reason, which is at least brief. So, Zenolith splits the work. The GPU swallows the prompt in one gulp, then hands off to the CPU courier, who
writes the reply word by word. Same warehouse, different road. Here's the developer's own table. Writing speed in tokens a second, medians of three runs. Zenolith, 18.39. llama.cpp on the CPU,
14.66. llama.cpp on the graphics chip, 11.96. Even llama.cpp's own CPU mode writes faster than its GPU mode. Divide one by the other, and Zenolith comes out about
54% faster than the graphics chip run, and about 25% faster than llama.cpp's CPU mode. The readme doesn't say why Zenolith CPU writing beats llama.cpp's. One caveat. It's one HP laptop, one
model, no independent check. That model is a mixture of experts, built from many small expert blocks where only a few work on each word. So, per Google's model card, each word reads about 3.8
billion of its 25.2 billion weights. A dense model would read far more, and the result could look very different. So, if you got an Intel laptop with integrated graphics, it's worth timing both chips
at writing. On this one laptop with this one model, the CPU won 18.4 against 12. Why the GPUs road is slower is still an open question, and the developers' best answer is for some reason.