Enjoying this issue?
Get tomorrow's AI & engineering digest in your inbox — hand-picked, summarized, and always spam-free.
TLDR
The claimed 15-25% speed advantage for Linux over Windows in local AI is misleading: when models fit in VRAM, generation is only 3-5% faster on Linux, but prompt processing can be up to 21.7% faster. The real issue is Windows' WDDM driver overhead and the fallback to system RAM when VRAM is exceeded, which can cause severe slowdowns.
Key points
A startup article claims Linux runs local AI 15-25% faster than Windows, but benchmarks show only 3-5% difference for generation when models fit in VRAM.
On an RTX 5090, Llama.cpp developer measured 419 tokens per second against a theoretical limit of 520, suggesting OS overhead accounts for about 20% of the gap.
Windows' WDDM driver adds 5-20 microseconds per GPU job launch compared to Linux, but this overhead does not significantly affect generation speed.
When models exceed VRAM, Windows spills into system memory via SISME fallback, causing 3-10x slowdowns, while Linux errors out.
Prompt processing on Linux can be up to 21.7% faster than Windows, a larger difference than generation.
Tools mentioned
Stop scrolling. Start reading smarter.
Receive the day's most important AI & engineering updates in one concise email. No spam.
Transcript (captions)
A startup fortune article says Linux runs local AI 15 to 25% faster than Windows. A Reddit table on the same RTX 5080 models fully on the GPU says 3 to 5% for generation. That article lists a
70 billion model at 128 tokens a second on a 16 GB card. That files bigger than the card, so reading it at 128 a second is quite a feat. Generation is memory bound. Every new token reads the model's
weights out of VRAM. On an RTX 5090, Llama CPP developer AM17 measured 419 tokens a second against a limit near 520. If it fits, Windows mostly isn't the problem. Picture a
dock. Weights are cargo. Bandwidth is its width. Each token needs it all through. Divide 419 by 520 and you get about 80%. Which suggests everything else, the OS included, shares the last
fifth. The picture breaks because the cargo never leaves the card. The suspect is WDDDM, the Windows driver model. An Nvidia engineer said in 2018 that under it, each GPU job launch takes a wobbly 5
to 20 microsconds more than on Linux. That's general guidance, not Llama CPP. If it hurt, generation would show it. Reddit user Okay, Mine 189 ran Llama CPPB 8,929
on an RTX5080. Windows to Linux Gemma 4 running 111.7 to 116.7 tokens a second. The 20 billion GPT OSS going from 195.8 to 206.2. The 27 billion Ken 3.6 going from 43.8
to 46.0. Call it 4.5 to 5.3% on these three on one self-reported machine. Since driver 5 36.40, Windows apps out of RAM spill into
system memory, Linux just errors out. Every token rereads all the weights. A 4090s RAM moves 1,08 GB a second. Its PCI Express slot about 32, so a spilled slice drags each step. Users report
several times to 10 times slower. Prefer no CIS me fallback trades that for a crash. Fit the model in RAM. Turn off SISME fallback. Check your backend in Llama CPP build. Vulcan swung both ways
and one slowdown was a build flag. Try a live USB Linux without installing before wiping anything. WSL2 still runs through Windows driver. If your model fits in VRAM, single digits on generation isn't
worth wiping a Windows gaming PC. Prompt processing ran wider up to 21.7%. When Linux is called 25% faster, does the model fit?