Enjoying this issue?
Get tomorrow's AI & engineering digest in your inbox — hand-picked, summarized, and always spam-free.
TLDR
A builder is running Kimi K2, a 2.8 trillion parameter model, on 16 Nvidia DGX Spark boxes at about 21 tokens per second — far below the 75 tokens per second the memory bandwidth suggests — because inter-box network links bottleneck every layer's parallel computation. The setup draws 2.3 kW on 3.8 kW rated power, costs roughly $111,200, and achieves 21 tokens per second only at short context with speculative decoding, which works well for code but drops to 5-9 tokens per second for chat.
Key points
A builder self-reports running Kimi K2 on 16 Nvidia DGX Spark boxes at 21.13 tokens per second.
Memory bandwidth allows up to 75 tokens per second, but the inter-box network is the bottleneck.
RDMA (writes into each other's memory) achieved 8.9 tokens per second versus 1.3 over TCP.
Fixing a hard-coded ring setting improved another user's speed to almost 18 tokens per second.
Four boxes running Qwen 3.5 397B reached about 17 tokens per second in a level one text test.
Tools mentioned
Techniques
- speculative decoding
- tensor parallelism
- MXFP4 4-bit quantization
- RDMA
- ring setting
Stop scrolling. Start reading smarter.
Receive the day's most important AI & engineering updates in one concise email. No spam.
Transcript (captions)
One builder runs Kim K3, a 2.8 trillion parameter model on 16 Nvidia DGX spark boxes drawing about 2.3 kW self-reported. On Reddit, the builder says their old GPU rig blew the fuses.
Memory sets the box count, the network sets the speed, and the power circuit caps the draw. My math says memory speed allows about 75 tokens per second, but the builder reports 21 at short context.
What slows it down? The Kim K3 download is about 1.56 terabytes in MXFP4, a 4-bit format. One box holds 128 GB, so you need at least 13 boxes. The KV cache and the operating system need room, too.
So, the builder use 16. Each token uses 16 of 896 experts, which are separate blocks of the network. Together, they activate about 104 billion parameters. At about 4.5 bits each, that's roughly
58 GB red per token. 16 boxes at 273 GB per second give 4.4 terabytes per second. So my math cap speed near 75 tokens per second. VLM's docs say tensor parallelism gives every box a slice of
every layer. So boxes usually swap partial results at least once per layer and Kim K3 has 93 layers. Each box has a 12.5 GB per second network link against 273 GB per second of memory. Picture 16
rooms passing notes through narrow doors. It breaks because a layer ends only with the slowest swap. Nobody has profiled the time, so the network is likely not proven. On an earlier build,
drop the beat measured 1.3 tokens per second over TCP and 8.9 over RDMA, where boxes write into each other's memory, self-reported. Fixing a hard-coded ring setting, the order boxes pass data gave
another user almost 18. The V5 build reached 21.13 at 100,000 tokens of context using speculative decoding which guesses ahead and lets the model check. It helped only coding. Chat fell to 5
to9. The power supplies are rated for 3.8 kW in total, but the builder measured 2.3. At $6,950 each, 16 boxes cost about $111,200 before the switch. The builder may have
paid less. For most viewers, a smaller model is the better choice. In a level one text test, four boxes ran Quinn 3.5 397B at about 17 tokens per second. A second
switch raised prefill, the speed of reading your prompt, from 750 to 910. Will decode rise too?