Don’t Turn On llama.cpp’s New 2x Setting Until You Watch this

0:00 The last row is what stopped me. In the author's own table for this llama.cpp update, one setting makes a mixture of experts model about twice as fast. In the last row, it makes it slower.

0:12 If your model's too big for your graphics card, this setting is aimed at you. Which of those two rows you land in decides whether switching it on helps or slows you down. On November 7th, the

0:21 change merged. Before it, an expert missing from video memory got copied over while the GPU waited. Now your CPU can compute some of those misses right where they sit in system

0:32 RAM, and a new script picks the settings. Treat this as part two of our October expert cache story. What I wanted from the pull request, its comments, the flag docs, and the tuner

0:42 source was one answer. Whether this helps cards like yours or only one machine. One video on October's cache going by its search listing called it four times faster. The original pull

0:53 request did measure a bit over twice on an RTX 5090, but that's one machine, and so is this update's headline. Among the reviewers, P. Wilkins saw a gain of only about 10% on their own card.

1:06 So by the end, you'll know why the same setting nearly doubles one run and slows another on the same computer. What the flag does and what to set on yours. Start with what a token needs.

1:16 A mixture of experts model splits its layers into many small subnetworks called experts, and a router picks a few for each token. Quen's model card for the 30 billion Quen 3 shows the scale.

1:27 It switches on only eight of its 128 experts per token. Most of the model sits unused on any single token, which is why a model too big for your card can keep its experts

1:38 in system RAM. Picture your GPU as a desk with room for a few manuals and RAM as an archive down the hall holding all of them. October's cache gave the desk a fixed set of

1:48 slots. In the docs, it's a video memory budget for experts that live in CPU memory. When a token needs a manual that's already on the desk, that's a hit. When the desk is full, the one used

1:59 longest go makes room. Least recently used in the jargon, misses are the slow part. Each missing manual gets carried down the hall over the PCI Express link your

2:08 graphics card plugs into while the GPU waits. Back in September, before that cache existed, a developer called Memoria Rue had already named the problem.

2:18 The same expert weights kept crossing that link over and over again, so he proposed a pool of slots to keep them. When hits are common, the cache shines. Sam Rose put it on a free 16 gig Tesla

2:29 T4 with a model too big for the card, and nearly every expert lookup came straight from video memory. Once it served several requests at once, that hit rate fell a long way.

2:39 On the author's desktop, the hall moved about 10 gigabytes a second, and the poll request doesn't list that machine's hardware. Every miss waited on that trip.

2:48 Meanwhile, the CPU, standing in the archive right next to every expert, was mostly holding the door for the cart. So AM 17 AN, who ported October's cache, added a second path. Once the cache is

3:00 full, a share of the misses skips the trip. In the archive, the CPU does that expert's math from its own copy, and the GPU's part and the CPU's part get added together. Speed is why that trade pays

3:11 off. On the same desktop, the CPU read experts from RAM about five times faster than the link uploaded them. If you divide one speed by the other, the

3:20 tuner's timing formula is built from those two costs too, upload volume and CPU expert volume. The layer gets built on both chips. On the GPU side, the layer runs from the

3:31 experts on the desk, and any expert handed to the CPU reads a zero there. Over in RAM, the CPU runs those experts from its own copies. Both outputs are added together before the layer

3:42 finishes. Results get handed over in the background, so the CPU doesn't have to wait for the GPU to finish and both work at once.

3:50 There's a condition, though. Every slot in a section of the cache has to hold an expert before the CPU share kicks in. Until the desk is full, misses still make the trip. My desk picture breaks

4:00 here. In an office, you'd wait for the missing pages before working through a binder. Your GPU doesn't. It runs the full layer with blanks where the CPU's experts go, and the CPU's work arrives

4:11 as a separate sum. paths then. stay on the GPU, misses still get uploaded, and the rest are computed by the CPU from RAM with

4:19 both answers added up. What's left is deciding how much to send. You set that amount with a new flag, dash dash m o e cache cpu share. The fraction of misses the CPU computes

4:30 instead of uploading. Which misses go to the CPU, the poll request doesn't spell out. It's default is zero, so the headline feature ships doing exactly nothing until you ask for it. You'll

4:41 need release B 12 191 or later, and the author disclosed it was built with Claude. Now back to the author's table from the opening, run on a Quen model with a small cache and the share off, it

4:53 decoded about 24 tokens a second. Push the share near full, so almost every miss goes to the CPU, and the same cache gave about 48. There's your headline. One odd detail, the hit rate went down,

5:05 and the only reason on record is that fewer misses were getting uploaded into the cache. One desktop though, so the comment thread matters more for you. Marcel Dev 89 on an 8 gig RTX 3070 ran

5:17 the tuner, going from share off to nearly full, call it one and a half times faster. Michael Diesel's 2080 Ti works out close to that too from the before and after

5:27 posted in the thread. Marcel Dev 89 also posted a baseline with no cache at all. Set his two numbers side by side, and the cache plus the CPU share comes closer to one and three quarters. On

5:38 small cards like these, I'd plan on half again, not double. You'd expect a bigger cache and the CPU path to stack. Then comes that last row. Give the same desktop a much bigger cache, and with

5:50 the share off, it runs around 65 tokens a second. Push the share near full, and it drops to about 54. Back to the question from the start. My reading, with a small cache, plenty of misses are

6:01 left, and taking them off the link saves a lot. With a big cache, fewer left. On top of that, the tuner's formula charges a flat cost whenever the share is on at all. When the misses run out, that fixed

6:13 cost is what remains. There's a second cost, too. A high share means fewer misses get uploaded into the cache. In the author's small cache run, that showed up as a lower hit rate.

6:23 For that desktop, the tuner recommended an even bigger cache with the share switched off. The script that ships with the CPU share told this machine not to use it. At least it's not on commission.

6:34 Copying the headline is the trap. Set the share near full because one table showed twice as fast, and on a big cache, you'd be paying for it. Run the tuner or leave the share off. To pick,

6:45 it runs your server at a small grid of settings. There are a couple of cache sizes, one about a tenth of your expert memory and one three times that. Each gets a few shares from off to nearly

6:55 full. It also records which experts your model routes to. From those runs, it fits a timing model. Uploads and CPU work get separate costs plus a flat cost for splitting and a term for overlap.

7:08 Then it replays the recorded routing to predict settings it never ran. Last, it picks the lowest share within 5% of the fastest result. A lean toward less CPU work is written right into the script so

7:19 a big cache can easily come back with the share turned off. If you run bigger batches, say for several users, remember Sam Ross's result. Hits dropped a lot. Since the tuner's own runs may not look

7:30 like your traffic, check its pick under your real load. If your cache holds a small slice of your experts, run the tuner and expect it to push the share up. On the author desktop, three times

7:40 the cache was enough to flip that by my count. So if yours holds a much bigger slice, expect it to say turned off and believe it. From the pull request, the command is Python 3

7:50 scripts/moe_cachetune.py -- then your usual llama server line with -c moe because it measures with real server runs, I'd give it the machine to itself. My call. If your

8:03 model spills into RAM on a small card, expect a solid gain short of doubling. So the last row I read at the start turns out to be the useful part. The CPU share pays off on small caches and the

8:14 full doubling was one desktop. The tuner tells you which side you're on. If you run it, what share did it pick on your card?