Enjoying this issue?
Get tomorrow's AI & engineering digest in your inbox — hand-picked, summarized, and always spam-free.
TLDR
Llama.cpp's MTP sampled drafts deliver 4-9% throughput gains on average, not the 14% claimed elsewhere. The PR uses rejection sampling to make sampled drafts mathematically identical to normal sampling, which helps at higher temperatures where greedy drafts often get wasted. The gains are modest and vary by temperature and task, with code showing larger improvements than prose.
Key points
The merged PR for llama.cpp's MTP shows average throughput gains of 4-9%, not 14%.
The PR uses rejection sampling to make sampled drafts mathematically identical to normal sampling.
At temperature zero, MTP throughput varied from -3.22% to +5.20%, indicating noise.
A downstream project reported greedy MTP drafts at 2.3-2.4x on code but only 1.4-1.5x on prose above zero.
Techniques
- multi-token prediction
- rejection sampling
- speculative decoding
Stop scrolling. Start reading smarter.
Receive the day's most important AI & engineering updates in one concise email. No spam.
Transcript (captions)
Llama CPP's MTP can now run faster at higher temperatures, but not by 14%. In the merged pull request, MTP's average gain is roughly 4 to 9%. MTP, multi-token prediction, has the model
guess its next few tokens, and the full model, the target, checks them in one pass. That guess, the draft, is by default its likeliest token. At temperature zero, the target also takes
its likeliest token, so a good guess lands. Above zero, the target samples, so the draft often gets wasted. So, why would a sample draft do better? Picture a checkpoint. The draft now samples each
token from its own distribution, Q. The target holds its own odds, P, and waves the token through with probability min of 1, P over Q. Sampling the draft sounds like it should loosen the output.
It doesn't. A rejected token gets redrawn from the leftover, where the target wanted more than the draft offered. The PR calls the result mathematically identical to normal
sampling. Rejection sampling is the one checkpoint where getting turned away still hands you a token. Why does that help when the target is flat? A greedy draft survives with probability P of its
top token, so a flat target rejects it often. Worked out from the PR's formula, a sample draft survives with the overlap of P and Q, which hits one when they match, and improves the closer they get.
The PR's tables, from an RTX 5090 and a DGX Spark, put numbers on it. Averaged over temperatures 0.5 to 1.0, MTP throughput rose between 3.78 and 9.18%. The best single cell was +12.91%.
No MTP cell reached 14%. The rows near 13% use a separate draft model. At temperature zero, where outputs are identical, the PR's cells swing from -3.22
to +5.20%. So, a few percent may be noise. The PR doesn't split pros from code. One downstream project reported greedy MTP drafts at 2.3 to 2.4 times on code, but
only 1.4 to 1.5 on pros above zero. That's their number, not llama.cpp's, but it points at pros. If you run chat or writing at 0.6 to 1.0, update and pass spec draft sampling
probabilistic because the default is still greedy. Expect single digits on average. Is pros where those digits show up?