Random Attention: They Deleted AI Memory at Random (And It Got 43% Faster)

0:00 Random attention reports more server throughput from random deletion, up to 43%. Your question stays protected while the model works. Why can its other working notes go? Those notes live in

0:12 the key value cache, the model's temporary record of what it has already processed. Deleting these entries changes what the model can look back at. It's trained weights stay untouched. The

0:23 researchers compare random deletion with a method that carefully decides which entries deserve to stay. Across their serving tests, random delivers 32 to 43% more output per second. We'll follow one

0:36 question through the cache, see what survives, and find where that speed comes from. Think of your question as the problem sheet. The model's reasoning is the working paper beside it. Keeping

0:47 those working notes nearby saves the model from rebuilding earlier keys and values each step. But a long solution fills the desk. A server answering many people has to make room for all those

0:59 desks. The September 2026 paper comes from Salesforce and University of Illinois researchers. Their public code is inspectable, but it isn't independent replication. I treat the headline as a

1:11 reason to investigate. You could score each note. An attention weight measures how strongly the current step reads an earlier entry. SnapKV uses recent attention to estimate importance, but

1:23 calculating that estimate takes time. You're paying to decide what to forget. Some older policies skip important scoring, too. Streaming LLM keeps initial positions and recent entries.

1:35 H2O accumulates attention, but isn't a main baseline here. The paper compares SnapKV, RKv, VaS, and Tri Attention at matched budgets. Let's follow an illustrative request using the paper's

1:48 engine settings. Your question fills 200 tokens, which means pieces of processed text. A head is one of the model's parallel attention lanes. Each key value head has a persistent

2:00 budget of 1,024 entries. Each layer creates keys and values from your question. Think of a key as an address label the next query can match. Its value carries information the attention

2:12 operation can combine. The cache stores numerical representations. Our picture follows the tokens that produce them. First, reserve the questions entries including system instructions and chat

2:23 formatting. 200 protected leave 824 persistent slots for the generated reasoning. Your question doesn't have to win the lottery. Then the model writes more

2:34 reasoning. The engine protects a separate buffer of the newest 64 entries. After another group arrives, the older buffer joins the eviction candidates.

2:43 The newest buffer stays outside that draw. In our example, 888 reasoning entries compete for 824 slots. 64 must go. Each key value head makes its own random choice.

2:57 The engine keeps the selected entries in their original order and packs the cache together. That head now holds 1,088 entries including its recent buffer. It continues generating from what remains.

3:10 A deleted entry doesn't reappear on the next draw. Older entries face repeated lotteries so surviving history thins with age. The paper uses random numbers followed by

3:20 top key which means picking the highest values. That's still computation. It skips calculating content importance. So that's the deletion rule. Its value depends on what your model loses when it

3:33 runs. The study uses four checkpoints from the 2025 generation. Three queen three sizes and five four reasoning. Six tasks cover math, science, and code. The cache budgets vary by task. Code gets

3:47 3,072 persistent entries per head. These are the studies checkpoints not current model recommendations. On the smaller Quinn model's math test, the authors report 93.9%

3:59 with full attention. Random retention scores 87.4. You lose 6 and 1/2 percentage points. I'd set your acceptable accuracy loss before comparing speed. Would you trust

4:10 random deletion if your original instructions could be thrown away, too? The authors separate that risk from forgetting working notes. They run the same policies with the whole prompt

4:20 protected, then measure what that change buys. On that math test, unprotected random retention scores 45.9%. Protect the prompt and it reaches 87.4. You stopped it from deleting the

4:34 question. I'd want this control before crediting a selector's importance judgment. The fix helps scored methods, too. On 5/4 science test, SnapKV rises from 44.2 to 66.7%.

4:48 The original comparison partly measured which method happened to preserve the instructions. Your question survives. Now follow the working state. Suppose a subtotal appears in an equation, then

5:00 gets restated while checking the answer. Those later steps carry the information forward. A newer representation can remain useful after an older entry vanishes. Different heads stored

5:11 different representations of a token. Think of people taking notes from the same lesson, each noticing different details. They aren't interchangeable backups, but useful information can

5:21 remain spread across the group after individual copies disappear. The authors plant a variable with the value 4,729, then control which heads keep it. Their

5:33 best single head retrieves it in 3% of trials. Two selected heads together reach 60%. This probe deliberately controls retention, but on real math traces, making every head keep identical

5:45 positions barely changes accuracy. The authors report 87.1% instead of 87.4. You shouldn't credit the whole result to head diversity. Repeated text already

5:57 helps. A separate August paper, prefix sliding, preserves the input in a recent window while discarding older reasoning. That independently supports simpler memory policies. It doesn't reproduce

6:10 random attention speed numbers. Both question how much working history you need. Give the model something it won't restate and the advantage can reverse. The authors announce a passcode once and

6:22 ask after 57 compression rounds. Random retrieval is zero. RKV retrieves it 83.6% of the time. Your subtotal has repeated opportunities to survive. A unique identifier from an

6:35 earlier tool response may not. Even in the main grid, try attention wins one code comparison on the largest Qwen checkpoint. The passcode result gives you a failure case to test. Protecting

6:46 the initial prompt can't guarantee every important later fact survives. So that's the quality trade-off. The speed comparison uses vLLM, software that batches requests and manages cached

6:58 memory. Both selectors share the same serving integration on one Nvidia H200. The authors measure their combined output. For 54 reasoning, try attention produces

7:09 1,212 output tokens per second. Random attention produces 1, 737. Subtract the first rate, then divide by it. About 43% more. The server receives

7:22 128 requests, each generating roughly 32,000 tokens from a 1,000 token prompt. The cache budget is 2,048. Smaller caches help both methods keep more requests in flight. Something else

7:37 separates them. Compression happens between batch decoding steps. The server pauses the batch, processes the eviction, then continues. Try attention reads candidate keys to compute its

7:48 ranking before compaction. Random selection skips that extra content pass. The other requests wait. The paper describes roughly 62,000 compression events in a long output

8:00 workload. Picture every desk waiting while one desk sorts its notes. Removing that work lets the room resume sooner. I'd measure that shared waiting before predicting

8:10 server capacity. For one request, the paper finds the methods within about 1% of each other. Your private coding session doesn't inherit the headline. Shorter outputs also change the

8:21 comparison with full attention. So, that's the speed. It comes from this workload's shared waiting. You can inspect the engine and evaluation scripts on GitHub. The serving

8:31 instructions pin a specific v l l m version and warn about preemption corrupting state. Preemption means suspending a request to free cache space, then resuming it. That transition

8:42 needs correct book keep. The runs limit concurrency to avoid that failure. The largest Qwen checkpoint is capped at 96 simultaneous requests. You should validate this research implementation

8:53 with matched budgets and prompt protection. Check answer accuracy, including facts introduced midway through a task. My winner is random attention as the first baseline to test

9:04 for serving long reasoning under a memory budget. It's reported throughput advantage survives a strong comparator. The pass code failure limits its scope. Before paying for clever ranking, make

9:16 it beat protected random retention on your work. The lottery works because your problem sheet stays, and useful working notes often have another copy. For rare facts introduced halfway

9:27 through, would you protect them explicitly or pay to rank the entire cache?