NVIDIA Just Made AI Memory Transferable Between Models (KV Cache Transfer)

7 seconds. That is how long a big model spends reading a long conversation before it writes one new word. Nvidia measured that. Then they did the same job in 278 milliseconds, 25 times faster. And here is the part that should not work. The second model never read the conversation. It borrowed the first model's reading, the notes, the memory, whatever you want to call the KV cache, which was supposed to be impossible. A cache belongs to the model that built it. Nothing else can read it. Nvidia's new paper says that wall is thinner than it looks. On the good pairs, one matrix multiply walks through it. Four of the six model pairs kept between 73 and 98% of the receivers's own accuracy. Two of them fell apart completely. And the reason those two failed is the most useful thing in the paper. So, what is actually inside a KV cache? Why can no other model read it? And what did Nvidia strip out to move it? Three tricks, six pairs, two failures. Start with the 7 seconds because that number is why this work exists. When you send a prompt, the model reads all of it at once in a single pass before it generates anything. That pass is called prefill and its only product is the KV cache. For every token you sent at every layer, the model writes down two vectors. A key, which is a label it will search for later, and a value, which is the content it pulls when that label matches. Generation then runs off those notes. Each new word looks back across the keys, decides what deserves attention, and pulls the matching values. Without the cache, the model would reread your whole conversation for every single word it writes. So, prefill is the expensive half. And it gets worse in two directions at once. Longer prompt, more work, bigger model, more work. An agent 20 turns into a session is dragging an enormous prompt behind it every time it speaks. Which is why prompt caching exists. Anthropic's own pricing page builds a cash read at onetenth of the base input rate. Send the same prefix again, pay 10%. That is the largest standing discount in this business, and teams do strange things to earn it, freezing prefixes bite for bite, so one cash keeps matching. But that discount has a hard edge. The cash only works on the model that produced it. Those keys and values came out of that model's weights at that model's layer count in that model's geometry. to any other model. They are noise and that edge sits exactly where production is heading. Routers now move traffic between models mid-con conversation. A small model for the easy turns, a bigger one when the question gets hard. Cheaper waits again when it softens. The moment traffic moves, the cash is dead. The receiver rereads everything from scratch at full price. Nvidia measured that penalty on their own hardware. 7 seconds for the 32B model at 32,000 tokens. 11 12 seconds for the 70B. Here's where the paper turns. If prefill's only output is the cache, then skipping prefill is not a computation problem. It is a translation problem. Take one model's cache, rewrite it in the format the other one expects. So the first question was whether that translation has any structure worth exploiting or whether it is noise. Take young ho's team at NVIDIA tested it in the dullest way available. Plain linear regression, one source layer, one target layer. Going from QN 314B up to 32B, a single layers linear fit reconstructed 56% of the variance in the big models keys, which means more than half of what a 32 billion parameter model would have written down was already sitting in the 14 billion 1's notes, reachable by multiplying by a matrix. The two models do not have the same number of layers, so there is no natural pairing between them. The fix. For each target layer, rank every source layer by how well it predicts that layer. Then feed the best eight in together. That takes keys to 79% and values to 65. The team's own ablation says this cross layer selection carries more weight than either of the other two tricks. Second trick, and this one is elegant, keys carry a position rotation baked into them. Rope the thing that tells the model where in the sequence a token sat. The same word at position 10 and position 10,000 comes out looking different. So they unwind the rotation before fitting. Learn the map in position free space. then reapply the receivers's own rotation at inference. The fit stops caring about length, which is why a map learned on thousand token samples still works at 32,000. Third, the map itself is ridge regression fitted independently for every layer in every head closed form, a solve, not a training run. The calibration set is 500 web documents of a,000 tokens each. That is the whole input. No gradient descent, no back propagation, about an hour on a single 8GPU node. And the map for that pair is finished. At inference, the mapper replaces the receivers's entire transformer body with one batched matrix multiply per layer. That is the whole reason it is fast. There is almost nothing to it. They tested six pairs across three families. Qin 3, Llama 3.1, and Mistl's minestral 3. Every pair stays inside one family with the small model handing off to the big one and the big one handing back. Best case Q&314B into 32B 97.6% of the receivers's own accuracy averaged across five benchmarks on Hella Swag. It came in at 101% marginally better than the model doing its own reading. The most extreme jump they tried was Llama 3.1 8B handing off to 70B nearly 9 times the parameters. That one still held 73%. The latency table is the part I would pin to the wall. 32,000 tokens small to large, 278 milliseconds for the mapper against 6,975 for reading it again. Across 70 measured cells, seven pairs by 10 context lengths, the mapper won everyone. Modest at short prompts where a fixed 14 millisecond floor dominates. Enormous at long ones because the mapper barely notices length and prefill grows with it. They also check the case. This is actually for 100 coqa conversations roughly 15 turns each models alternating turn by turn drift over 10 turns stayed under two points going up and about a third of a point per turn coming back down. So here's a question worth answering before the paper answers it for you. If your router could skip a 7-second reread but the receiving model kept 97% of its accuracy instead of all of it, would you take that trade? Keep your answer because two of the six pairs did not get 97%. Both ministral pairs feeding the 14B model landed at 44 and 42% average retention. Grade school math shows what that means. 95.6% of the arithmetic retained on the best pair, 1.6 on the worst. It did not degrade. It was gone. And the obvious explanation is wrong, which is the best thing in this paper. You would expect the broken pairs to be the ones where the regression fit badly. They are not. Lama 8b into 70B fits with an R 2 of 0.84. 84 and keeps 94% going up and 37% coming back down. Minestral 3B into 8B fits at exactly the same 0.84 and keeps 93% both ways. Same fit quality, opposite outcomes, which means the size of the mapper's error is not what decides this. Where the error lands is attention does not weigh every direction equally. It scores keys against the model's queries, and most directions in that space are ones the queries barely look at. Error that lands there is invisible. Error that lands where attention is reading corrupts the answer. Ho's team measured that directly. Cosine similarity between the attention output computed from the mapped cache and from the real one. That predicts retention at a correlation of plus 0.57. The calibration R 2, the number you would naturally trust, lands atus 0.2 backwards. The metric that looks like it should screen your model pairs is the one that misleads you. The proof is what happens when they swap the linear map for a small neural network. Same data, same pipeline, only the shape of the mapper changes. The worst pair goes from 59% on Helella swag to 95 1/2 plus 37 points. And on the pairs where the linear map already worked, the neural one came out slightly worse. So it is not smarter. It just parks its error somewhere attention is not looking. Now the bill because this is not free. A mapper runs between 1 and 3.4 4 billion parameters 4 to 12 GB on disk and it is directional. Small to large and large to small are two separate maps. So a fleet of P models needs up to P * P minus one of them. Three models roughly 39 GB. Five models 131 which is more than a single H100 holds. It sits on disk or host memory rather than VRAM and pages in when a pair goes active. The limits are strict and the authors are straight about them within one family only. Both models must share key value headcount and head dimension. Dense attention only, so sliding window and hybrid designs are out. Calibration came from a single data set and the paper ships no code. So my read this is not a capability result and it never claims to be. It is a serving economics result and those are the ones that move what you actually pay. If you run one model, this does nothing for you today. If you run a fleet behind a router, which is where every serious deployment ends up, this is the first evidence that the tax on each swap is optional, and that recovering it costs a matrix multiply instead of a training run. The deeper thing it proves is about representations. Two models trained separately at different sizes end up storing their reading of the same text in spaces a linear map can mostly translate between. Call that an engineering trick if you like, but it reads more like a finding about what these models actually learn. which leaves the question the paper puts in its own future worklist. Every pair here was one family talking to itself. No result yet shows a Quinn Cash being read by a llama. If that one lands too, does a conversation still belong to a model