Enjoying this issue?
Get tomorrow's AI & engineering digest in your inbox — hand-picked, summarized, and always spam-free.
TLDR
Sam Altman's claim that local AI wastes memory holds up on efficiency grounds: sharing model weights across users on servers gets more work per memory chip than dedicating 16GB to a single local conversation. The transcript's breakdown of weight sharing versus KV cache overhead supports this, but the argument ignores privacy benefits of local processing.
Key points
Sam Altman claimed that PC memory is used only 5% of the time.
OpenAI ships a model that fits in 16 GB of memory.
For local inference, each word requires pulling weights from memory, and moving bytes takes longer than the math.
On servers, one copy of model weights can serve a batch of requests, with each user adding only a KV cache.
VLLM measured that for a 13B model on a 40GB A100, about 65% of memory held shared weights and 30% held KV caches.
Tools mentioned
Techniques
- shared weights inference
- KV cache management
- batch processing
Stop scrolling. Start reading smarter.
Receive the day's most important AI & engineering updates in one concise email. No spam.
Transcript (captions)
Sam Alton says, "The memory in your PC gets used 5% of the time, and Open AI ships a model that fits in 16 GB of it. So, is your local AI wasting the world's memory?" On the Cheeky Pint podcast with
John Collison, he called it quite terrible to have the world's limited DRAM, the memory chips themselves, sitting in everyone's own computer. 5% is his round number, not a measurement.
Check it against GPTO SS20B, Open AAI's own Open. 21 billion parameters, the numbers it learned in training, squeezed into 16 gigabytes, so it fits a home machine. Open AI built it to run on your
desk. Its CEO would rather that memory lived somewhere busier. On the memory math alone, he's mostly right. The easy dismissal is that he sells cloud, so of course he'd say it. That's fair context.
It just doesn't change what happens inside the memory. So watch the 16 GB. For models like this, each word or piece of a word means pulling weights back out of memory. And moving those bytes takes
longer than the math does. On your desk, that whole 16 gigabyte load serves one conversation. So servers usually read one copy of the weights per step and use it for a whole batch of requests. Each
extra person mostly adds their own running record of the conversation so far, called the KV cache. Same weights, many users. The team behind VLLM, software for serving these models to
lots of users at once, measured the split in 2023. They served a 13 billion parameter model on a 40 GB Nvidia A100 graphics card. About 65% of memory held the shared weights and about 30% held
the KB caches for every request in the batch altogether. That's an old setup, not Open AI's and Open AI's own numbers aren't public. So if the question is how much work the world gets out of each
memory chip, sharing one copy of the weights wins. But if you run models at home, so your prompts stay on your machine, his math never touches that reason. That verdict is my own read.
Which of your prompts need those 16 GB to be yours?