Why Is DeepSeek So Cheap? We Finally Got the Answer

0:00 I went into DeepSeek's new engine repo with one question. Could the machine behind their cheap API run on a desk? I found the answer in the repo's deployment Q&A. The smallest efficient

0:10 setup adds up to 128 data center GPUs. So, if you were picturing DeepSeek's prices running on your own graphics card, that number ends the plan. And the cheap part is real. Send DeepSeek a long

0:22 prompt, then send the exact same prompt again. On the second send, the price for reading that prompt falls to a sliver of what you paid the first time. 50 times cheaper for repeating yourself. DeepSeek

0:34 has now open-sourced DeepInfer, the engine behind that price. According to its read me, it powers DeepSeek's production services. The obvious conclusion, and the one I'd have drawn,

0:44 the cheap machine is now yours. Partly right. You do get the real production code under an open license, but it targets Nvidia's data center Blackwell chips and Huawei's Ascend 950. Your RTX

0:55 5090 is a very fast card, and it's the wrong very fast card. Further down, the read me lists what this release leaves out. The store for saved prompts, the handoff between prompt reading and

1:06 answer writing GPUs, and the load balancers. I'd expected the hardware to be the catch. What's missing is bigger. So, hold on to one question. What remembers your prompt between sends, and

1:17 is it in the repo? By the end, you'll know why the second send is cheap, why the engine needs so many GPUs, and why one dead chip can sink at least 63 others. Three ideas from DeepSeek's own

1:27 docs explain all of that, and they build on a problem a Berkeley team attacked first. In 2023, researchers at UC Berkeley, Wusha Quan and Jot Hun Lee among them, went after a waste in how

1:39 language models get served. Follow one request. When you send a prompt, the model first reads all of it in a single pass, a step called prefill. As it reads, it writes working notes on

1:50 every token. Those notes are the KV cache. Then it writes the answer one token at a time, checking the notes at each step, and that part is decode. Those notes are what make writing

2:00 affordable in engines like this. Without them, every new token would mean rereading the whole prompt and everything written so far. With them, each step only adds the newest token's

2:10 share of work. But, the notes take memory, and they grow with every token. Berkeley's team found that existing systems threw away most of the memory set aside for those notes. Their fix cut

2:20 that waste to under 4%. LMSYS, the group behind chatbot arena, served its models on it from mid-April 2023 with half the GPUs. That work shipped as vLLM. DeepSeek built its own engine from an

2:35 early fork of it. And Deep Infers readme still credits vLLM as its main base. So, the usual mental model is one engine on a group of GPUs reading and writing every request with its notes in GPU

2:47 memory. Now, drop DeepSeek's own model in. Their new flash model is a mixture of experts, many smaller expert blocks, and each token visits only a few. Per the tech report, the backbone holds 552

3:00 billion parameters. While writing, each token reads only about 16 billion of them. One caveat on size, Hugging Face lists a bigger total for the October checkpoint, and neither source explains

3:11 the gap. Either way, it's a lot of model to hold in memory, and that matters soon. By my rough division, each token touches about a 35th of the model. So, you'd think one big machine could

3:22 cope, but every expert still has to sit in memory ready because the next token might need different ones. So, back to your RTX 5090 and the first place it breaks.

3:32 I checked Nvidia's own CUDA list expecting the newest GeForce to sit beside the data center Blackwell chips. No such luck. Nvidia sorts chips by compute capability, a version number for

3:42 what the hardware supports. Data center Blackwell, the SM 100 family, is version 10. GeForce RTX 50 cards are version 12. A higher number sounds newer, so you'd guess it's covered. Deep Infers read me

3:56 list SM 100 and nothing from the GeForce line. None of the sources documents an RTX build failing, but the memory math doesn't need one. An RTX 5090 has 32 GB. On my arithmetic, a model that size

4:10 doesn't fit at any precision the model card lists, and it isn't close. Say you had the right chips. DeepSeek's deployment Q&A in the same repo puts the smallest efficient unit at 64 GPUs

4:22 reading prompts, matched by as many writing answers. No reason for that size is given. DeepSeek just states it. Then there's failure. In the same document, every GPU in a decode group works as one

4:33 linked unit, each waiting on the others at every step. So a single failed GPU makes the whole group unusable until it's rebuilt on whatever GPUs are still healthy.

4:44 My cluster intuition broke on that line. You'd expect more GPUs to mean more spares. In this design, more GPUs mean more ways for the whole group to stop at once. For a while, the engine itself

4:55 wasn't going to be released at all. On April 14th, 2025, DeepSeek published a short post saying it would not open source its full inference engine. Three reasons came with it. First, the engine

5:07 was an old early fork of vLLM. Second, it was tightly coupled to internal infrastructure, down to the cluster management tools. Third, DeepSeek called itself a small research team without the

5:19 bandwidth to maintain a large open source project. Instead, it would contribute pieces to existing projects. Read that against what we just found and it fits. A decode group that only runs

5:29 as one unit, rebuilt by the platform whenever a GPU dies, is what tight coupling looks like. 18 months on, by my count, the small research team found the bandwidth. But the read me's list of

5:41 missing parts reads a lot like the internal infrastructure that post pointed to. So what is a system if it isn't one engine on one box? Idea one, reading and writing get separate GPUs. A

5:53 paper called Dist Serve from January 2024 explains why serving systems in general do this. Reading a prompt is judged on one clock, the wait for the first token. Writing is judged on

6:04 another, the gap between tokens after that. Put both on the same GPUs and they get in each other's way. Say your answer is streaming when a stranger's enormous prompt lands on the same GPU. Your

6:15 stream waits while theirs gets read. Dist Serve calls that interference, a finding about serving systems in general. DeepSeek didn't publish its own

6:24 measurement. DeepSeek gives each job its own hardware. Prompt reading runs on ordinary eight GPU machines. Answer writing runs on super nodes of up to 128 GPUs. Reading hands its notes, the KV

6:37 cache, across to the writing side, which picks up where reading stopped. That handoff is on the readme's missing list, though the readme adds that most of the engine work it needs is already in the

6:47 code. Even the model gets used differently on each side. According to DeepSeek's tech report, writing touches about twice as many parameters per token as reading.

6:57 You can even put the two halves in different data centers. DeepSeek says that works, then warns that the fiber between data centers is easily cut by construction work. So, a service moving

7:07 trillions of tokens a day has road crews in its threat model. Idea two explains why the writing side is so wide. On that side, DeepSeek spreads the models experts across every GPU in the group,

7:18 so each one holds only a small slice. Wide expert parallelism is DeepSeek's name for it. Writing is a loop. For every new token, each GPU reads the weights of the experts it holds before

7:29 anything comes out. Fewer experts per GPU means less to read on each pass. DeepSeek's own write-up from February 2025 says exactly that, less memory access, so lower latency. Throughput is

7:41 the other reason DeepSeek gives. With experts spread across so many GPUs, the system can batch far more users into each step, and bigger batches keep the GPUs math units busy. More tokens out of

7:53 every GPU hour is, as I read it, the other half of why the prices can stay low. Think of a hospital where each specialist works on a different floor. Each patient, here a token, goes to the

8:04 few specialists it needs, and crowds of patients move through every step together. What it costs is traffic. Every step, tokens travel between GPUs to reach their experts, and the load has

8:14 to stay even so no floor gets swamped. DeepSeek names both costs, communication across GPUs and load balancing. Here the hospital stops working as a comparison. A real one keeps running when a

8:26 specialist calls in sick. DeepSeek's can't because every floor is part of one linked unit, so a single failed GPU stops the whole group. Recovery is blunt. DeepSeek's platform counts the

8:37 GPUs that are still healthy, picks a new group size from those, and restarts. A restart measured in minutes for one bad chip. Back to those bigger batches because they have a catch, too.

8:48 DeepSeek's decode chart shows it. The more users share each step, the slower each answer streams. Busy GPUs and fast answers pull against each other. Those two ideas explain the hardware and how

8:59 much each GPU hour gets through. Splitting the work is why there are separate halves. Spreading the experts is why the writing half is so big and why it fails as a group. Neither one

9:09 explains the cheap second send from the start. My guess going in was that the cheap second send came from some clever GPU trick. Idea three is storage. In the usual model, the notes from reading live

9:20 in GPU memory while a request runs. When a request ends, DeepSeek doesn't throw the notes away. It files them on roughly 100,000 terabytes of SSD in its own file system called 3FS. All that storage so

9:34 the model doesn't have to read your system prompt twice. Those notes stay for days. Three for agent traffic, seven for chat. DeepSeek even put a price on the extra days.

9:43 Trimming chat down to the agent setting would cost a sliver of cache hits, and it keeps the longer setting anyway. Why is disk good enough for this? Back to the tech report. The model keeps

9:53 about 890 bytes per token in what the report calls its global cache. That's tiny. By my math, even a 100,000 token prompt comes to under 100 megabytes. Small enough to park on an SSD and fetch

10:06 back. That's my reading of why disk works. The report doesn't draw the link itself. Your price hinges on one rule from DeepSeek's API docs. A request only hits the cache if it fully matches a

10:17 saved prefix, a run of text from the very start of the prompt. Same opening, same notes, no re-reading. One trap follows straight from that. Put a timestamp at the top of your system

10:28 prompt, and every request starts differently. You've paid full price to tell the model what time it is. Now the opening price makes sense. On a miss, the reading GPUs work through your whole

10:38 prompt. On a hit, the notes come back from disk. Most of the reading never happens. As I read it, skipping that work is what the lower price reflects, and disks make the skip possible. Hits

10:49 are common, too. In February 2025, DeepSeek published one day of serving its older V3 and R1 models, and more than half of input tokens hit the on-disk cache. I compared those

11:00 retention windows with DeepSeek's public API guide, and the guide is vaguer. Caching there is best effort, and unused caches usually clear within a few hours to a few days. I'd count on hours, and

11:11 treat days as a bonus. Which answers the question from the start. What remembers your prompt between sends is that KV cache store, and it tops the readmes list of what isn't in this release. How

11:22 it picks what to throw away, and how it finds a match across all that storage isn't documented in anything released. One more missing item decides where traffic goes. A real service runs many

11:32 copies of both the reading and writing groups, and something has to send each request to the right copy. Deep Infers Readme calls that load balancing across replicas for both sides and leaves it

11:43 out, too. You might expect the release parts to be the hard ones to get anywhere else, but in May 2025, the SG Lang team set out to reproduce DeepSeek's serving design, reading and

11:53 writing split apart, and experts spread wide. They did it on 96 of Nvidia's older H100 GPUs. Their reading speed came to DeepSeek's published profile. They estimated their

12:04 cost at about a fifth of DeepSeek's own chat API price at the time. SG Lang got there well before this release and measured itself against DeepSeek's published numbers. So, the

12:14 release fills in the half outsiders had already rebuilt. The prompt store and the balancers, along with the handoff, are still on the not included list, and the Readme doesn't say why. On to the

12:25 money. DeepSeek's deployment Q&A says a single cluster serves 155 trillion tokens a day. Priced at off-peak rates, that cluster earns $38 per GPU hour. Read that as revenue.

12:38 Profit and running cost are different numbers, and nothing released gives the cost. Hacker News split right on that line. A commenter named Abound argued that owning a data center full of GPUs

12:48 costs far less than that per card per hour. Harhar very funny replied that calling it profitable means nothing if you ignore what the model cost to make. So, two strangers ended up doing

12:58 DeepSeek's accounting in a comment thread for free. DeepSeek's own older numbers point the same way as Abound on serving costs. A February 2025 write-up from DeepSeek

13:08 showed a 545% cost profit margin on older hardware. DeepSeek itself called that theoretical since its web and phone apps were free. As I read it, that margin, like Abound's

13:19 math, counts only the GPUs serving tokens, not training. My read on that revenue line, it's what one of their GPU hours earns with every closed part attached.

13:29 Run the open engine without the cache store and the balancers and you aren't buying that number. What you do with all this depends on which kind of developer you are. If you

13:38 call Deep Seek's API, the cheap part is already yours and the rule fits in a sentence. Keep everything that doesn't change at the very top of your prompt in the same order every time and put what

13:48 changes at the end. Say your agent sends a long system prompt and a tool list on every call. Keep those two identical down to the character and add new conversation turns at the

13:59 Then each call has a chance to match the saved opening instead of paying to read it again. Watch the clock, too. If your jobs can wait, off-peak is half the peak price

14:08 and the peak windows, weekday hours in UTC, are listed on the pricing page. What Deep Seek did release is real and that's the benefit for everyone else. You get the production engine for its

14:18 flash model under a license that lets you build on it. Until this week, how Deep Seek runs the split and the wide experts lived only in write-ups and outside rebuilds. If your company runs

14:28 data center Blackwell or Ascend 950, that's a serious starting point. Budget for the missing pieces, though. You'd build the prompt store, the handoff, and the load balancing yourself and Deep

14:39 Seek's efficient minimum is still 128 GPUs. If you've got an RTX card, read this repo as a manual. Split reading from writing, spread the experts, keep

14:49 prompts on disk for days. The first two explain the 128 GPUs. The third is the one behind the cheap line on Deep Seek's price list. My call. For API users, this release is the clearest explanation yet

15:02 of why prompt order is worth money. For teams with SM-100 clusters, it's a head start that stops right before the hardest part. For everyone else, it's a rare inside look at serving at this

15:12 scale, written by the people who run it. Back to that second send. It cost 50 times less and Deep Seek could skip most of the reading. Your first send's notes were waiting on an SSD in its cluster,

15:24 and the part that remembers is the part they kept. Today the repo covers a single model. I'd watch whether that cache store ever lands in it. Until then, open your own prompts and look at

15:34 the very top. What's sitting there that changes on every call? Tell me what you find.