Why SSD inside GPU doesn't work..

0:00 If solidstate drive is the cheapest memory option out there, why can't we just bring SSD directly into the fabric of GPU? Yeah, it's sort of a Frankenstein build, but at least we

0:09 finally have cheap memory available in GPU, right? SanDisk and SKH Highix recently released a more concrete standard called high bandwidth flash trying to make this happen. But neither

0:19 of these companies manufacture GPUs. So, they have to entice GPU manufacturers like Nvidia and AMD to adopt HBF as a new standard. And yet it's been rather radio silence. But why? Bringing cheap

0:32 memory near GPU seems like a perfect idea. There's actually a bit of a problem with this. Large language models today easily go up to 2 TB in size. And as you can see, we only have so much

0:43 memory to go around. Say 141 GB in GPU, 64 GB in RAM, and 3 TB in SSD. We could fit this entire model in SSD rather comfortably as you can see, but we have to bring them all the way up to the GPU

0:57 core where the actual computation is done. And the current maximum throughput is about 14.8 GB per second, which means you have to wait more than 2 minutes just to get the entire model's weights

1:08 streamed to the GPU, which is not a great solution. Now, SanDisk and SKH Highix's proposal here is to create a new tier of memory right here. And because flash memory is cheap, we can

1:19 easily support 2 TB of memory directly into the fabric of GPU. And since this is a new tier of memory, they support up to a throughput of 3 TB per second using UCIe standard. And now using HBF, a two

1:31 TB model can stream to the GPU core in less than a second. Surely, this has to be a genius idea, right? This is just one of those ideas where it sounds good on paper, but in reality, you're getting

1:42 the worst of both worlds. Typically, we use GPUs as an accelerator to train models or run models through inference. And turns out, HBF actually makes both training and inference a little bit

1:53 worse. Let's find out why. Currently, high-end GPUs can pull memory from HBM at a speed of 4.8 tabytes per second. So, right off the bat, you're looking at a 60% faster speed from existing HBM and

2:06 comparing them to the newly proposed HPF. Now the problem with HBM is that as the model sizes grew over time, GPUs didn't really catch up at the same time and this very gap here is really where

2:18 the problem is. And SanDisk and SKH High's solution was to fill this very gap with HBF as a periphery option. And until now we have been approaching this problem by stitching together many GPUs

2:29 to run at the same time. So intuitively we can just pull together many HPM all together and split the model across them. let's say 16 GPUs to make up for the size deficiency. And since the two

2:41 terabyte model is now split across 16 different ways, each GPU only needs to pull about 125 GB of weights. Which means at a speed of 4.8 tabytes per second, you're pulling the model at an

2:52 incredible speed of 0.026 seconds compared to a single HPF that takes about 0.67 seconds. And since the medium that actually connects these GPUs together still operates at a faster

3:05 speed of 3.6 6 terabytes per second. HPF still falls short in large when rubber meets the road as you can see. But what about inference? Can we just use this newly minted space to store KV cache

3:17 during inference? Using HPF to store KV cache is actually very different since KV cache often grows as the model processes more and more tokens which means you have more write operation

3:29 going into HBF than read operation reading from HBF. And nan flash is notoriously slow in write operation since they need to be erased before they actually are written. When you zoom out

3:40 a little bit, DRAM is really meant to be used for working memory, meaning read and write speed is incredibly fast since they are dynamic RAM. NAN flash is different by nature. It was always meant

3:51 for persistent storage, which means compared to dynamic RAM, read and write speed is slower in comparison and often times write is significantly slower than read. So using nanflash for KV cache

4:03 means write operation is more dominant than read since KV cache tends to balloon as the model generates more tokens. In fact, there's already a paper out there with the title called HPF

4:15 sucks finding only 1% improvement in latency using HPF as KV cache as I scale the theoretical speed to 3.75 times. So, as you can see, simply bringing cheap memory up the stack and beefing up the

4:28 bandwidth to 3 terabytes per second still falls short when it comes to inference. But what about training? Is there benefit to be gained to use HPF during training? Not really. And here's

4:38 why. Training large language models benefit a lot from scaling the number of nodes and having a faster bandwidth. And it has less room to actually benefit from when we actually make improvements

4:48 on memory bandwidth and memory capacity. This looks very different in inference work where the compute profile for prefill stage is a lot more computebound and memory capacity bound. Same thing we

4:58 can say for decode stage is also bandwidth and latency bound. So naturally improving the memory capacity by bringing up cheap memory near the GPU actually stands to benefit inference

5:10 more than it does for training. So going back to the geometry of compute where inference actually reads the weights from memory to produce new information in the form of KV cache training large

5:20 English language models need to use the model's weights to generate the next token. It also needs to generate gradients and actually apply the gradients through optimizer to update

5:30 the model's weights. In other words, training isn't exactly dominated by read operations. But in reality, there's a lot of back and forth, which makes the theoretical benefit of HPF not really

5:40 realized. So, it really remains as this awkward middle ground that doesn't really stand to gain huge benefits. But whether or not we resolve this memory problem with HPF or not, at the end of

5:51 the day, it's still important problem that needs to be resolved. For Nvidia's case, the way that they've been doubling down on is in their CMX solution or context memory extension. Because in the

6:02 application layer of the AI stack, the use cases in this highest abstraction are longer running agents, multi- aent systems, and loops. It puts a tremendous pressure downward to not only make more

6:13 KV cache happen, but also hold on to these KV cache for longer duration of time. And while HPF from SanDisk and SKH Highix targets to alleviate this downward pressure at the chip layer by

6:25 adding HPF in the chip, Nvidia has been betting on alleviating this pressure in the infrastructure side by selling a joint system in a pod structure where you use blue field and spectrum X that

6:36 contains dedicated storage and networking. So essentially we contain the inference problem that we have into the single unified structure instead of having a more customized chip at the

6:47 chip layer that solves a more specific issue by adding HPF attachment. But GPUs are really meant to be more general purpose and companies like SanDisk and SKHEX don't really own the AI stack the

6:58 way that Nvidia does. So their collaboration led to establishing this specification at the chip layer by introducing what's called high bandwidth flash or HPF which is a new technology

7:09 that's meant to expand the memory capacity at the chip