The Best Local AI GPUs to Buy Right Now (Buyer's Guide)

0:00 It's getting pretty crazy out there for local AI GPU prices. I want to take you through this new tool that I put together on digital spaceport.com that can help you get the best price and also

0:10 gives you some key information that you might be interested in about a wide variety of GPUs and AIO units out there. These are, in my opinion, all of the main GPUs that you should consider.

0:22 There's 17 on here. There might be room for 18, but definitely the 5090 has been going through some crazy stuff recently. If we check eBay, 5,400 is the lowest price now for used ones if you wanted to

0:35 buy it now. Maybe you could get a better one on an auction, but good luck with that. And that is a huge delta off the MSRP. And if you watched, you know, I have lots of complaints about the way

0:45 that Nvidia has priced out their MSRP in recent years. So, the delta from MSRP is basically the street price minus the launch MSRP. Negative means that it's cheaper than when it listed. And you can

0:57 see that right now $3,400 is the premium on a 5090. The price per gigabyte of VRAM, 32 GB of VRAM there, $168.75. Definitely by a long shot the leader

1:12 here as far as price per gigabyte of VRAM. That is a insanely large price. I'm not even including the RTX 6000 because it is so freaking expensive right now. It is in a different price

1:24 category entirely. Like if I'm going to include that, we're going to have to put like H100s and everything else in that. Maybe sound off down below if you're interested in me doing that. The price

1:33 per gigabyte second. So, how many gigabytes a second can the bandwidth of the system go? 1,792. So, 1.792 terabytes per second. You're paying $3

1:47 and a penny right now at this price for the lowest used price. That is pretty insane. Of course, all of this has links and you can find this sheet linked in the description below and in the pin

1:58 comment. And so, in my opinion, 5090 fever has picked up to a point such that it's an international phenomenon. And that's why we're seeing the prices right now go insanely FOMO. The Chinese are

2:09 FOMOing into them. There's huge amounts of export ones being bought in the United States going overseas right now. Next up, 4090s. And the 4090 is going to be, and I can see this already from

2:19 yesterday to today, the next follow-up GPU that is zooming off the shelves and off the used shelves especially. MSRP on that one was $1599. One terabyte per second roughly was the bandwidth that

2:32 that had only 24 gigabytes. You look at the MSRP price on that. The delta is from 1599 to 28.50, 50. The lowest price used one right now, $1,250 roughly. So that is over MSRP. Price per

2:48 gigabyte of VRAM, $11,8.75. So both of those are in some insane category pricing for the price per gigabyte of VRAM. The dollars per gigabyte second of bandwidth is $2.83.

3:02 Of course, very high for that one. Also, still one of my favorites. what I got rocking 3090s 24 GB 936 GB a second system bandwidth on that and the MSRP is $14.99 that was what it launched at a

3:18 long time ago too. It was kind of expensive when it launched but it was a different time. It was kind of crazy for GPUs already going into its launch. So it commanded a higher price. Lowest

3:28 price right now $1,300. So it's actually $199 cheaper than what MSRP was. Grantite, you probably have to put some money into servicing them. If they've never been repaded and repasted, you

3:39 should definitely account if you buy one on a repad repaste. Of course, you can follow along with the guide that I just put out on the uh PTM material. Really good at the temps and the like

3:49 consistency of the temps on mine have been just like under ADC. Like it caps out at ADC with 100% GPU utilization. So, highly highly recommend that price per gigabyte of VRAM 5417. Still, you

4:01 see like half the price of a 4090, a third the price of a 5090 dollars per gigabyte second. And this is where I think you start to see $139. There's a lot of offerings around here that I

4:14 would call if you're looking for purchasing for performance. So, let me hover this a little really quick. Price performance, lowest street price divided by memory bandwidth in gigabytes per

4:22 second. So, this is basically what price performance means in this instance. And if you look at gigabytes, uh, price per gigabyte of VRAM, you can see that's price per capacity. So, you want to

4:33 factor both of those in. Can it run the model and how fast can it run the model? Those two things go hand in hand. Moving down here, this one's a little bit different. And this is the 2080 Ti at 22

4:43 GB of VRAM on this. And this is, of course, a modded one that has been uh soldered, the chips on it. 616 GB a second of bandwidth that retailed for $9.99. The lowest price right now on

4:57 that is 575 used. So of course there are concerns you should have around that $424 under what the MSRP was though. And if you look at the price per gigabyte of

5:09 VRAM 264 and the price per gigabytes per second of bandwidth is 93. So of course pretty freaking attractive. And honestly it is actually pretty freaking attractive. 5060 Ti. I remember when

5:24 this one came out and I was like, "This is a great one. You should seriously consider this. At $429, this is the local AI GPU to get." And it was and it has been got. And unfortunately, the

5:35 price has gone quite high. So, we see that 16 GB of VRAM is now commanding $650 from an MSRP of $429. You're looking at $221 above MSRP for that. $4063

5:49 per gigabyte of VRAM and $1.45 45 per gigabyte second of bandwidth. Next up, looking at the 4060 Ti. This one is one to avoid. Actually, I just I put this in here actually because I wanted to say

6:01 this during this video. Avoid the 4060 Ti. I have no idea why they're still commanding $600 used that for a $500 GPU when it launched and like $288 gigabytes per second of memory bandwidth. Not that

6:18 good. Not that good. was it was blasted for that when it launched. It still should be blasted for that. You're looking at an MSRP delta of $101 over where it was on MSRP launch. The price

6:30 per gigabyte of VRAM 37.50 and the price per gigabyte second of bandwidth $28. So looking through these, there's a couple that really do stand out still like what you're paying and what you're getting as

6:42 winners, and that is the 3090 and the 2080 Ti. Next up, the lowest end that I would ever even begin to consider, and I do have them myself, and actually I'm considering getting two more myself so

6:52 that I can have 48 gigabytes on those 3060s. Uh the 12 GB 3060 was a really good GPU. It was There's a reason that it was like one of the most popular GPUs for a really long time. 360 GB per

7:04 second of bandwidth on the memory. $329 was the launch price. Lowest used price right now is $300. Those are on the creep up. So, those are still $29 under what the MSRP was. And that is $25 per

7:17 gigabyte of VRAM and 83 cents per gigabyte second of bandwidth. So you see like 2080 Ti, 3090, and the 3060. The next one up is everybody's most recent favorite. This is more of an

7:30 enterprise class or Xenterprise class GPU, the V100. There is a 32 GB version and a 16 GB version, and those are in PCIe. If you wanted to go SMX, there are some additional interesting options

7:42 there. But I put the 32 gigabytes in here because there's a really important compromise that if you're going to do this, you probably want to have as much capacity as possible. So it makes sense

7:51 in my opinion to go with the 32 over the 16. That is 900 GB per second of bandwidth. So really impressive. And that was $11,500 on launch. Again, X Enterprise. This was

8:04 data center kind of GPU. The lowest price used right now, $625. $10,875 underneath what the MSRP was. That's attractive, right? $1953 per gigabyte of VRAM. And that comes out

8:21 to 69 per gigabyte second of bandwidth. Factor this in. Caveat, if you are running this, you need to have a blower that is cooling it. And this does not have fans built in. So that blower

8:33 definitely needs to be able to move a lot of air over it. They're loud. This is also why if you're going to go loud like that, go with 32 GB over 16 GB. In in addition, like 32 GB is a really good

8:46 amount of system memory to have. You can do quite a bit with it versus 16 GB which can get a little bit limiting. In the 20s is where you can run a ton of models. 24 GB has been a popular target

8:58 quans and tuning for a long time and there are really good quants and tunings out there fine tunes out there for that size. Next up we got the P4 and also it could be said the P100 24 GB 346 GB of

9:12 bandwidth on that. So not very much to be honest with you. That was about a $6,000 card when it launched. Now about a $300 card. I believe the P100 is even cheaper than that. That is a about a

9:24 $5,700 price delta. $12.50 per gigabyte of VRAM and about 87 cents per gigabyte second of bandwidth. This is also why I would probably recommend going with the V100 over the 24 GB variants of the

9:38 Pascals. 900 gigabytes per second, 346, different different classification of the tokens per second. The next one that in my opinion you may have missed the boat if you didn't get in on this one

9:50 already and that is the CMP 170HX. This can go up to 64. I believe there might be even 80 gigabytes. I've been reading about this a lot on X. There's some people reporting they're having problems

10:02 with some of them that they bought. I would put this in a are you really good with tinkering? Like I wouldn't be comfortable buying this and tinkering on it myself at the current price. This was

10:12 something to have gotten into in my opinion a while back, but basically this is an amper generation A100 that kind of got knocked down from Nvidia turned into and the memory cut down and the PCIe

10:26 bandwidth cut down drastically to a uh mining GPU that they launched towards kind of the mid end of the Ethereum craze. and it doesn't come I think it comes in much lower gigabytes of VRAM

10:39 that it's reported. Finding one that is actually pre-moded and has been tested and will have a guaranteed period of warranty that is fairly decent would be something I would check for on that 1.5

10:53 terabytes a second though like serious bandwidth on those. And for doing things like uh text gen, you're you're going to be like really grateful that you have something that can like hit that amount

11:05 of bandwidth. That is insane. Those were 4,299 as mining GPUs when they launched. Now 2,700 was the cheapest I could find for one that seemed to fit the criteria that I would consider interesting. That

11:17 is still, if you think about the MSRP, 1,599 off what the MSRP was. So you're not overpaying necessarily from what it launched at. Uh 4219 was the dollars per gigabyte of VRAM and $180 is the price

11:32 per gigabyte second of bandwidth performance that you can get. So a very attractive option if you are a highly skilled uh tech tinkerer. It it had a time period where these were really

11:43 really cheap and that time period has passed. So that may not be the thing for a lot of people out there. That right there wraps up pretty much the Nvidia discrete GPUs. Moving on to the AMD

11:56 retail side, the 9060 XT. So, did a video on that. The performance on it's actually really good if you're looking at uh text and just kind of back and forth with it. I have not hooked it up

12:07 yet to an Aentic workflow and I need to do that to test it out. So, there will be a future video coming on that. 16 GB of VRAM on that. 320 kind of puny gigabytes per second of bandwidth on it.

12:18 But it in the video you can check out. I'll link that in the description below. I compared it to the 5060Ti and it held actually pretty darn decent. Of course, this one also subject to inflation, but

12:30 it had such a low MSRP, 349. And if you recall back, this is why you hit subscribe. This is why you ring the bell because I launch price alerts when I see good deals. And those price alerts

12:42 sometimes net people out really good things cuz my wife checks all of these and updates this sheet. Now, a couple of times a day, there's a really killer deal. I will for sure drop a

12:53 notification if she says, "Hey, by the way, that thing's actually like pretty pretty decently off." And we've seen that happen a couple times with the 960 XT coming into this time of year. It

13:02 could happen again. Right now, it's at 500, but I believe I picked mine up at 389, but I paid it was more than MSRP, but it was still a discount, and it it's a

13:13 pretty decent performer. I need to do another follow-up video because it's been a while since I did a video on that one. Currently, 151 over MSRP. The price per gigabyte of VRAM 3125 and the price

13:24 per gigabyte second of bandwidth $156. The 7900 XTX 24 GB 960 GB a second. I've heard I don't have one of these to test hands-on, but I've heard that they can be very good under certain conditions

13:40 for certain types of workloads. $899 was the MSRP. Cheapest right now is about 900 bucks used. And that is about a dollar over your MSRP. 37.50 for the gigabytes uh per gigabyte of VRAM and 94

13:54 cents per gigabyte second of bandwidth. So pretty attractive on the price per gigabyte second given that it's a 24 GB footprint. One that I am seriously considering the

14:04 Pro R 9700. And the reason I'm considering it is because I've seen some people's numbers that look really, really good. Uh there's a guy I follow on X named Mike. He's got a pair. I'll

14:14 link him down below. Uh this has a 32 GB VRAM footprint. Also very attractive to see that 32 GB VRAM footprint come in. 640 GB per second of bandwidth. Respectable. 1299 was the MSRP. Current

14:29 cheapest that I was able to find $1675. So that's $376 over what MSRP was. you're paying $52.34 for each gigabyte of VRAM and that comes out to $262 per gigabyte second of

14:43 bandwidth. It's a it's an attractive option. Next up, we're moving into the AIO's. And so this is Apple, Nvidia, and AMD that we're going to take a look at here. The 395 Hello Stricks kind of

14:55 class, which is 128 GB in the one that we're going to look at, which is probably the most common one out there. They also share some of the same common characteristics though like the 256

15:06 gigabytes per second of bandwidth that that has retailed was$199 so almost 2,000 right now lowest price 3350 so quite a bit higher that delta 1351 and if you look at the price per

15:21 gigabyte of VRAM on that 2617 and the price per gigabyte of bandwidth was $139 so not something that I would consider The Nvidia DJX Spark 128 gigabyte very popular 273

15:39 gigabytes per second. That is of course NVFP4 darling. And the reason NVFP4 again this is a quant 4. So if you're trying to run like a full precision FP16 dense on a DGX Spark, it's absolutely

15:54 going to like melt. It's just not going to have a good time. It's going to go way too slow. That's where you need discrete GPUs. if you want that kind of precision and

16:03 performance along with it. But $273 GB per second is the bandwidth on that. $3,999 MSRP about 5K right now for the cheapest one of those. It's about a,000 bucks

16:14 over and you're looking at $396 per gigabyte of VRAM and that's about $18.32 per gigabyte second of bandwidth. So it it it is very very popular. If you get a high performance network switch, you can

16:28 actually have very good compounding effects by networking and clustering them together. The DGX Spark over the AMD stricks has been favored because of that additional capability because it

16:40 has Melanox, which is I mean, if you've ever gotten to use Melanox networking gear, it's primo. It is good and it is expensive for a reason. Now, of course, you can check out my old guide on how to

16:52 get up and running with like 40 gigabit networking, which is actually quite cheap, and including things like Infiniband, you can get into for very, very cheap. I mean, old switches that

17:00 are Infiniband still 40 gigabits per second. They're they're very affordable. Like, very, very affordable. Even the 100 Gbit per second switches. And finally, we've got the Apple AIO's here.

17:11 And I just threw in the M1 Max and the M4 Pro. These both seem like very, very popular options out there. the M1 Max at 64 GB of VRAM on that well essentially unified memory and 400 GB per second of

17:25 bandwidth. Now, where this one really falls down though is it doesn't have tons of compute. So, it's going to get nailed really hard on doing prompt processing and especially in deep aentic

17:36 workflows. Both of these are not going to be very happy uh campers if you go deep on them with certain types of models, especially if they're dense, if they're high active counts. The further

17:47 and further you get into the context window, the worse and worse that performance is going to be. Decode interactive chat back and forth, they're going to perform pretty fine. That time

17:56 to first token though, uh when you're deep in a context window with them is reported to be quite high. The MSRP on that one was $19.99. Lowest price currently is $1,300. So, that is about

18:06 $700 off. So, that actually is kind of attractive. And I would expect as we get closer and closer to the new uh Apple lineup launching, which is right around the corner, people are already preload

18:17 pre-ordering them. They're waiting, many people for the larger one, the 512 GB one. And I think that could be a mistake that could blow up on them because they Apple might see what kind of demand

18:27 there is. and like nothing seems to be improving as far as the availability of memory out there for the manufacturers. So prices could go up on that one. Uh we'll see what happens, but I think

18:39 there is some good reason to consider that these older ones may even come down more in price as people sell them off getting ready for the new lineup. $20.31 per gigabyte of VRAM, system unified

18:51 memory, whatever on that one. And $3.25 per gigabyte second of bandwidth. The next one, the M4 Pro, 48 gigabytes, 273, kind of puny as far as the bandwidth and gigabytes per second. MSRP on that one

19:05 was $17.99. And the lowest price right now, 1,250 bucks. So, that one is about $549 off the MSRP, 2604 as far as what you're paying per gigabyte of VRAM, and about

19:19 $4.58. Of course, I made this so you could go ahead and sort this if you wanted to see things like highest gigabytes per second or if you're looking for a specific

19:29 comparison on something like you want to see the gigabytes per second of VRAM capacity, what you're paying for each one and you want to compare on that, you can really start to see why I think the

19:39 3090 and the 97 or 9700 are kind of comparable in certain regards. And that really comes down to the value proposition. $54 versus $52. So, some things to consider there. You can find

19:53 this sheet linked down below. And huge hat tip to everybody who's helping get us to 100K subscribers. We are so close. So, have you rung the bell? Have you hit the like? Subscribe. And huge hat tip to

20:05 everybody who shares this out there. Also, if you're looking for more information, like I mentioned, you can find out more about the 5060 Ti in the video that I've got here

20:16 and about the 9060 XT in the video I've got here.