ULTIMATE GPU Maintenance Guide

0:00 It's way too expensive to lose a GPU right now due to lack of maintenance. And if you're running a Gentic local AI, you're running hotter than you have before, especially if you have things

0:10 like 3090s, 4090s. I'm going to take you along the maintenance with me today as we fix and rectify a problem in one of my GPUs. I'm going to show you everything that you need to know for any

0:21 GPU to completely maintenance it. I'm going to take you through what it looks like when you have a problem, how to identify and label your GPU so you can know which one is which. How you can

0:30 disassemble any GPU out there and not screw up and take notes along the way exactly how to document it. How to clean your GPU die. How to replace your pads when you need to replace your pads. How

0:42 to apply the latest and greatest in thermal protection to your GPU die, which is Thermal Grizzly's Phase Sheet PTM, the best out there for keeping long-term sustainability in your GPUs

0:53 without pumpout. How to reassemble completely your GPU. At the end of this, we're going to go through a cost sheet and look at what the total cost of maintenance looks like versus just

1:02 replacing one of the GPUs had I failed to do this maintenance and it turned catastrophic, which you better believe it can. There are lots of chapters below, so be sure to use those. There's

1:12 also a tremendous amount of information on the article at digital spaceport/GPU rig-maintenance. Now, when you're looking through what you're going to see if you've got a

1:24 problem, this is going to look like possibly this. And this right here, as you see on GPU one, is a just the boost is not staying there. And it keeps throttling down quite a bit. The fans

1:37 are pegged at 100% yet confusingly the temperature looks like it's around 72 75 centigrade which should be okay. This is a clear sign that you've got a problem. Usually that problem is going to be pump

1:52 out of your thighs paste. So that's what we're going to get into today. We're going to get into GPU one. Change that out. I I'm going to film that process. I'm going to do it on all of my GPUs

2:02 because it's just time. like this one's first and soon there will be more is pretty much a guarantee because I did all of these at the same time about 2 and 1/2 years ago. So looking at this

2:13 though, this is how a problem usually presents. You're not able to maintain a very, you know, consistent boost and you see it throttling down into a P2 state with Nvidia GPUs. So this does also take

2:26 quite a bit of time. I would factor at least an hour and a half for each GPU to really thoroughly clean it and service it. There is also if you are going to be replacing your thermal pads and you want

2:36 to do a bulk buy, especially if you've got like four GPUs like I've got, probably want to consider something like that because it can save you a lot of money versus buying pre-cut sheets. You

2:46 then you definitely need to get the measurements for all your pads. I'm going to show you how to do that today and how to calculate how much you need so that you can then get enough for all

2:54 of them that show up and then you be able to address all of them at the same time. That does mean also downtime from one of your GPUs, which will remain in a disassembled state until those pads show

3:03 up unless you can get the measurements. I urge you not to trust what you find online for measurements and instead to measure yourself because I actually trusted what I saw online and it was

3:14 wrong versus what I measured. So, I ended up actually incurring quite a bit of extra expense with a particular depth of millimeters on my Ventus 3Xs. So, I would recommend definitely if you've got

3:25 a lot of GPUs you're going to be doing this on, do the measurement yourself. Don't trust what you find online. So, you might be wondering, how can you tell which one is which? So, there's a

3:35 command called Nvidia SMI and hyphen capital L that will give this to you and that works both in Windows and in Linux. So, that just looks like and we can see the GPU IDs. So device 0

3:52 GPU0 device 1 GPU1 and so on. So the one definitely in question for us is the one here that ends in A94 and E16. So as you're doing this, the best thing to do is remove after you've done your first

4:10 Nvidia SMI-L with the all of them listed. kind of create a list like this and then pull one out at a time, restart the system, rerun it, and then by process of

4:22 elimination, you can figure out which GPU is actually which ID. Now, it's easier if you're adding in GPUs, just add them in one at a time, and then that GPU, you can run the command, print the

4:33 sticker, put it on there. Something you should do when you're setting up, tearing down, or doing maintenance, cuz it's a real pain in the butt to figure out which GPU is which outside of this.

4:42 So, I just use a label maker and stick a little sticker on the back edge of each of these. And that way, I've got all of them labeled so I know which GPU is which.

4:54 Make sure you have your power supplies turned off. I would go ahead and unplug them and also drain the power out of it by holding the power switch for like five

5:07 seconds. And now you're ready to start working on the system. Next up, I'm going to go ahead and remove the power for the GPUs. And I'm going to tie those up with some

5:28 Velcro so it's out of the way. And then working from this side to the other side, undo the PCIe risers one at a time. Kind of tuck it down a little bit. and up and out.

5:55 And D. So, what we're going to be using today is Thermal Grizzlies Fase Sheet PTM. This is a great great new innovation, especially with even larger die sizes. These do fit like what we've

6:06 got with these 3090s. This is something that you want to before you apply it, actually put it in the fridge for about probably 30 minutes or in the freezer for about 10 minutes so that you harden

6:17 it up and it's easier to cleanly peel it without tearing the sheets for the application. It'll make it a little bit easier. And of course, you can find links to these in the description below.

6:25 Each one of these that comes apart is going to take some time to get apart, so take your time. This is the time to check for your thermal pads to see whether or not they're good or not. If

6:34 you notice on these, you'll see this green poking out here. So, I did these with critical, which is probably one of the best. However, it is almost insanely impossible to find them now. So, I do

6:46 not think you're going to be able to get that. I would also recommend going with what you can get from Thermal Grizzly as far as the pads for repading. Definitely, it is something that when

6:54 you're looking at something like 5 years, probably a really serious consideration, especially on a GPU like a 3090 that has serious serious heat, you should be setting a power limit on

7:04 your GPUs and you won't see a huge impact on that if you set it down to even like 20% off. So, you can actually save yourself a significant amount of wattage if you're looking at four GPUs.

7:15 That changes the equation quite a bit. you could be looking at just using about 1 kilowatt of power, which is better than looking at using like 1.3 1.4 kilowatts of power if you ran them

7:26 uncapped with very minimal difference in what your performance is going to look like. So, highly recommend doing that. Also, I'll put all the commands to do that on digitalport.com/gpu

7:37 rigmaintenance. We're going to start by removing the fan shroud. So inside here between the fin blades, we are going to get the little screws that are in there. It is very helpful to have a small size

7:51 screwdriver. And if it has a magnetic tip, all the better. This one has a very lightly magnetic tip. And I like to use one of these mag trays also so that I don't lose the

8:03 screws, which probably when you get past this point want to start taking a picture of where screws were and what the screws looked like with your cell phone so that you know which screw

8:14 actually went where. There you go. And so as soon as you get that taken off, we're going to next take off these neat little power supply connections. And you really do need to

8:28 get a nail underneath these usually. And there we go. So, our fan assembly is actually remarkably clean still, but it's easy to clean off the backside of the blades when you have this like this.

8:41 Just uh some isopropyl alcohol, a Q-tip upside down, and you can get it that way. You don't really have to remove your fans or anything like that to get it. But that's essentially your step one

8:53 right there. Now, I've got another tray that I'm going to use for the screws that are going to come next. So, these kind of I can just put over there. All the rest of the screws are going to go

9:02 into this one. So, we're going to remove the back plate next. In some GPUs, there will be a few screws that may be slightly longer. So, pay attention to

9:15 any that are slightly different length that come out of the back plate. Like, you can kind of see it here. This one was shorter and the front two were longer. So, these front two were

9:26 longer and the back ones are going to be shorter is what it looks like. So, you can even move those like that inside your tray. And I'm actually going to take notes

9:36 here. Crappy drawing on this scratch pad that I've got of the back plate that. And I'm going to put circle circle long long. And then one, two, three, four, five,

9:53 six. So, do take notes. Also, you probably will run into something like this where one of these screws will be coming through

10:10 in a somewhat more complex manner, especially on a 3090. These have bigger heat sinks and stuff like that. So, there's one, two, three here. And it's never a bad idea to take a

10:23 picture with your cell phone just so you know which ones went where. So yeah, take pictures that way you know one, two, three went on this brace that basically reinforces and gives it some

10:36 strength here. That is a really long one. So I'll take a note of that. And then there's these two shorter ones. So, this is absolutely the process that

10:52 you should go through. That way, you don't screw anything up. There's no such thing as taking too many pictures. You very well could find yourself needing them.

11:05 And these are the short ones. Again, those are the exact same ones as these other short ones. So, now we've got essentially three different types that have come out of here. So, we've got

11:16 short and short. One more on this side and then the front plate. So, next we'll remove the face plate. I had some pads that were attached to

11:50 it. So, I'm going to take a picture of those also just to make sure that I got those right. And we got a retaining screw down here that we need to get out. pads do look like they're in good shape

12:13 still. The things that you need to do to be able to change your pads out is basically be really meticulous in measuring, documenting, and actually putting things back in the same spot

12:25 that they came out of. So, first step, take a picture of your current pad location placements on the back plate and also on the front of where your GPU die is going to be exposed. The next

12:37 thing that you need to do is one by one remove the old pads. Now, these are, of course, like I said, good pads. I'm just going to show you how to do it. I'm not going to actually change my pads because

12:46 these are quite expensive and they're already done. So, one at a time, we're going to remove a pad and document where this pad was. And then in like a photo editor or something like that, go in and

12:59 type the millimeters that that pad was in that location. Get a like plastic sandwich bag or something like that. You can kind of put these on there and that keeps them fresh so you can peel and put

13:09 them back on again if some of them are needing to be replaced and some of them are not needing to be replaced. Do beware, you usually cannot just buy one and done unless you actually buy a set

13:20 that is made for a specific GPU. You're usually going to need to buy 1 millimeter, 2 millimeter, and possibly 3 millimeter. That will give you what you need to be able to get to the various

13:29 different depths. And for most of these, you should look for a semifirmer that will have a better longer lifespan versus something that's very flexible. So, the first thing you're going to want

13:39 to do is zero your digital calipers. And then go ahead and open it up maybe a couple of millimeters here. You don't got to go too crazy. And we're going to pull one of the pads off here. And so,

13:52 I've got it set at two. That's probably not going to be correct. Let's see here. Maybe kind of hard to see because let me see if I can do it against the black here. That probably is better for you.

14:03 So definitely two is too big. So the next thing that I'm going to do is probably take it down to 1.5 or so. Maybe just a little bit bigger than that.

14:17 So that's 1.53 millimeters. And yeah, that looks like this is about a 1.5. So the next thing that I'll do is go down to 1.2. 25 and we'll see if it fits in there. Just

14:32 a little bit up. So that's 1.32, but I have a feeling it's not going to fit. No, does not fit. So we've got a 1.5 mm here. So you would

14:49 take this, put it on your plastic bag or whatever so that you can reuse it. So that's 30 mm by 15 mm. So document your dimensions and put it down for each one of the pads. You can get the calipers

15:05 like this from Amazon. I got mine really cheap. Got a link to those in the description below. If you look at the chips here, you can see usually the pressure points where these are making

15:14 contact. This looks like everything is making contact very well with this. And so now we're going to take off the retainer. And I wouldn't take one side off all the way fully at any time. I

15:24 would just kind of work in a pattern around it. Kind of work from the back and slowly pry up. Textbook pump out. You can see this is

15:45 more viscous. What's remaining? What very little is remaining on the die. All of the solid stuff is moved out to the side and just kind of blown out. You can really see it just like blew out.

15:57 Definitely if you look at the heat sink here, there's like nothing in the middle here. So, very little viscous stuff around the sides and then just the blowout on the edges here. So, that is

16:07 exactly what I was expecting to find. And this is exactly what the new face sheet will fix. You're going to want to remove any big chunks and then carefully,

16:18 and I say carefully for a very specific reason, you don't want to get the isopropyl rubbing alcohol on the thermal pads at all. So, you might even want to remove the thermal pads if you don't

16:28 have very steady hands, but you want to give this a nice clean off with some 91% or greater isopropyl rubbing alcohol. And if you're still pulling stuff up, keep going. So, still pulling stuff up.

16:42 We're looking pretty darn good here now. And work with the grain and against the grain. And the final one, you might just want to do dry. Get all the remaining

16:51 isopropyl off there. Although, if you're using 91 plus, it'll evaporate. Don't use 70. It'll leave a lot of residue behind when it evaporates eventually and it'll evaporate very slowly. Bingo.

17:05 Cleo. So, in between each one, go ahead empty it out. And then even after it looks like it's finally done, I'm going to use one more little light pass with the

17:40 brush here, kind of brush stuff up onto if there's anything left or the top of the GPU chip. An old toothbrush that is a soft bristle

17:51 brush can work here also. Or well, you could go buy a new one actually that's probably a little cleaner. And if you're still picking up even a little bit of residue, keep at it.

18:05 Some areas you might want to dabble. All clean. Now, let's get that phase sheet applied. Okay, 15 minutes out of the freezer here. and back into the freezer. So, great

19:17 news, you can get two out of one sheet. So, that can save you a lot of money right there. Nice. Xacto makes this a lot easier. And there

20:16 we go. It's actually going to be a little bit better if you go ahead and get them put on here on one side because they're going to try to fall off. I just tried that. So, I actually removed them

20:27 from here, put them back on, lined them up as best as possible with their little indention marks, and now it's time to get this hopefully situated in one perfect placement.

20:43 There we go. That was actually pretty good. And now for the retainer. Yes, this all looks very good. So, at this point, we're reassembling

21:38 everything basically in reverse order of how we took it apart. This is really why I was telling you you need to take good photos of everything going into this, including possibly even doing some

21:48 handdrawn little sketches or notes. Those things are going to really save you here because every GPU is different as far as the screws where they go. So, while this is a good guide, especially

21:58 if you had a MSI 390 Ventus 3X OC, this is not universally applicable to every single GPU. However, every single step of this process, every decision I made, everything that we did is replicatable

22:13 on your GPUs. And I want to give you a big congratulations if you are actually doing this. This is a lot of work, but at the time that this is, there is just no availability of GPUs for cheap. So,

22:25 you really need to keep your GPU in tip-top shape, and doing proactive maintenance like this is the absolute best way to do that. All right, time to power it up and find out if there's

22:38 going to be any magic smoke. Everything seems to be working. I think we're in luck here. Fans all spinning. Nothing smelling weird.

22:55 Weird noises. Zero. Okay, I think we've got this. Awesome. Let's dive in and check out what the idols are looking like. Then we're going to load it up and then we're going to finally do our

23:06 actual like cost analysis of what it would cost if you ended up with a broken GPU versus the incredibly cheap maintenance that you should be doing on your GPUs. Now, especially if you're

23:16 seeing signs that you've got problems, I wanted to show you the before and after of the idle, and then we're going to test it out under load after this. But at idle, this was from 829. This was

23:26 when we were testing out the Quinn 3.8 a next flash. And this basically was about 41 to 45 centigrade that the GPUs were idling at. So pretty high, right? And clearly there was some problems going

23:41 on. Well, I've got great news. The great news is they are now at just 30 centrade and 29 centigrade. So 30 to 29 centrade. That's insane. That is absolutely perfect. I could not be happier with

23:57 that result. Now, I did actually install them backwards. So, this is A B C D instead of A BCD. So, these are in reverse order here, but still you can definitely tell A94 was the one that was

24:10 having the problem before. So, that is GPU 2 down here and that is idling at 30 centrade. We'll keep our eye on that as we're running it to just make sure that everything stays good with it. But, so

24:21 far, got to say, very happy with what I'm seeing. So, let's get up and running with our VLM, a punisher of a workout. And you'll also note that I do not have these capped right now. Basically, they

24:35 are at their full 350 watts. Now, usually I'm going to have these capped down at 250. And you can do that with the commands that I showed you or by going to

24:44 digitalspaceport.com/gpuyrienmaintenance. And you can get information there on how to set that up. So, while that's kicking up over there, let's get logged into our Hermes.

25:00 And we're going to just go ahead and have it do the spin up 10 sub aent swarm and research and update all the information you can find about the new AMD Halo Workstation. A pretty good grer

25:10 here for it to run through. Looking about like 225 or so on the power, up to 290, 300 now on some of these. So, definitely not capping the power on them. We're going to see what the temps

25:24 look like here and see what the fan response looks like as it ramps up here and see especially what we can hold as far as the GPU's performance here. We're definitely seeing 1980 MHz, which is

25:38 great for these. Of course, the MIM is going to just stay kind of flat at 9500, but seeing 1980 and it holding flat, we'll see if it can keep that up for a decent amount of time. hopefully seeing

25:50 350 and 325 being used on the GPU. So, they definitely are feeling like they've got the headroom right now to run it and they are running it. All right, now

26:01 we're crossing into 120, 130, 140, 148, 155 decode tokens a second there. And not really seeing that big of an impact on the temps here. fans hanging around 70

26:16 to 57. Not seeing the GPUs do crazy stuff, big dips down, things that I am looking for to see whether or not we have an actual problem with any of the GPUs outside of the pumpout. Looks like

26:28 we got this problem caught in time and corrected and that is a huge win for us. All right, we are 30 minutes in now. I thought that would be a good amount of time for us to kind of get a picture of

26:39 what we're seeing, whether or not we're seeing any abnormalities. And everything looks like it is doing really darn good here as far as the temps, as far as the boosts, as far as the fan speed, and the

26:51 temperature is really, you know, it's pretty much just capped at 80 centiggrade. It hasn't really gone over that. So, most of what we've been doing uh for the past 30 minutes, a lot of

27:00 prefill. If you check here, peaking at about 18,434. That's a lot. And if you check on the decode side, I guess we have that one there. 208 decode. Although we might be

27:13 hitting a new one here. We're at 203 right here. 207 and it's just going to stall out a 27. Darn. I was hoping to catch that one live. But definitely massive amounts of work has been going

27:24 on this past 30 minutes and the GPUs are looking great. So, the next thing that we're going to do is take a look at the cost breakdown of how much this cost me versus how much it would have cost for

27:34 me to go get a used one of these 3090s on eBay. So, perusing eBay, the lowest price for a non-completely broken 3090 right now is $1,300. There are some that are $11.49 that are

27:50 overheating. Uh, still not a great price, but $1,300 for non-broken. If we're looking for the Ventus here, you can see those are $1350 plus uh $31 shipping. Let's toss that into our

28:02 maintenance spreadsheet just so you can get like kind of a ballpark for why you want to make sure you don't have your GPU break right now. And it's very important that you do whatever you can

28:12 to keep your GPUs alive if they are 3090s or worth as much as a 3090 or a 4090. So I ran through two scenarios. One is just one GPU repasting here. That's really all I had to do here

28:23 today. And I also gave us just I guess worstcase estimates for hours. Like two hours is pretty outside the worst case most likely for you. And I just threw a rate in there of $25 an hour. I don't

28:37 know. Whatever you make, do you can follow along and do this yourself. $50 of total time spent or uh you know, if you're looking at a four full repads and repasts, eight hours, $25 an hour, $200.

28:50 We'll take a look at both of them. So, with one GPU, $1,300, you're looking at $14 right now for the phase sheet PTM. That's good enough for two GPUs, by the way. And the time spent, we'll say 50

29:03 bucks. That comes out to $1,236 that you're saving by not breaking a GPU. It's significant. If you look at four GPUs right now, uh $1,300. Granite. I mean, I'm just going to have one in

29:16 there because if you're going to buy the repad stuff for one, you kind of end up with almost enough for the repad stuff for four if you're buying the 100x 100 sheets. Now, there are cheaper ways you

29:27 can buy the kind of smaller ones from Thermal Grizzly. Definitely, uh I would recommend their their pads are probably what you should go with at this point in time. I've got quite a large variety

29:37 that needs to happen on these GPUs. So, sometimes stacking is what you have to do and you know, maybe a 0.25 25 and a2 or something like that to get like to a 2.25

29:48 and time $200. Still still saving $632 versus just one if one broke. So, it makes a lot of sense to not have this go south on you if you have a nice GPU right now. This is the entire message of

30:05 this video. I have checked pretty extensively all the big creators, all the medium creators, all the search terms for GPU maintenance that are out there. I've watched their videos. None

30:16 of them are really good to be honest with you. Most of them have been insanely specific about one GPU here or there. There are a couple bigger creators that tried but kind of failed

30:25 in my opinion at doing a good job at a generic. This is how you take care of your GPU if you have a GPU. And these are the modern things you should be putting on it like the phase sheet PTM

30:36 looking at some of those huge ones out there especially in the tech scene. But definitely I think this video smokes them as far as comprehensiveness covering every single little nook cranny

30:46 of what you should be doing, what you should be cleaning, and how you should be doing it. So hope you've learned a lot through this. I have certainly put a lot of effort into it. So I hope you

30:54 don't mind hitting like, share, and if you want to also consider joining as a member, I give you a hey chat tip for that. And if you are looking to get out a coffee for me, you can check out the

31:05 buy me a coffee page linked down below. And also thanks to our patreons and just everybody out there that shares this out there. That is one of the best ways to amplify this. And I do appreciate that.

31:14 If you're looking for some more insane builds for local AI, check out our playlist here, which is chocked full chocked full of great hardware and different stuff that you can put

31:24 together to get your local AI smoking.