Mac Studio M5 Ultra is a DGX Spark Killer for Local AI

0:00 Apple has announced a I'm gonna say it spark killer. And this is actually the first time that I've seen something that looks like it would actually be a very very compelling from a value standpoint

0:12 argument offering from Apple at prices that are well insanely high for sure. And it depends on what you're after with your local AI workloads for sure. But definitely the performance I do expect

0:26 to be roughly double what you're seeing on a DJX Spark easily as far as prefill. But I do expect especially the text gen to be insanely faster. As a matter of fact, it's going to be fast as most like

0:40 faster than a 4090 on textG and that it's going to make it a very interesting value proposition. Now I want to talk a little bit about like the bigger broader scope of economics for local AI. what

0:54 people are probably interested in, some of the trends that I see that are disturbing, some of the things that I think are good, some of the things that I think are value propositions you

1:02 should consider, and also the economics of where we're heading, not just in the United States, but globally as inflation is going to continue to trickle up and where we see probably ourselves in

1:14 October time frame when the M5 Max Studio Ultra 512 GB is actually available is going to be someplace that I think you would be interested and potentially if you are into local AI

1:26 considering upgrading into that system. So let me frame out a little bit of what I think is important to consider going into this announcement. And one of the most important things that I think is

1:37 important to consider is budget. I I you'll note this channel does not spend infinity money on local AI endeavors. That that's very counter to what other channels do. They actually I mean I let

1:49 me say it is not in the benefit of this channel to also do that. You get huge views by having the latest greatest product that you can review always. This means you chase always that new product

2:00 that also gives you sponsorships all sorts of other things. And if you think about like where you see channels become they're just product channels. This is exactly where it happens. And so huge

2:09 hat tip to all the channel members, everybody who has liked, subscribed, shares this out there. You help keep this voice very independent. There's always something that comes along with a

2:17 gift and gift is a very interesting way of looking at things like sponsorship, but there's certainly certainly tremendous amounts of money to be had there as well. And it really does become

2:27 a game because you have to keep numbers up to get valuation. Keeping numbers up means what do people want to watch? People always want to watch product reviews. The biggest channels on tech

2:37 YouTube are always doing product reviews. As a result of that, that changes what a channel does. The composition of a channel becomes quite different. This however is something

2:46 that I think makes a lot of sense if you're looking at the new M5 Studio 512 gigabyte for local AI at even its insane price. Not just for one consideration factor and that is its performance will

3:00 be probably very acceptable for prefill but to get comparable systems at 512 GB with 1.2 terabytes a second of effective bandwidth. It's it's very very very very very cost ineffective to do it any other

3:15 way. And if you so when I rebuilt this rig recently and this also talks about the reality like do you need 512 GB? Do you need 128 GB? So here here's my breakdown on that. In my way of thinking

3:28 about things my productivity on things and what economic value I generate for myself factors in incredibly heavily to my purchasing decisions of everything. And if I I like I know am I generating

3:43 $100,000 of economic activity with this machine right here? Well, no, I'm not. As a matter of fact, I'm generating next to no economic value for myself. I am definitely generating quite a bit of

3:54 enjoyment. It has uh changed the way that I do almost everything in my life and I am doing some app development and cool things, but I I think I encounter the same problem everybody does. Am I

4:05 going to be married to the slop that I ship? And that is a big hurdle to cross especially if you've ever owned a business before and you understand things like business insurance and the

4:13 necessary uh kind of things that can happen along the way if there are things like breaches. So you have to really think about what kind of a product you would ever put out there. Also not not

4:22 not every kind of product is something that you want to slop together and stand behind and purchase insurance for which is also something that's a very high cost. So one of the things I would urge

4:32 everybody in the channel in general to think about about every system regardless AMD, Apple, Intel, Nvidia is what is the economic value proposition. You should not have infinity budget for

4:45 this hobby. Approaching a hobby with a budget is a really good thing. I like to actually joke. I was joking recently about this, but I mean it's not actually a joke. Like I drive a 2015 paidoff CX-5

4:57 and I love that car. It's in great shape. It is a still fun car to drive for me. It does not have all the latest and greatest in anything and you know I have not had to make payments on a car

5:08 and so that really has funded quite a bit. You would be surprised how much money it is for car payments, extra insurance when you've got car payments for uh you know upgraded insurance

5:17 that's required. That savings monthly really compounds quite a bit. And if you apply things like that to your thinking, you can make a lot of economic things happen for a hobby like this in

5:28 particular. Should you be spending $10,000? Should you be spending $5,000? Should you be spending $1,000? Should you be spending 20 to $100,000? Which I I think this part disturbs me not from

5:42 the fact that people are spending it, but from does it actually make economic sense to even be doing that for that person locally with things like Kimmy K3. And Kimmy K3 is a big model. And

5:53 there's one reason that I would say as far as what we see as far as the size offering of the Ultra capping out at 512 here. That's going to be a little bit of a miss. That's going to turn out to be a

6:05 little bit of a miss. This is my predictions here. But that's going to turn out to be a little bit of a miss for one specific reason. the price point 20-ish thousand or something like that

6:14 is going to be where that big big boy lands is going to be first off you're you're not going to be able to run the latest and greatest of October. So this is coming out in October time frame.

6:25 We're already in three trillion parameter models. Do you know how big that actually is? So you're going to be running quants first off. You're going to be running quants. that impacts the

6:36 performance capabilities of precision, especially for things that are like you need every ounce of it. So, if you're running, let's say, a task that 95% of things are going to be very similar,

6:48 that's not going to be a problem to fly down the tunnel of that descending gradient and just land at a really good acceptable output. But when you are trying to do a one-off that may be a

6:59 little bit hard, kind of a needle in a hay stack or something that is looking for a bit more of the edge intelligence, then that is where you want full precision. And that's where all of the

7:12 systems that are 512 and under even like had 512 system have 512 system still over there not actively really using it for any of the models that are even in that size category. Why? Because

7:23 performance speed in the agentic age has just outclassed things. I've said this, I made a video on it. Uh your agent is basically the structure for your brain and it makes your brain so much better.

7:38 It's the difference between a really disorganized mind and a insanely focused organized mind. Your outputs become much better. Things like Quinn 3.8 27B running at FP16.

7:51 Nobody out there can credibly say that that's not substantially the best way to run it. It is substantially slower. Q8 has been a letown. The Q4 has been amazing. I mean, for what you get for

8:04 the Q4, that is absolutely mind-blowing. Still, the FP16 is going to smoke the Q4. It's going to smoke the Q8. Like, nobody can argue that. Not even close. But it does come at a cost of slower and

8:18 needing 96 GB of VRAM. I really do think there's an argument to be made for if you can find yourself in the for a foreseeable future in theund 128 to maybe 168 172 maybe up to 192

8:35 gigabytes range. You're going to be able to run insanely capable models. Today you can run on 96 gigabytes the full context window of Quinn 3.827B at FP16. I do that and as a matter of

8:49 fact that changed what I would consider for spending like this machine is now optimized for throughput of tokens with faster CPU and completely not optimized for RAM system bandwidth which the Epic

9:05 system provided substantially higher RAM system bandwidth. I've made the executive decision myself not important to me because what is important to me being able to generate TG at extra

9:18 tokens per second which faster CPU definitely does and 4.3 is a whole gigahertz higher than 3.3 3.25 25 or whatever the 7702 is. Lower system bandwidth overall though. Now I know

9:33 free token the video that I just did definitely could change all of that and I am putting that back together over there. So like all the rigs exist still. I'm just shuffling stuff from one place

9:43 to another. This rig here though with the Thread Ripper on it uh for Quinn 3.827B 827B is insanely well optimized and for anything that can fit in 120 GB of VRAM footprint if I have all five of

9:57 the GPUs running on it insanely optimized. So the Mac Studio coming in at 1.2 terabytes a second is going to provide very very good text gen performance. I

10:11 mean very good text gen performance throughout its polycontext window. Will you see its prompt processing, which has historically been the weak edge of the M5 lineup, uh, be better? Like, I think

10:23 you're going to see it be actually quite better. I think you're going to see 2x the DGX performance. However, this comes with an important caveat. That would be 2x the DGX performance for on one device

10:36 of the DGX size, 128 GB machine there. If you wanted to get to 512, you're going to need to have four DGX sparks and you're going to need to have the Sunundry networking. You're looking at

10:48 $20,000 by the time you wrap all that up or in that ballpark kind of range. And you're going to take impacts on your performance as you're sharding across a network that's just going to happen.

10:59 It's not going to be huge, but it will be a little bit of impact that you're going to see. That right there is why this is a better offering than four DGX Sparks. Now, Nvidia has a lot of

11:11 opportunity to pivot here. And I'm going to guess Nvidia is thinking a lot about pivoting today. There are other really interesting like the Zyomi uh 160 gigabyte local AI box that is coming out

11:24 that also has 1.2 uh terabytes per second of bandwidth on it. So, that should be a really great local AI performer. And I think most of the people that are actually So, we haven't

11:36 really seen this yet. This is why GPUs are still so inflated in price. GPUs are so inflated in price because nobody's selling a bunch of GPUs off to go fund their blank blank blank all in one box

11:48 for the most part because the performance has not been there yet in a compelling enough manner that makes it really like I'm going to go do this today. It doesn't make sense to do it

11:58 today. Even like 3090s 3090s are I mean I was 650 when I bought mine three three and a half years ago or something like that, maybe four years ago. I mean, when you think about the price that they're

12:09 at now, $1,200, that's double when you look at what the value that you or like if you're looking at buying one of the existing apples out there. And this is probably going to change a little bit

12:20 here and be impacted by this announcement because most people are going to not probably be like, I want to spend double what I paid for the prior generations 512 Mac Ultra, 512 GB, uh,

12:31 which is what you were able to see, like doubled the price. When you look at certain GPUs, they just are substantially more expensive on secondary markets, including things like

12:41 4090s from where they were when you were looking at them like a couple of years ago. We've had es and flows. We've certainly had some crazy crazy times. If you remember back to like December 24

12:54 going into like January 25, that was like another crazy time for GPU prices, but they kind of bounced back down. But this might be the first time that we actually see such a compelling offering

13:06 that people are like, I'm going to sell off all sorts of GPUs. So hopefully we can see GPUs like 3090s actually come down in price back into the 700 or something, 750 range. I would expect

13:19 that that actually could happen. There are 3090s out there still. There are a lot of 3090s out there still, but there are definitely different classes of buyers. So, when you look at like who's

13:31 buying what, much of what's been bought by people has in the all-in-one realms really been biased a lot by what is the wattage efficiency of it. What is the level of effort? So, like everybody

13:43 blows past this one. This is the biggest one, by the way. Most people out there are like, I want to buy a box, plug it in, it works. Like, that's I would say 80% 75% of the people out there want to

13:55 do it like that. They do not want to actually have to put something together, tinker with it, have a random thing break here or there, deal with that. Like it's a different mindset for the

14:03 person that just wants set it, plug it in, go versus build it, tweak it, make it my thing, upgrade it over time. Like you're not going to really upgrade your all-in- ones over time. upgrade it over

14:14 time, be able to adapt it, have it do versatile things in every sense versus fairly specific, not necessarily the best in everything, but a good all-arounder in everything usually. So,

14:26 there's a different class of who buys these machines. People chastise people over this like all the time, and I really I have no tolerance for that. Like, that's just ridiculous because

14:36 people of course have substantially different expectations of what they want to do. Now, there's also a lot of people that are in constrained environments. This is square footage like this is

14:46 square cubic footage here and cubic heat generation off of that cubic footage for sure also. So cooling uh some of the electrical that you could end up needing to run like forward done this build huge

15:00 construction experience in my background like huge construction experience in my background when I was way way way way younger guy but know how to do all this stuff like ran my own 125 amp sub panel

15:10 rewired my entire house like did all this stuff. This is why this channel is quite interesting is because I take you along those journeys sometimes and show you stuff that is hard to understand if

15:20 you have not been there, done that, and seen it done by somebody else. Good reason to hit that like and subscribe button. But definitely when you look at where somebody can be at in this kind of

15:31 a setup with a bunch of GPUs, like two pretty powerful uh PSUs on our 120 VAC US standard, I'm looking at, hey, that's one circuit and that's another circuit. Now, they're both going up to the 125

15:45 amp sub panel that's in my garage that I installed, but they are running on separate circuits. Do they technically have to? Well, most of the time I haven't seen any load that has, you

15:55 know, pulled that heavy and I have everything capped on the power side on the GPUs because really dropping it from 350 on a 3090 down to like 250 on a 3090 for the wattage draw is maybe a 10% max

16:08 drop in your peak performance. Not that big of an impact really. And it is substantial wattage and also substantial heat that you're going to reduce. Those things are not necessarily factors

16:19 anybody under the sun wants to deal with. Like 75% of the people don't want to deal with that. They're just going to be like, "Buy me the box. I don't care if I pay a premium. I want to set it

16:28 down, plug it in." And you can certainly find links to what I think is the best DGX Spark in the description below, which is the Asus Ascend. I think that one has probably the best

16:38 characteristics. I think a lot of what has been a miss for Nvidia on this, it it has been a really successful, I would say, six months to maybe nine months of run for the DGX Spark as far as the

16:51 marketing side. Definitely have seen a lot of DGX Sparks given out. There's definitely heavily promotion uh in the influencer realms around it. So, expect all of your influencers, you know, I

17:01 don't have one. That's because my purchasing decisions have to make sense to me. And that may not make sense to everybody and it limits a lot of what I cover on the channel because does he

17:13 think it makes sense? May not make sense to that 75% of people who are like buy me the effing box and just set it down because I'm like let's tinker with everything. Let's make it all this

17:22 stuff. Like let's go crazy on it. Let's take apart like oh wires everything. Love that stuff. Not just like buy me a box, set it down and plug it in. I think that's boring for me. I don't know. You

17:33 let me know in the comments below what you think. For me, that's boring. I like messing with stuff. Now, I would say also the price that you're going to see for this makes just let me ask you, are

17:45 you ever going to generate $20,000 worth of economic activity out of what you're doing with local AI? The answer for like 95% of you is unfortunately no. So, should you do this? Well, I would say

17:58 you should maybe consider your budget very seriously and be like, am I spending money I have or am I spending money I don't have? And I would really really say you should not go hog wild on

18:11 anything like being kind of moderate with your budget is a really good thing especially if you don't have a lot of like recurring EV revenue like income that that's something you should

18:22 definitely factor in. I will say the Apples have increased in price on resale value, which is not untypical for Apple. As a matter of fact, it's probably going to be really hard to get the 512

18:32 gigabyte M5 Ultra, the big boy, is it going to be comparable to a bunch of RTX 6000s? I mean, no. If you can buy four RTX 6000s, get your power delivery for that, set that system up, but you're

18:46 looking at a class of machine that now the RTX 6000s coming and clocking in in the 15 to $16,000 price point each is going to be like I mean when you factor in the board and

18:58 the Thread Ripper and everything, you're looking at a 75K proposition. Are you are you going to generate 75K? Like no. Most people not most people not like almost all people. And if you think

19:11 about it, is 512 even enough? Like I keep saying this, is 512 even enough? No. 512 is not enough? We are in the trillion parameter era. And you're going to be wanting to run the highest quant

19:24 you can. And honestly, you want to run full precision whenever you can because man, it's effing the best. it is the best. Now, of course, we see the networking uh looking like 120 gigabits

19:37 per second on the bandwidth on this. So, I think that's the same that you're able to get right now with your Thunderbolt experience. You can cluster them using Exo. So, if you wanted to spend 45

19:49 50,000 and get about a terabyte, I do think there is a potential that you could run into some pretty serious bandwidth constraints. I think the DGX Spark Edge is out there, but really

20:01 Nvidia should have probably gone with a much higher amount of system RAM in the DGX Spark. 256 would have been a bridge that I mean, they could have stomped the competition so hard that it may not have

20:15 been possible for them to come back. I think Nvidia, you're going to have to see a relaunch or a upgrade. Maybe kind of like GPUs get supers every now and then. Um, I think you're going to see

20:25 something come down the pipeline pretty soon. And I think the DGX Spark for 128 gigabytes at the time really, you know, met where the competition like the Halo was at, smoked it of course and I think

20:38 that it, you know, really has had a lot of success regarding thing regarding its uptake specifically around that. But it's not going to stay something that is compelling right now. You can read

20:49 through X today and people are like where like I'm g I'm upgrading to this like should I sell my sparks like a lot of people talking about it should you sell your sparks to buy this. I mean I'm

21:00 not going to tell you what you should and shouldn't do but if I was a spark owner myself which I'm not because I would not organically have bought a spark. See how that is somebody who does

21:10 not have industry influencing them to influence you versus just this is my feelings. these are the things I do because economically they make sense to me versus the other way. Like it's a big

21:24 difference, right? It's a huge difference if you think about it. And you know, you know, you know exactly what I'm talking about. But I think there is definitely an interesting what

21:33 happens next with the Spark lineup. I was just saying yesterday Spark 2. Somebody came in, probably Spark owner said, "Nah, they have no incentive to do any Spark 2. Not going to happen." I was

21:43 like, "Okay." I don't think they knew. Uh I don't think they were aware, but like this is definitely a reason for Spark 2 or Spark 1S or something like that with an increased amount of RAM.

21:55 And probably I mean one of the things that really was interesting about the Spark lineup to me was there was kind of a self-inflicted. It's just so small. Fans have to run real high. It's going

22:06 to generate too much heat. And like it was just the thermodynamics of it that was apparent right off the bat. You couldn't have that performance characteristic in that size footprint

22:14 without having that be the side effect. And I'm very on to Nvidia for the 2 plus 6 uh pin design. So I think, you know, making good thermal decisions is something that should never be

22:28 second-guessed. So hopefully Spark Refresh is actually a little bit larger of box all-in-one and has better cooling capabilities, better cooling characteristics, less fan noise when

22:41 cranking hard. I think those things are very important. A lot of people like to, you know, kind of draw another interesting metric that is like what is the amount of compute that's able to be

22:52 done? I think that's a very valid uh metric and usually this comes down to discrete GPUs still winning. Discrete GPUs do not win against the server class like you know big big systems like the

23:06 data center systems. Of course uh the scales of efficiency just scale up as you go bigger and bigger as the actual amount of work that is actually produced or capable of being produced on a token

23:20 side is substantially higher. You're going to find that is going to almost universally be limited by system design and generation. That is just a characteristic of most of the

23:29 electronics. And you see we are in a firm Gen 6 SE segment with the new Apple lineup here. And we should see probably whatever is coming down the pipeline for any competition that is serious being in

23:43 that same pipeline as well. So even if you are on like a gen 4 like 3090 4090 gen 4 class DDR4 class can kind of match up with that very very well RTX 6000 like I have a Thread Ripper Pro with a

23:59 WRX90 platform and uh DDR5 ECC and I I I look at that and I'm like am I going to buy RTX 6000s? No, I'm not going to buy RTX 6000s. Those things are too expensive. Like I'm

24:14 not going to am I going to generate $17,000 worth of economic activity with it? No. Even if I got a bunch of views on the channel, like am I going to generate like you see what I mean? Like

24:24 what does it make economic sense? That's how you keep your hobby going. How you burn out in a hobby is blowing all your money at once and not thinking about what I buy next to have fun next. And

24:36 like I think about that and like I'm more excited by some of the midsize and small size and older stuff as a result of that. That's kind of the home lab ethos honestly has been quite something

24:46 for a long time. So maybe that's just ingrained in me and buying latest and greatest is not the thing that I want to hype up and be that creator because there's that creator everywhere and that

24:56 creator gets a little bit old. So overall I think that you are looking at a spark killer even at $5,000 for the 256. I think it is going to be a spark killer and I look forward to reading

25:10 your comments below. I think discrete GPUs still are for the people who want maximal performance and I still think they will absolutely. Your RTX 6000s are still absolutely going to smoke this.

25:21 You're looking at a much much more expensive system. Can AMD Hello AMD where are you at? Can AMD step into this realm on the software side really is where it's got to happen. Can they step

25:32 into this realm and offer something competing? Yes, they can. They have the bandwidth. It's just that software side stack of the stack has been a little bit wanting. Can Intel show up?

25:45 I don't know. They they release things that get so much hype and then there's just no supply and then there's just so what happens most of the time it pers out. I mean, I don't see Intel really

25:55 playing the game like the game would have to be played if they were really serious about winning the game. And that's kind of been what Intel's been doing for the past several years. At any

26:02 rate, coasting on legacy, I guess, could be one of the things while AMD starts serving and eating the cake non-stop. I think this is a pretty serious competitor. And I think it's something

26:12 that if you can get in even whatever price it is, it probably appreciates in value quite a bit on the 512 gig side. Even the 256 gig side actually makes sense versus DGX Sparks if you're an

26:24 all-in-one buyer. Does it make sense compared to a 3090 setup? Does any of this makes sense compared to a 3090 setup? Here's why it doesn't. Like this is a critical thing. Like you need 96

26:34 gigabytes to run insane. You're going to have a good time uh intelligence in your home. Like the the takeaway here is like so anticlimactic, isn't it? This is not me telling you to go buy anything. This

26:46 is me telling you you can run 3090s and get there also. Yeah, you can run 3090s and get there also. And actually that's there's other ways to do it also. There's a lot of other ways to get

26:58 there. But being able to hit 96 GB of available system VRAM today gives you a capability set that is beyond what you're going to ever be able to probably fully utilize yourself. and being able

27:11 to scale that to double in the future. Probably will be able to hold that parody. That's my plan. Probably be able to hold that parody for the next several years. Probably year, year and a half,

27:21 two years. That's my take on it. Is it a downer? Is it an upper? No. Just a reality check. So, if you're looking for more reality checks, be sure to hit like, subscribe. Huge hat tip again to

27:33 all of our members. And you can check out the video if you are looking for more builds and you are that tinkering person and you want to get into things like home lab and creating huge high

27:42 performance storage arrays and stuff like that in our playlist that we've got over here.