Before You Run AI Locally, Watch This

0:00 Picture a laptop with a 6-GB graphics card. Ollama lists Qwen 3 8B at 5.2 GB. 5.2 is less than 6, so it fits, right? Maybe. Ollama's own docs show what a near miss looks like.

0:14 Their example is a different model, but the status command Ollama PS reads 48% CPU, 52% GPU. It loaded, and about half of it ended up on the slow side of the machine. And the 5.2 on the listing

0:28 couldn't have warned you because it only counts part of what gets loaded. The download size is the furniture. It's the model's weights, and that's all it covers. The conversation brings boxes

0:38 called the KV cache, and they keep arriving as the chat gets longer. Then you need room to walk around. Your graphics card doesn't care what the download page promised. We'll follow

0:47 three example machines the whole way through. A laptop with a 6-GB GPU and 16 GB of RAM, a desktop with a 24-GB GPU, and a Mac with 64 GB of unified memory. These are example machines, so any

1:02 number I give you for them is math from the official docs, and it'll say estimate on screen. By the end, you'll read a model listing and estimate whether it fits. You'll tell the two

1:11 kinds of slow apart. You'll pick between Ollama, LM Studio, llama.cpp, and MLX, and choose a setup for each of these three machines. Fitting and speed are separate questions, and we'll answer

1:23 them separately. Start with a real listing. On Ollama's Qwen 3 page, Qwen 3 30B is 19 GB with a 256K context window, and Qwen 3 32B is 20 GB with 40K. They're a gigabyte

1:38 apart. Load them into the same truck, and they take up almost the same floor space. Most of that size comes from parameters. A parameter is one number the model learned during training, and

1:48 the model stores billions of them. More parameters usually means a bigger file. On the same page, the 8B tag, 8 billion parameters, is 5.2 GB. So, why would two models of nearly the same size behave

2:01 differently? The 30B is a mixture of experts. According to Quen's model card, it stores 30.5 billion parameters spread across 128 experts. But, for each token, only eight of those experts switch on.

2:14 That's 3.3 billion active parameters per token. A dense model like the 32B uses every weight for every token. The mixture of experts works more like a 128-person

2:25 office where eight people attend each meeting. You still pay rent for the whole building. That's the first split this video keeps coming back to. Memory has to hold all the weights, active or

2:35 not. The work per token only touches the active ones. So, the 30B and the 32B ask for about the same truck and very different amounts of work per token. A token is the unit the model reads and

2:47 writes, a word, or a piece of one. Context windows, cache sizes, and speeds all get counted in tokens. The 256K on that listing is how many tokens the model can keep in view at once. Remember

2:59 that number because it costs memory. So, how do 30 billion numbers fit into 19 GB? Quantization. Training keeps each weight at high precision, and quantization rounds weights to fewer

3:10 bits. Hugging Face's table for Llama 3.18B shows it. 16 GB at FP16, 8 GB at FP8, 4 GB at int4. Same model, a quarter of the furniture. The arithmetic fits on a napkin.

3:25 Parameters times bits per weight divided by eight gives you bytes. 8 billion weights at 16 bits is 16 GB. Drop to four bits, and it's four. That's the napkin version, and the listing's file

3:37 size is the real number you should trust. Names like Q4_K and Q6_K show up in GGUF files. GGUF is a format that packs the weights and a standard set of metadata into one

3:50 file. So, a tool can open it and know exactly what it's holding. Hugging Faces docs map each name to bits per weight. Q4_K is 4.5. Q6_K

4:02 comes out at 6.5625 bits per weight, which is a wonderfully exact way of saying about 6 and 1/2. More bits means a bigger file and less rounding. Fewer bits means a smaller

4:13 file and more rounding. How much quality you lose to that rounding depends on the model and the task, and the listing can't tell you. That's why the final chapter ends with

4:23 testing your own work and not just with a fit chart. Compatibility is the last thing on the label. The format decides which engine can open the file. LM Studio runs llama.cpp on every platform

4:35 and adds Apple's MLX on Apple silicon Macs, and some files need special builds. One giant model later in this video only runs on one specific engine and nowhere

4:45 else. The truck is your memory, and three kinds show up in this story. DRAM sits on the graphics card right next to the chip doing the math. System RAM sits on

4:55 the motherboard and serves the CPU. And on Apple silicon, unified memory is one pool the CPU and GPU share. On the laptop, picture a small truck with 6 GB of cargo space towing a 16 GB

5:08 trailer. Anything in the truck is right where the GPU works. Anything in the trailer can still be used, but those layers get handled on the CPU side. That handoff is the split you saw in a

5:19 llama.cpp. The desktop gets a bigger truck, 24 GB of VRAM, with its own trailer of system RAM. The Mac works differently. One long truck with 64 GB and no trailer. The GPU can use that

5:32 shared pool, although macOS limits how much memory the GPU can wire for itself. More on that later. Here's a reference ladder from 4 to 512 GB with every rung tagged by memory

5:43 type, graphics card VRAM, system RAM, and unified memory. This is just how much each one holds, not how fast it moves. At the top sits 512 GB of unified

5:54 memory, which one model page names as the target for holding a roughly 750 billion parameter model entirely in memory. Everything you load goes into three

6:04 piles: furniture, the weights, which are fixed once the model's loaded, boxes, the KB cache, which grows with the conversation, and aisle space, the working memory the software needs to

6:14 actually run. Estimators like LM Studio add these up for you based on the context and GPU offload you ask for. Why do the boxes keep arriving? Each time the model reads a token, it works

6:26 out some notes about it, called keys and values, so it doesn't have to redo that work when it writes the next token. Those notes get kept. More tokens in view means more notes and more boxes.

6:37 Hugging Face gives the sizes for Llama 3.18B. At 1,000 tokens of context, the cache is 0.12 GB, which is about a shoebox. At 16,000 tokens, it's 1.95 GB. At 128,000,

6:52 it's 15.62 GB. The weights at 16-bit are 16 GB. So, at full context, the boxes take up nearly as much floor as the furniture. Hugging Face says it directly, "Near the

7:04 maximum context length, the cache uses as much memory as the weights. You moved a sofa and brought a sofa's worth of cardboard." So, which context size counts? The one on the listing or the

7:14 one you use? The one your software reserves. The listing's 256K is the most the model supports. The tool decides how much room to set aside, and a larger setting needs more memory. A

7:26 Llama picks that setting based on your VRAM, counted in gibibytes. A gibibyte's just a binary cousin of a gigabyte, close enough here. Under 24, the default context is 4K. Between 24 and 48, it's

7:40 32K. At 48 and up, it's 256K. So, the same model gets a different size box pile on different machines before you've typed a word. One catch, a Llama's FAQ page still says

7:53 the default is 4,096 tokens, full stop. Two pages of the same docs, two answers, one very normal documentation experience. Trust what your machine reports instead. A Llama PS has a

8:05 context column that shows what actually got loaded. Back to the laptop and Qwen 1.3 8B, the file is 5.2 GB, which leaves about 0.8 GB of the 6 GB card for boxes and aisle space. A Llama's default here

8:20 is 4K context. For Llama 3.1 8B, 4K of cache works out to roughly half a GB from Hugging Face's table, and Qwen's cache will differ. It's tight. Tight is where things spill. When everything

8:33 doesn't fit in VRAM, tools can put some layers on the GPU and the rest in system RAM. The llama.cpp engine exposes this directly with a setting for the most layers to keep in VRAM. The model still

8:45 runs, just across two vehicles now. The desktop's 24 GB card holds Qwen 3 30B's 19 GB file with about 5 GB to spare. A 24 GB card sits right on the edge of a Llama's 24 to 48 band. So, check the

9:00 context column to see whether you got 4K or 32K. Whether 5 GB of boxes covers 32K depends on this model's cache per token, and that's exactly what an estimator calculates. The Mac takes Qwen 3 30B or

9:14 Qwen 3 32B whole with plenty of floor left over. If a Llama sees 48 or more GB for the GPU, it'll reserve a 256K context by default. So, check the context column

9:26 here, too. The more interesting question is a much bigger model. LM Studio has a command that estimates memory without loading anything, and its docs use GPT- OS 120B

9:38 as the example. Estimated GPU memory, 65.68 GB. The Mac has 64 in total on paper before the operating system takes its share and before macOS decides how much the GPU may use. A 65 GB sofa and a

9:53 64 GB doorway. And macOS is already standing in the doorway. That figure is LM Studio's example estimate, not a measurement on this Mac. That estimate isn't fixed. LM Studio's docs say honors

10:05 flags for context length and GPU offload. Shrink the context and the box pile shrinks, which helps when a model's borderline. Parallel requests push the other way. A llama says required memory

10:16 scales with the number of parallel request times the context length. So, the capacity question has three parts. Furniture comes from the listing, boxes come from your context setting and how

10:26 many chats run at once. Isle space comes from the software. Add them up, compare against the truck, and you'll know whether it fits. Fitting tells you whether it'll load. It says nothing

10:36 about how fast it'll move. A perfectly packed truck can still sit in traffic. So, speed gets its own half of this video. When you hit enter, a few weights happen in a row. Loading moves the

10:47 weights off the disk and into memory. Prompt processing reads your whole prompt and builds a KV cache for it. Then generation writes the reply one token at a time.

10:56 The pause between hitting enter and seeing the first word is called time to first token. It's mostly prompt processing and it grows with how much you paste in. A short question starts

11:06 quickly. A 50-page document means the model reads every page that fits in its context before it says anything. And on a 4K default, most of those pages won't fit. Generation is the steady drip

11:17 afterward, measured in tokens per second. For each new token, the model works through its active weights. That's where the mixture of experts pays off. 3.3 billion active parameters per token

11:28 instead of all 30.5. Much of that work is moving weights from memory to the chip, and LLM Fit, a tool we'll use at the end, builds its speed estimates on a memory bandwidth model. Capacity is how

11:40 big the truck is. Bandwidth is how fast you can unload it. This is where a spill hurts. If part of the model landed in system RAM, those layers get handled on the CPU side, and every token has to

11:50 pass through them. The model fits, technically. That's how you end up with a model that loads fine and then crawls. The usual first move is getting the busy parts back into the truck. So, when it's

12:01 slow, ask which weight is the slow one. A long pause before the first word points to prompt processing. So, look at prompt length and prompt caching. A slow drip after that points to

12:11 generation. So, look at where the weights live and how many are active. Different causes need different fixes. With capacity and speed separated, choosing software gets easier. There are

12:22 four main options, and they're more related than their logo suggest. Some are engines that do the math, some are apps wrapped around engines, and one is Apple's own framework. Start with

12:32 llama.cpp, the engine underneath a lot of this. It runs on many hardware backends. It can split a model between CPU and GPU when it doesn't fit in VRAM. And it exposes the knobs. How many

12:43 layers go on the GPU? Flash attention, the KV cache type, and more. Its server has a fit option that adjusts any settings you haven't set, so the model fits in your device's memory. Prompt

12:55 caching is on by default. If you want every lever and don't mind a terminal, llama.cpp is the most direct route. It also has an option made for mixture-of-experts models. Keep the

13:05 expert weights of the first n layers on the CPU. That matters for our 30B because only eight of its 128 experts do any work on a given token. Llama is the quick start. You pull a model by name,

13:17 like Qwen 3 30B, and it picks defaults for you, including that VRAM-based context size. Then a llama ps tells you where the model landed and how much context it got. It's the fewest steps

13:29 between you and a running model. The trade-off is that defaults decide things you might not notice. Context length, the CPU and GPU split, and how many requests run in parallel all move memory

13:39 around. Ollama lets you change each of them with environment variables, and the exact names are in the description. LM Studio is a desktop app with a GUI. It runs llama.cpp on every platform and MLX

13:52 on Apple Silicon. So, on a Mac, you can choose either engine. It also ships a command line tool, and that's where the memory estimate we saw lives. That estimate-only mode is why I'd point

14:02 beginners with big ambitions at LM Studio. You can ask whether a model would fit with this context and this much GPU offload before you commit memory to

14:11 loading it. MLX is Apple's array framework for Apple Silicon. Its arrays live in unified memory, so the CPU and GPU work on the same data in the same pool. MLX LM sits on top of it

14:23 for generating text and fine-tuning language models. MLX LM brings its own tools for our two problems. For prompt processing, it can cache a prompt so you don't pay to read it

14:32 twice. For memory, a rotating KV cache caps how many boxes you keep. And if big models run slowly, its docs say to raise the wired memory limit, a system setting. Side by side, Ollama gives you

14:45 the fastest start and the least control. LM Studio gives you a GUI, both engines on a Mac, and an estimate before you load. llama.cpp has every knob and the most typing. MLX is Apple Silicon only

14:58 and built around unified memory. None of them changes how big the truck is. Then there are tools with different jobs. Open Web UI is a front end, a chat interface in your browser that talks to

15:08 Ollama or to anything with an OpenAI compatible API. It doesn't run the model. Think of it as a dashboard bolted onto whichever engine you already picked.

15:18 vLLM and SG Lang are serving engines built for throughput. Many requests and many users usually on a GPU server. SG link describes itself as a high throughput serving framework. For one

15:30 person on a laptop, they're usually more machinery than you need. VLLM also has requirements worth reading first. Natively, it's Linux only on Nvidia GPUs with compute capability 7.5

15:42 or higher, which is Nvidia's version number for a card's feature set. On Windows, the install page's position is politely, "No." You go through WSL, the Linux layer built into Windows

15:53 instead. And on Apple silicon, you need a plugin. Free token is the newest name here. It's a research project with a paper built around mixture of experts models.

16:03 The paper's pitch is to stop thinking of your PC as one small GPU and use all of its memory as one pool. In practice, it moves experts back and forth between CPU and GPU memory, keeping the busy ones

16:14 close. The paper claims a lot. A 35B model on a laptop, 284B on a gaming desktop, and a 753B model on a single workstation GPU. Those are the paper's own results, speed

16:28 figures included, and they haven't been independently checked yet. It targets Nvidia RTX 30, 40, and 50 cards on Windows and Linux. There's no Mac support.

16:38 So, free token belongs in the past VRAM toolbox for RTX owners who want to experiment. We'll come back to what streaming experts actually costs. For a first setup, I wouldn't start there. If

16:49 you want one tool for now, here it is. Pick a llama for the fewest steps and LM Studio for a GUI and estimates. Go with llama.cpp for control and MLX if you're on a Mac and want Apple's own stack. Put

17:01 the same 30B crate in front of all three trucks, and by the napkin math, you get three different outcomes. One probably splits, one fits, one fits with room to spare. What decides it is where the

17:12 memory lives. On a Windows or Linux PC with a separate graphics card, VRAM is the fast truck and system RAM is the trailer. The goal is simple, keep as much of the model and its cache in VRAM

17:23 as you can because whatever spills out gets handled on the slower side. The laptop's 6 GB card is the tightest truck of the three. Models around the size of Qwen 3 8B sit right at its

17:35 limit. The 16 GB of RAM gives it a trailer, so bigger models can still run in hybrid mode with the CPU carrying the layers that don't fit. And with no usable GPU at all, llama.cpp can run on

17:47 the CPU alone. It works. The CPU does all the math reading from system RAM, so capacity is suddenly much larger and speed is what you pay with. The desktop's 24 GB card is the

17:58 comfortable middle. Qwen 3 30B's 19 GB file fits in VRAM with room left for cache. On this machine, the mixture of experts helps twice. It fits and each token only

18:10 activates 3.3 billion parameters, so the work per token is small for a model of that size. Swap in the dense Qwen 3 32B at 20 GB and it also fits with less room to spare. Every token works through all

18:24 of its weights. You get the same truck space and more work per token. If both fit, that's the speed trade you're choosing. On the Mac, the shape of the problem changes. There's no trailer

18:34 because the GPU draws on the same 64 GB pool as everything else. That's how a Mac can load models that would never fit in 24 GB of VRAM. Shared really means shared though. The

18:45 operating system and your apps live in the same pool and macOS limits how much memory the GPU can wire for itself. MLX LLM's docs tell you to raise that wired limit if large models are slow. How much

18:56 headroom a given Mac leaves is something to check, not assume. On the Mac, you've got both engines, llama.cpp directly or through LM Studio and MLX through MLX LLM or LM Studio.

19:09 Pick the model file made for the engine you chose since a GGUF and an MLX download aren't interchangeable. And then there's the person with two graphics cards. Two 24 GB cards give you

19:19 48 GB of VRAM on paper. In practice, you've got two GPUs in a group chat that never goes quiet. And how fast they reply decides your speed. Llama.cpp's default gives each GPU a contiguous

19:32 slice of the model's layers, like two trucks each carrying half the rooms of a house. The work passes from one to the other in order. And the docs say this mode slowly links between cards. There's

19:43 a faster, pickier mode, and we'll get to it later. So, for any machine, the question is the same. What lives in the fast truck? What lives in the trailer? And how often does the work have to

19:53 cross between them? That's the lens for making things faster. Speedups come in two kinds. Ones that shrink the load, so more fits in the fast truck, and ones that skip work. Some help prompt

20:04 processing, and some help generation. None of them comes with a guaranteed speedup. So, treat each one as something to test. Varying the volumes. Parsed lever placement in Llama.cpp.

20:16 A setting controls the maximum number of layers stored in VRAM. Put them all there if they fit. If they don't, the fit option can tune the settings you left alone to match your device's

20:25 memory. In a Llama, check the split in a Llama.ps and aim for 100% GPU. Mixture of experts models get a smarter version. Instead of pushing whole layers to the CPU, Llama.cpp can keep just the expert

20:38 weights of some layers there, while the rest of each layer stays on the GPU. You move the furniture only some trips need, and keep what every trip touches in the truck. Second lever, a sensible context.

20:50 If your chats are short, you don't need a 256K box pile reserved. A smaller context frees memory, which can pull spilled layers back onto the GPU. And remember the 24 GB band, or a Llama may

21:02 pick 32K for you. Set what you actually need. Third, quantization, going from Q6_K at 6.5625 bits to Q4_K at 4.5 shrinks the

21:15 furniture by roughly a third. That can be the difference between one vehicle and two. Test your output quality afterward because rounding has a cost. Fourth, flash attention, a leaner way of

21:27 doing the model's attention math. A llama's FAQ listed as a way to reduce memory use and llama.cpp has a switch for it. Any speed effect depends on your hardware and model, so measure it

21:38 yourself instead of trusting a headline. Fifth, shrink the boxes themselves. KV cache quantization stores those keys and values at lower precision. A llama's docs say Q8_0

21:49 takes about half the memory of F16 and Q4_0 about a quarter. On a long context, that's a lot of cardboard gone. Check answers on your real prompts afterward. Sixth, prompt caching, which targets the

22:02 first weight. If you send the same long system prompt or document again, a cache can skip rereading the part it's already seen. The llama.cpp server keeps it on by default. MLX LM has a command to

22:14 cache a prompt ahead of time. It helps most when the start of your prompts repeats, like a coding assistant with the same instructions or several questions about one document. It won't

22:24 help the steady drip of generation. That's a different weight. Seventh, speculative decoding, which targets the drip. A small draft model guesses the next few tokens and the big model checks

22:35 them. When the guesses are right, the big model can accept several at once. LM Studio's docs say the draft has to share the main model's vocabulary. The docs also say plainly that generation speed

22:46 might not increase and could even decrease. It's a performance feature whose own manual warns you about the performance. Run it with and without on your actual prompts. Multi-token

22:56 prediction or MTP is a cousin of the same idea where the model has extra machinery to propose tokens itself. The catch, per VLLM's docs, is that it only works for model families that support

23:07 it. You can't just switch it on for any model, so check the supported list before you plan around it. Group the levers by the weight they fix. A long pause before the first word calls

23:17 for prompt caching and shorter prompts. A slow drip calls for placement, fewer spilled layers, a smaller quant, and speculative decoding if it pays off. If it won't load at all, reach for context,

23:29 K V quantization, flash attention, and quant. On the laptop, here's the order I'd try. Keep the context modest, pick a quant that leaves room on the 6 GB card, and watch the split in Ollama PS. If you

23:42 need a model bigger than the card, accept hybrid mode, and expect the drip to slow down. What if the model you want is bigger than every truck you own? There are three ways past VRAM. Spill

23:52 into RAM, stream pieces from somewhere slower, or add GPUs. Each one makes something possible, and each one makes something slower. RAM offloading is that trailer on the

24:02 laptop truck. Hybrid CPU and GPU inference lets a model go past your VRAM, as long as it fits in VRAM plus RAM. On the laptop, that's roughly 22 GB of combined space instead of six, before

24:15 the operating system takes its share. The cost shows up in generation. Every token passes through every layer, including the ones in the trailer that the CPU handles. You've traded a crash

24:25 for a slower drip. For some jobs, like a long batch that can run overnight, that's a good trade. Mixture of experts models get a better version of this trade, which is why they keep showing up

24:35 in this chapter. Since each token only uses a few experts, you can park expert weights on the CPU and keep the parts every token needs on the GPU, using that llama.cpp option from earlier. Before

24:47 you push the boxes anywhere slower, shrink them. On a single card, Q8_0 roughly halves the cash, so fewer boxes have to leave the truck at all. With two cards, one split mode spreads the boxes

24:59 across both and we'll get there in a minute. The next step past the trailer is expert streaming. Instead of holding every expert in fast memory, you keep the busy ones close and

25:09 fetch the others when a token needs them. That's free tokens approach, streaming and caching experts between CPU and GPU. What it makes possible, according to the paper, is very large

25:19 MOE models on consumer and workstation RTX cards. The paper reports 39.3 tokens per second for a 35B model on an 8 GB laptop GPU, but anything fetched is something the GPU waits on, so time it

25:33 on your own machine. Then there's a warehouse down the road, the SSD. The developer antirez published a GGUF of Deep Seek v4, One Flash, a roughly 750 billion parameter model with

25:45 instructions for running it on a single 128 GB Mac using SSD streaming. The trick is that the full model never gets loaded into memory. It reads only the rows it needs from disk as it needs

25:57 them. The page names a 512 GB Mac as a target for keeping the whole thing resident and says to keep the file on a fast local SSD. Picture a librarian who won't shelve the encyclopedia and just

26:09 runs back for each page you ask about. That's a roughly 750 billion parameter model reading from a drive. The page lists no measured speeds, so this goes under possible with the speed left

26:20 blank. It's also a specialist setup. These weights need antirez's dwarf star engine, which runs them on Apple's metal, and they don't work with ordinary tools. Treat it as a demonstration of

26:30 what's possible and start somewhere simpler. Back to the two-card convoy. The multi-GPU docs in llama.cpp list three split modes. You've already seen the default, layer split, where the

26:43 cards pass work along like a relay. The other two modes are where it gets picky. Tensor split is experimental. It divides both the weights and the KV cache across the GPUs, so they work on each token

26:54 together. The docs say it favors generation speed, but depends on a fast connection between cards since they're talking constantly. The third mode, row split, is marked deprecated, so that's

27:05 one decision already made for you. If you don't know how fast the link between your cards is, layer split is the safe choice since it tolerates slow links. So, past VRAM, here's the scoreboard.

27:15 RAM offload gives you bigger models and a slower drip. Expert streaming promises much bigger MoE models at speeds only its paper has measured. SSD streaming reaches enormous models on a specialist

27:26 setup with the speed unknown. More GPUs add capacity and speed depends on the split and the link. Choosing comes down to estimating first and checking second. Two tools do the estimating, LM Studio's

27:38 estimate only load, which we've already seen, and LLM fit. That second one detects your hardware. For each model, it estimates whether it fits and roughly how many tokens per second you'd get.

27:50 Its speed numbers come from a memory bandwidth grounded in runtime sampling and community measurements. They're estimates and the readme says so. Treat them like a weather forecast. Useful for

28:00 packing and you still look out the window. The process works the same for any machine. Estimate the fit with LLM fit or LM Studio at the context you'll really use.

28:11 Load the model and check where it landed in Ollama PS or your tool's equivalent. Time your real prompts, then judge the answers on your real tasks. The laptop with its 6 GB GPU and 16 GB of RAM.

28:24 By arithmetic, models around 8B at 4-bit sit right at the card's edge, so keep the context small and check the split. Bigger models run in hybrid mode through the RAM trailer. I'd start with Ollama

28:36 or LM Studio and let the estimate tell you when you've crossed the line. One more laptop option to test is a mixture of experts model with its experts parked in system RAM. The 19 GB Qwen 3 30B file

28:49 won't fit in 6GB of VRAM, but only three. 3B parameters work per token. It's tight against 22GB combined once the operating system takes its share, so estimate that one before you download

29:01 it. The desktop, with its 24GB GPU, by file size, Qwen 3 30B at 19GB fits in VRAM with about 5GB left for context and overhead. Run the estimate at 32K, since that's Ollama's default for the 24 to 48

29:17 band, and lower the context if it spills. If you want more control, llama.cpp is worth the extra typing here. If the 30B's answers aren't good enough for your work, Qwen 3 32B is the

29:30 dense alternative at 20GB. It fits, too, with less room for boxes, and every token uses all of its weights. Compare both on your own prompts before you decide. The Mac, with 64GB of unified

29:42 memory, by subtraction, the 30B and 32B fit with plenty of room, so you can afford longer contexts. LM Studio's own example, GPT- OS 120B, estimates at 65.68GB,

29:57 already over the whole machine before macOS takes its share. Rerun the estimate at a shorter context if you're curious, but don't plan on it. And try MLX alongside llama.cpp.

30:07 For any machine on that 4 to 512 chart, the same rule holds. Find your memory type, leave room for the operating system, and add up the furniture, the boxes, and the aisle. The bigger rungs

30:18 just let you bring more boxes. Here's the checklist to keep. One, read the listing, and remember the file size is the furniture. Two, note whether it's dense or a mixture of experts, which is

30:29 total size versus work per token. Three, check the quants bits per weight and the format your engine needs. Four, set the context you'll really use, because that's the box pile, and parallel

30:41 requests multiply it. Five, run an estimate with LLM Fit or LM Studio. Six, load the model and check the CPU and GPU split plus the context that actually got loaded. Seven, time both weights, the

30:55 pause before the first token and the drip after it, and fix whichever one is slow. Eight, test answer quality on your real tasks. An estimate tells you it'll run and only your own prompts tell you

31:06 whether it's any good. So, what we learned today, a model listing tells you the weights, not the whole bill. Memory is weights plus a KB cache that grows with context plus working overhead. And

31:17 mixture of experts models need memory for all of their weights but only compute with a few per token. Tools set context defaults for you, so check what actually loaded. Fitting and speed are

31:27 separate questions and speed splits into prompt processing and generation. Past VRAM, RAM offload, expert streaming, and SSD streaming buy you capacity and cost you speed.

31:39 Extra GPUs add capacity, too, and how fast they run depends on the split and the link between them. My verdict, if you're a beginner on a Windows or Linux GPU, start with Ollama or LM Studio and

31:50 run LLM Fit before every big download. If you're on a Mac, start with LM Studio so you can compare llama.cpp and MLX side by side. If you like knobs, go straight to llama.cpp.

32:03 One question to leave you with, if expert streaming and SSD tricks keep improving, does how big is my truck stop being the first question and how fast can I unload it take over? Thanks for

32:13 watching and all the best to you guys.