Can You Fine-Tune a 27B Model on a Laptop? (I Did the Math)

0:00 The lid of your laptop says RTX 4090. The chip behind that badge holds 16 GB. The desktop card wearing that exact name holds 24. Same four numbers, 8 GB apart. That gap is why the fine-tuning guide

0:13 you read last night will not run on the machine you own. So, let us do the arithmetic out loud. Can a 27 billion parameter model be tuned on a laptop? Three walls stand between you and yes,

0:25 the memory bill, the escape hatch, and one repository that disputes both. Well, one gets skipped in almost every tutorial. Running a model and training one are different jobs. To run it, you

0:36 hold the weights and little else. To train it, the weights are the cheap part. Hugging Face publishes the receipt in its own docs. Mixed precision with AdamW costs 18 bytes per parameter. 6

0:47 bytes for weights because you keep two copies, eight for the optimizer state, four for the gradients. Multiply 18 by 27 billion and the bill lands at 486 GB. That is 30 laptop cards stacked before

1:00 you load a single training example. So, a full fine-tune on a laptop is a no and it is not close, which is why the whole field went sideways instead. Sideways has a name and a date. In 2023, Tim

1:13 Dettmers and three co-authors published a method called QLoRA and their paper states the reduction in a single line. A 65 billion parameter model went from more than 780 GB of GPU memory to under

1:25 48, one card. Same task performance as a 16-bit baseline they measured against. The trick is refusing to train most of the model. You freeze the base, squeeze it down to four bits, and leave it

1:36 alone. A frozen tensor gets no gradient and no optimizer state. So, 12 of those 18 bytes stop being charged. The optimizer states and the gradients have been the expensive part from the start

1:47 and now they only exist for the sliver of the model you are actually changing. That sliver is a thin pair of matrices bolted onto each layer. It rides alongside the frozen weights, learns the

1:58 shape of your data, and gets saved on its own. The scale of that is worth sitting with. The current 27 billion checkpoint is 55 and 1/2 GB of weights on disk. The adapter you produce from it

2:09 is tens of megabytes. You download a warehouse and you make a sticky note. That ratio is the reason any of this fits on hardware you can buy, and it is also the reason the result travels. The

2:19 paper carries three parts, and the middle one is the cleverest. A 4-bit format built specifically for weights that fall on a bell curve, so the buckets sit where the numbers

2:29 actually are. Then a second quantization pass over the constants the first one produced, and a paged optimizer that survives the memory spikes instead of crashing on them. But 4-bit weights do

2:39 not make the rest free, and this is where the tutorial stop reading. Activation still scale with batch size and sequence length, and they are what actually kill your run, which raises the

2:49 only question that matters at a keyboard. What fits? Unsloth maintains the table people copy into every thread sorted by parameter count. 9 billion wants 6 and 1/2 GB. 14 billion wants 8

3:00 and 1/2. 27 billion wants 22. 32 billion wants 26, and 70 billion wants 41. Their own caption calls those figures the absolute minimum, not a target. The floor beneath which the run does not

3:14 start at all. Read two of those rows against your hardware, and the whole video resolves. 22 GB fits inside a 24 GB desktop card with two to spare. 22 does not fit inside 16. The desktop card

3:27 clears the row. The laptop card wearing the same badge misses it by 6 GB, and no configuration flag closes that. 14 billion though lands at 8 and 1/2. On a 16 GB laptop that leaves you 7 GB of

3:40 head room for everything the table did not count, and head room is the entire game because the note printed directly above that table says the usual reason you run out of memory is batch size, not

3:51 model size. That is not a footnote. It is the shape of the problem. Weights are a fixed cost you can predict from a spreadsheet. Activations grow with whatever you feed the thing. One

4:01 measurement makes it vivid. On a half billion parameter model with a 150,000 token vocabulary, at batch eight, the logits tensor alone reached 8.7 GB. That single tensor was 146 times larger than

4:15 the entire layer buffer pool sitting beside it. A memory budget that counted only weights would have waved that run straight through. So, the table is a floor, not a promise. Every gigabyte the

4:25 table does not spend is the batch size you get to use, and batch size is what decides whether a weekend is enough. Same table, other column, for scale. That 27 billion tune cost 22 GB in 4-bit

4:38 and 64 in 16-bit. Quantization is not a detail here. It is the difference between owning the hardware and renting it. At which point the obvious idea arrives, and most people have it. If the

4:48 model does not fit in the card, spill the rest onto the solid state drive. The drive is enormous, and it sits right there. That idea exists already, years of engineering deep. DeepSpeed calls it

4:59 Zero Infinity, and it does offload training state onto solid state drives. The config key is two lines. Before you copy those two lines, read who wrote them and for what. The paper

5:10 demonstrates the method on 512 V100 GPUs sustaining 25 petaflops. The single machine case it advertises is a DGX-2 node. That is 16 data center cards in one

5:22 chassis, a machine that shipped at around a quarter of a million dollars. DeepSpeed's own getting started tutorial trains its examples on eight cards, then 32. Nowhere in that document is there

5:32 one laptop, because that was not the design target. The physics underneath is why. Card memory is roughly an order of magnitude faster than system memory, and system memory is another order faster

5:43 than a drive. Each hop down costs you a decimal place. So, here is a decision worth making before you read another thread. Would you accept a training run an order

5:52 of magnitude slower to fit a model that is 6 GB too big? That question stopped being hypothetical this month because somebody ran the experiment. 8 H100s, a full sweep, and they published the log

6:04 while it was happening rather than a report afterwards. DeepSpeed did not fail slowly. It failed at step zero. Every zero stage crashed on an empty parameter group the moment a LoRA

6:14 adapter was involved on the PyTorch version current that week. The full fine-tuning control trained to completion in the same session, which isolates it precisely. The thing that

6:23 broke was the combination a laptop needs, and only that combination. And when it does run, the same log recorded eight cards of 03 coming in slower than one card training normally.

6:34 Sharding buys you capacity. It does not buy you speed, which brings us to the repository that says this whole framing is out of date. Most people have not opened it. It is called soup. Apache

6:45 licensed, roughly 2,000 stars, a command-line tool driven by one YAML file, written mostly by a single developer. The claim on its front page is the one you came here for. An 8

6:56 billion parameter model fine-tuned on a 4 GB laptop card, not projected, measured with the row printed 119.6 tokens per second. 3.32 GB peak on a mobile 3050. That is a card people were

7:11 told to buy for indie games. The mechanism is called layer streaming, and once you hear it, it stops sounding like magic. The frozen base sits in system RAM. One decoder layer at a time gets

7:22 copied into a small buffer on the card. That copy rides a separate stream, so it lands while the previous layer is still computing. Peak memory stops being the size of the model and becomes the size

7:32 of one layer plus the queue. The cost is stated plainly, which is rarer than it should be. Each layer is read twice per step, once going forward and once when the backward pass

7:42 recomputes it. Their docs call that physics, not a bug. Two details tell you a real engineer wrote this. The first is an admission. 2.1 of that 3.32 GB is not the model at all. It is the embedding

7:55 table and the output head. Those two stay resident and unquantized, and they are why 8 billion parameters lands right up against a 4 GB ceiling rather than comfortably inside it. The second detail

8:06 is that they went hunting for their own bottleneck and found their public claim was wrong. They had been saying the method was bound by transfer speed, so they tested it. They deleted every byte

8:15 of host to device traffic and ran the step again. It got 1.4% faster. 6.8 GB of traffic per step removed bought almost nothing. The run had been compute bound the entire time. The largest

8:29 streaming specific cost turned out to be the 4-bit unpacking at 9.8% and they wrote that correction into the docs directly above the old claim. Against DeepSpeed's own CPU offload on

8:40 the same box with the same model, they logged 2.9 times the throughput in 9.7 times less peak memory. That is the comparison that matters. So, the walls moved. Except the number you actually

8:52 want is still missing, and the project is the one telling you that. Start with the headline. That speed was measured on a version from before a correctness repair that cost roughly 5% and it has

9:03 not been rerun on a 4 GB card since. The docs say so in brackets under the number. The repair matters because what it fixed was the worst kind of defect. At 32 billion parameters, eight gradient

9:15 tensors out of 256 came back correct. The other 248 were wrong, and the loss curve looked completely ordinary while it happened, which means anyone training through that window would have shipped a

9:26 broken adapter believing it worked. It is repaired now and re-verified at 256 out of 256, then again at 72 billion and on eight H100s, which is the catch. Stated in their own words, "Every large

9:41 model result in that repository was gated on data center hardware because that is the only place a reference run fits to compare against. There is no 27 billion demonstration on a small card,

9:51 not a failed one, not a partial one. The row you want has not been measured by anybody yet, and pretending otherwise would have been the easy thing to do." They did put a proof on free hardware,

10:01 and it is worth what it is worth. An 8 billion run on a Colab T4 capped by force to 4 GB, seven steps, adapter written, exit zero. And directly underneath it, a paragraph listing what

10:14 that does not establish. No throughput quoted because a card under an artificial cap is not a benchmark. That sentence is the most trustworthy thing on the page. There is a smaller story in

10:24 there that explains more about this field than the benchmarks do. Every pre-ampere card in the wild was streaming a number format its silicon has no units for. The check for that

10:33 format takes a flag that includes software emulation, and it defaults to on. So, a T4 answers yes, falls back to emulation, and the entire free notebook tier ran slow with nothing reporting a

10:44 thing. The first attempt at fixing it asked the same question the same way, so it was a no-op on exactly the hardware it targeted. That was caught by running the notebook on a real T4, not by

10:54 reasoning about it. So, take the arithmetic home, tier by tier. 8 GB buys you a 7 to 9 billion parameter tuned comfortably today on tools that have shipped for 2 years. 16 GB, which

11:07 includes every laptop badged 4090, buys you 14 billion with headroom for a real batch size. 27 billion does not fit, and that is the whole answer to the title. 24 GB means a

11:18 desktop card, and it clears the 27 billion row by 2 GB, which is exactly why every post you read saying 24 is enough was written on a tower. Apple sits outside this argument because

11:30 unified memory lets the graphics side address system RAM directly, and a Mac Studio configures well past 96 GB in a box you can lift. You pay for that in bandwidth rather

11:40 than capacity, so the tune takes longer than it would on a card that fits. That is a trade with a known price, not a loophole, and a full fine-tune, updating every weight, is still off the table

11:50 above roughly a billion parameters on anything you own. That door did not open. Everything above is a way of walking around it. The tooling around all of this is in good

12:00 shape, and the star counts tell you which bet paid. Llama factory sits near 74,000. Unsloth is just behind at roughly 73. Deep speed holds 43,000. PEFT 21 and 1/2. Axolotl 12. And Torch

12:14 Tune, the library meta itself built for exactly this, opens its readme with a warning that it is no longer maintained. Development wound down during 2025. Read that next to the two projects at the top

12:25 of the list, because it is the clearest signal in the ecosystem about where the work went. The libraries that grew are the ones obsessed with fitting one card. The library that stopped was the one

12:35 built for a cluster by the company with the most clusters. Follow the constraint, not the logo. There is a matching lesson buried in that same eight-card log about how far to trust

12:44 documentation. Two well-known speed features were measured against their published claims for the first time. One was documented at two to four times faster and came in at 1.015.

12:55 The other claimed 20 to 60% and delivered five. Both numbers now sit corrected in that repository. A third documented feature had not launched at all because the multi-GPU entry point

13:06 was handing the launcher a compiled binary where a script path belonged. Dead since it shipped. Invisible to single-card testing. So, when a memory table and a speed claim disagree with

13:15 your terminal, the terminal is right. Every figure in this video that survived was one somebody remeasured, and several of them arrived that way from the vendor. One caution before you spend a

13:25 weekend on this, the part that tutorials have not caught up to yet. That 22 GB figure comes from a generic table sorted by parameter count. It predates the current 27 billion model and Unsloth

13:37 publishes no figure for that specific one. Which matters because it is barely the same shape. It's own config file shows 64 layers and 48 of them are linear attention rather than the

13:47 ordinary kind. Three to one in a repeating pattern. It carries an extra prediction head for speculative decoding and a vision card or stacked on top. So the adapter

13:56 targets and the four-bit kernels for it are untested territory, not a copied config. As of today, no published run has fine-tuned that model on a consumer card. If you get there first, the useful

14:07 thing you can do for the rest of us is write down the number, including the one where it failed. So here is the call. The name printed on your graphics card is marketing. The number that decides

14:16 what you can train is VRAM and laptops hand you less of it than the badge implies. The spec sheet does not lead with that. If you own 16 GB, tune 14 billion and ship it this weekend. That

14:28 is a real model doing real work on your own data and it fits with room left over for a batch size that finishes. If you want 27 billion, the path is a desktop card with 24 GB or an hour of rented

14:39 time that costs less than lunch. Waiting for a laptop to catch up is waiting for a memory bus that is not coming. The strongest case against me is soup and I would keep watching it. Layer streaming

14:50 is sound engineering with its own failures published in the open, which is rarer than the benchmarks it beats. But an unmeasured configuration is not a plan and the 27 billion row on a small

15:00 card is still empty. When somebody fills it, that changes. Until then, the arithmetic decides, not the ambition. Which leaves a question underneath all of it.

15:10 If the base is frozen and the thing you actually make is tens of megabytes, what are you training, the model or your data?