Unsloth vs PyTorch: The ONLY Video You Need to Understand the Difference

0:00 two times faster, 70% less memory, and zero loss in accuracy, which is the part that should not be possible. Normally, you buy speed with precision. You approximate something, you round

0:11 something off, and the model comes out a little worse. This one does not do that. Same gradients, same loss curve, same final weights. The library is called Unsloth. Free, open- source, Apache

0:23 licensed, and it runs on top of PyTorch, which makes the fight in the title a strange one. Unsloth is not competing with PyTorch. It is built out of it. So the real question is narrower and

0:33 better. Where does the two times actually come from? Speed is never free. Something in that training loop is being paid for. And you are about to see exactly what. The answer starts with one

0:43 sentence written by Daniel Han on the day he and Michael Han launch the thing. His words, PyTorch's Autograd is reasonably efficient for most tasks, but if you want extreme performance, you

0:54 have to derive the matrix differentials yourself. He is not saying PyTorch is slow. He is saying PyTorch is general and general has a price. Autograd will differentiate anything you write which

1:05 means it can optimize for nothing you wrote. Autograd is one of the great pieces of software engineering. You write the forward pass and it hands you the backward pass for free. It has to

1:14 work for any graph anybody can build. So it is not allowed to assume anything about yours. Han's bet was that for one specific shape, a frozen weight matrix with a small trainable adapter bolted

1:24 onto it, you can beat that by hand. In that shape, the attention block needs six matrix derivatives, and he sat down and worked out all six. Then comes the part that sounds too small to matter.

1:35 Matrix multiplication is associative. A * B * C gives the same answer whichever pair you multiply first. The cost of getting there is not the same at all. The adapter matrices are skinny. ranks

1:47 of eight, 16, maybe 128. The model's weight matrices are 4,096 wide or wider. Bracket the multiplication one way and you build an enormous intermediate matrix. Use it once and throw it away.

2:00 Bracket it the other way and you do not build it at all. Same answer, wildly different bill. Autograd cannot make that call for you. It does not know your rank is eight. It sees two matrices and

2:11 multiplies them in the order you type them. So he rewrote the hot paths as kernels in Triton, OpenAI's language for writing GPU code in Python. The rotary embeddings, the feed forward blocks, the

2:22 layer norms, the language model head, handwritten, fused in place, and it worked immediately. On a free collab card in December, the Alpaca data set took the hugging face implementation 23

2:34 hours and 15 minutes. Their build finished the same job in 2 hours and 34 minutes, which raises the obvious objection. If the maths is identical, why does PyTorch not just ship it?

2:45 Because a framework that ships that trade has to ship it for everyone, and it stops being true the moment your rank is large or your model is not a transformer or your adapter is somewhere

2:55 else. Generality is PyTorch's product. Unsloth is allowed to be narrow, and narrow is where the speed lives. The speed is only half the story anyway. The memory half is stranger because it

3:06 starts with a tensor most people have not thought about once. At the end of every forward pass, the model turns its hidden state into a score for every single token in the vocabulary. Long

3:16 sequence, big vocabulary, and that one array of numbers becomes the largest object on the card. In November, a team at Apple published a paper about exactly that. Eric Weidman's and four co-authors

3:28 called it cut cross entropy, and the numbers they published are worth stopping on. The memory used by the loss computation on the model they measured fell from 24 GB to 1 megab. The

3:39 classifier head as a whole went from 28 GB down to one. The trick is to skip the array entirely. You compute the loss in slices on the fly and throw each slice away as you go. The gradient that comes

3:51 out the other end is identical. Unsloth ships that plus its own chunked version that picks the slice size at runtime from whatever memory you have left, plus a checkpointer from April 2024 that

4:02 pushes activations out to system RAM and now costs about a tenth of 1% in overhead. Add it all up and the numbers stop looking like a speed story. On an 8 gigabyte card, Unslaught's published

4:13 benchmarks fine-tune an 8 billion parameter llama at just under 3,000 tokens of context. The hugging face path with flash attention runs out of memory before it starts. Go up to 80 GB and it

4:25 is 342,000 tokens against 28,000. 12 times the context, same card, same model, same four-bit weights and the ceiling moved again. Working with Stas Beckman on a

4:36 technique called tiled feed forward layers, they pushed a 20 billion parameter model past 500,000 tokens of context on one 80 GB card. The previous number was 80,000. So the two times is

4:48 real. But two times against what? Because that missing word is the most misleading thing in this entire field. And Unsloth is one of the very few projects that tells you the answer. Take

4:58 their mixture of experts kernels shipped in February. The headline is 12 times faster. read the blog and that 12 is measured against transformers version 4 against version 5 which already uses

5:09 PyTorch's own grouped matrix multiply the same kernels are about twice as fast both numbers are correct they answer different questions and only one is the question you are asking their published

5:19 table makes it planer fine-tuning the same model on the same blackwell card at a thousand tokens of context unsloth is 1.4 times faster at 22.4 at 44.8 8 at 8,000 7.3 and at 16,000 the

5:36 comparison stops because the baseline runs out of memory and posts no number at all. That is a curve, not a speed up. The advantage is modest when the work is small and it keeps growing until the

5:47 alternative simply stops working. Which means the useful form of the sentence has an object attached to it. They do the same thing with their packing work. The headline says three times faster.

5:57 The blog then deres the formula out loud and says the gain depends on how many short rows your data set has and that it collapses toward two if your rows are all long. Two times faster than what at

6:08 what context length on which card measured after the compiler finished warming up. Four questions. Unsloth answers all four on its own pages. The ecosystem quoting Unslo answers none of

6:19 them. It takes the biggest cell in the table and prints it on a badge. There is a second thing that badge hides and this one is a real limit rather than a framing problem. It is multiple GPUs.

6:31 Read Unsloth's own multiGPU page today and it says the process is complex, requires manual setup and that official multiGPU support is being announced soon. Read the read me on the same day

6:41 and it says multiGPU is available now. Read the pricing page on that same day and the free columns still list multiGPU under coming soon while the paid tiers sit behind a contact form with no price

6:53 printed anywhere on the page. Three of their own pages, one afternoon, three different answers. So if your training plan needs tensor, context, or expert parallelism as a thing you configure

7:04 rather than a thing you route around, Axelottle and Llamaactory document that matrix and Unsloth does not and a smaller trap that will confuse you on day one. Unslaught's own benchmark page

7:14 warns that torch compile takes about five minutes to warm up and tells you to measure throughput only after it finishes. Measure a short run and you will conclude the whole thing is

7:24 oversold. Measure a full apic and you get the number they published. It is not only unsloth making the claim either. Nvidia's own developer blog last December put the figure at two and a

7:34 half times the hugging face transformers library on Nvidia hardware and called Unsloth one of the world's most widely used open-source fine-tuning frameworks which brings us back to the verses in

7:45 the title in April this year the PyTorch ecosystem working group announced four new projects joining its landscape unsloth was one of them pietorch's own sentence about it reads utilizes PyTorch

7:57 to enable training of models and Torch compile for optimizations. The library built by deriving its way around Autograd is now formally part of the thing it derived its way around and it

8:08 runs both directions. Unsloth's newest mixture of experts path is built on PyTorch's own grouped matrix multiply and Han's release post thanked PyTorch, Torcho, and Hugging Face by name. The

8:21 project itself has grown into that. 74,000 stars, an Apache license, a Y Combinator batch, and a team of eight people. the last time the page was updated. There is a cleaner example of

8:32 what that relationship actually buys the rest of us. In October 2024, Benjamin Marie reported that gradient accumulation was producing higher training losses than plain full batch

8:42 training, which should be mathematically impossible. Unsloth found it. The cross entropy denominator was being averaged inside each mini batch instead of derived across the whole batch, which

8:52 made the final loss come out G* too large, where G is the number of accumulation steps. That bug was sitting in nearly every trainer in open source, including multiGPU runs that were not

9:03 using accumulation at all. Two people with a Triton habit found it, wrote the proof out in public, and HuggingFace patched it upstream. They are not only shipping kernels now either. On the 11th

9:14 of August, they released a desktop application that runs and trains models locally on Mac, Windows, and Linux. And the announcement has been seen about 3/4 of a million times. the release after it

9:24 merged more than 200 poll requests nine days later. Whatever else is true about the marketing, the pace is not a marketing artifact. So, here's the call. If you are fine-tuning on one GPU, and

9:36 almost every tutorial you will ever read assumes you are, Unsloth wins, and it is not close. Same weights, roughly half the wall clock, a fraction of the memory, and the receipts are published

9:46 with their baselines attached. The 5% it loses is real, though. multi-node shops and anyone who needs parallelism as configuration instead of as a workaround. Those people should be

9:57 running axelottle or llama and unsloaf's own documentation will tell you the same thing if you read past the readme. The thing worth being annoyed at here is not a company. It is the multiplier with

10:08 nothing attached to it. Two times faster is a sentence with a missing object and a field that repeats it without the object teaches people to shop for numbers instead of measurements. Which

10:18 leaves the one question I would carry out of this. Unsloth exists because the two people who built it looked at a default that had not been rederived in years and did the calculus by hand

10:27 anyway. So, what is still sitting in your stack that nobody has checked?