NVIDIA's Secret 2 Bit Model

0:00 Okay, so for a long time, the rule of thumb for running models locally has been pretty straightforward. Take the biggest model parameter-wise that you can fit at 4bit and don't go much lower

0:10 than that because below 4 bits is where you really start to see deterioration. Now, if you remember back recently, I did a video about the Microsoft local AI launch and at that launch, they

0:20 mentioned a secret 2-bit model that they're going to run on the RTX Spark laptops. Well, Nvidia has now shipped that model and it ignores all the rules that I talked about before. It's a 74

0:33 billion parameter model with all the experts stored at 2 bits. So, the whole thing fits in 24 gigs. And they're claiming that it beats models with a similar memory footprint that are

0:43 running at 4bit. Now, at the time when I made that video, I was really suspicious. I said it in the video and I promised that I would follow up on that. So in this video I'm going to look at

0:52 exactly what they've done here because it turns out that the quantization is just the very last step here. A lot of the interesting things that they did actually happened before this. Okay. So

1:03 the model that I'm talking about is Neotron 3.5 Bolt 74B A9B. So that's 74 billion parameters in total with around about 9 billion active parameters per token. It's a mixture of experts models.

1:18 It's Nvidia's hybrid member transformer design. And the cool thing is that it takes images and video and it's got up to a 256K context. So, like I mentioned before,

1:30 Microsoft announced this at their event that was all about the RTX Spark laptops. And the whole pitch is that this is basically an agent model you can run locally on one 24 gig device

1:43 alongside your other apps. Now, in a little bit, I'm going to run some demos and we'll have a look at the outputs ourselves. But in many ways, I think the recipe here is actually more useful than

1:52 the model itself. If you understand how Nvidia got from the Neotron 3.5 super model, which is 120 billion parameters down to a 20 gig file, you've basically got a map of every technique people are

2:07 using to make smaller models these days. You've got quantization, pruning, distillation, low bit training. All of them were used in making this model. And I kind of feel this could be a gamecher

2:20 for a lot of models that are going to be in the sort of one bit, two bit range that we're starting to see on hugging phase. And it's also interesting to point out that this is different than

2:30 just a lot of the quantization techniques that people are actually doing out there. But before I get into that, one of the things I want to ask from you is that my editors pointed out

2:38 that the majority of people watching our videos are not subscribed. And nowadays with YouTube's algorithm, it really is basically a game of luck if you're going to see the next videos we make if you're

2:48 not already subscribed to the channel. So, as much as I hate to do the YouTube thing, if you find the videos useful, please click like and subscribe so the videos actually get in front of a lot

2:57 more people and that hopefully you see them as they come out. So, let's start with the basics. When you download a GDUF file or you pick a Q4 or Q8, what you're usually picking in that case is a

3:09 post-training quantization. The model was actually trained in 16 bit. Then someone has come along afterwards and rounded every weight down to a fewer bits. And there's a bunch of different

3:19 ways people do that. Probably the most common ones now would be the NVFP4. You've got the unsloth ones. You've got a few other ones that perhaps becoming popular as well. Usually this is done

3:31 with some form of calibration data. So that is basically where you've got a data set and you're rounding the errors in a way that it's going to have as little effect as possible on the outputs

3:43 there. So at 8 bit that's basically free. At 4bit for most models it often is pretty close to free. That's why 4bit basically became the default. But here's the thing. Below that four bit

3:57 post-training quantization starts to break down. And generally what they've found is that the sort of phase transition between two bits and three bits is where you see a huge amount of

4:08 deterioration. Often at 3s and above you can get decent results and the quantized model stays reasonably close to the original. At two bits and below though representations change dramatically. The

4:20 model basically has to relearn things. So if you take a model and just round it to two bits after training, you're asking it to be a very different network without really giving it a chance to

4:32 adapt. That's why two bit and one bit quant that you see from the community are often just kind of rough. Now one of the papers that talked about this was a paper from Meta called Parto Q. And one

4:44 of the key things that the Parto Q found is actually the bit that Nvidia is kind of focusing on here. That if you do train properly at low bits, two bit and threebit models can actually beat post-

4:56 training quantized 4-bit models at the same memory budget. So basically bigger model with fewer bits per weight wins. Now, I'll admit that even now I'm still very suspicious of this. And I think

5:10 it's mostly because it's not something that's actually easy for third parties to do. This is something that you really want to see coming from the people who actually made the model. And I think the

5:21 Bolt model really is a test of this idea because it's a big capable model that you can actually download it and check it on a machine with a consumer grade GPU. Now, the temptation is basically to

5:34 just dismiss this as something that Nvidia made to fit into those RTX Spark laptops, the ones that have only got 24 gigs. And while I do think that is true to an extent, there are some other

5:46 things going on here that really do matter. So, the first one is speed. When you're running locally, you're usually running one request at a time, a batch size of one. You're not limited by

5:55 compute, you're limited by how fast you can stream the weights through memory. Nvidia actually makes this point in their model card. At low batch sizes, kernel latency improves roughly in line

6:06 with how much you've compressed the weights. Fewer bits per weight, fewer bytes to move, more tokens per second. The second reason here is headroom. If you're building an agent, the model

6:18 isn't the only thing running. You've got your KV cache going, maybe an embedding model, maybe a speech model, a browser, an IDE. You've got other things running on the machine. And you really want a

6:30 model that leaves some memory because that actually allows you to have the apps around the model be able to make use of that model. The third reason, one that I think people are often missing

6:40 here is that the quality gap between a big model squeezed down versus small model trained from scratch is actually a design choice. You could ship a 35B model at 4bit. You could ship a 70B

6:54 model at 2bit in roughly the same space. Nvidia is clearly betting on the second one and it kind of does make sense when you look at the results that we'll see here. Now, there definitely are some

7:05 issues here. The two-bit number is for the experts. The member layers are actually NVFP4, right? So, it's not like the whole model is in two bit, right? The member layers are 4bit. The

7:18 embeddings and the output layer are actually 8 bit. So, really this is a mixed precision model, not just a twobit everywhere model. And that's definitely not a slight on Nvidia. This is, you

7:29 know, something they're being totally upfront about. And actually, it makes sense to actually have the layers be at a quantization level that makes the most sense for that particular layer. The

7:41 other thing too here is that speed-wise, this is not the fastest thing in its class. Nvidia's own speed table has it around 84 tokens a second on the RTX Spark. Now, that's actually behind the

7:53 Quen 3.6 six and the Gemma 4 models, right? So, it's important to remember here that this is a smarter per gig model, not a faster per gig model. All right, so how do they actually make

8:04 this? There are basically three stages that matter here. So, stage one is pruning. So, they start from the Neutron 3.5 SuperVL, which is 120 billion parameter model with 12 billion active.

8:17 Then they run something called star elastic. You can kind of think of this as it's an architecture search with a memory budget. So a smaller router learns layer by layer how many experts

8:29 to keep and how wide each mixture of expert block should be. They also cut the number of experts active per token from 22 down to 9. And they clearly have a budget here. They didn't just say make

8:42 it 74 billion parameters. They said this needs to fit under 24 gigs at a batch size of 1 with a context window of 128K. And that budget already accounts for the fact that the experts will end up at 2

8:55 bits. The member state stays at 16 bits. So the pruning knows about the quantization that's coming later on. They're then able to basically improve it by distilling from the parent. And to

9:08 do that, they're actually doing that with about 120 billion tokens in total. Most of that distillation is at roughly sort of an 8K context purely because it's faster and then the last chunks are

9:19 at 128. So the long context stuff kicks back in and starts working. Stage two is just normal post training here. They reattach the vision encoder which they had separated before. They do some more

9:31 SFT merge checkpoints and run several rounds of RL for agent tasks. At this point it's still in Bflat 16 here. And then stage three at the end is the quantization, but not just post-training

9:45 quantization. So this is where quantization aware training comes in. And it's worth pausing on a bit because this is starting to become pretty mainstream. The idea here is pretty

9:55 simple. During training, you basically want to simulate low bit rounding on every forward pass, but you keep a full precision copy of the weights for the updates. So the model itself kind of

10:09 learns to live on this quantization grid instead of just getting rounded at the end there. So Google themselves have been doing this for quite a while. We've seen this in the Gemma models. Gemma 3

10:21 had QAT versions. QAT being quantiz training. They hamped them at Q4 and then back in June they actually released the QAT checkpoints for the whole Gemma 4 family from the E2B up to the 31B plus

10:35 for a drafter model. The two mobile versions there are actually the interesting ones for this video. So for the E2B and the E4B, they push the token generation layers down to 2bit while

10:48 keeping the core reasoning layers at a higher precision. And that actually got the E2B down to below a gigabyte. So in many ways that was actually the same sort of mixed precision idea as Bolt but

11:00 being done at a scale that you can actually run it on your phone rather than run it on a full-size GPU etc. Now I should point out that it's not just Google that have done this. Open AAI did

11:10 this for the GPT OSS models. It's one of the reasons why their 120B model can actually fit in a single 80 gig GPU. Moonshot have done some things like this for the Kimmy model and it's really

11:22 becoming a pattern now that the big labs are actually planning for quantization during training instead of just leaving it to the community afterwards. They really understand that if they want to

11:33 have a model that people like and actually use on consumer grade GPUs etc those quants have to be really good. So we're seeing more and more of the top labs actually release their own

11:45 quantized versions which benefit from the training as well. Now Nvidia here is using a little twist called QAD. So this is quantize aware distillation. The quantized model is the student and

11:57 they're actually distilling down from the version of the model that is not quantized. And their argument for this makes a lot of sense, right? This allows you to sort of benefit from all the SFT

12:06 any sort of RL any sort of merging. then rerun the whole pipeline with quantization switched dot and then be able to do distillation at the end without even having to use all of the

12:18 same training data etc. All right. So what does all this quantization actually you know cost us? Well Nvidia's numbers are going from the sort of Bflat 16 version to the 2bit file. They're using

12:30 a thing called pinchbench which from memory is one of the opencore benchmarks that they've used before and they're seeing basically on that one it goes from sort of 90 to 88 on the browser on

12:42 things like GPQA diamond it goes from about 89 to 86. Now those are quite substantial hits to take except when you remind yourself that the full version is 148 gig model and the new one is only a

12:55 22 gig model. When you look at it that way, the trade-off starts to really sound like it makes a lot of sense. Now, if you are interested in sort of looking in this in more details, there's a

13:04 couple other techniques that are worth knowing about. So, the first is expert pruning. So, this is where you drop whole experts from a mixture of experts model instead of shrinking everything.

13:15 So, Cerebus has a really nice method called reap that does this. And the key finding there is that pruning experts works actually much better than trying to merge them because merging actually

13:26 breaks the router's ability to pick them. But when you're just cutting them out, which experts decide actually depends entirely on your calibration data. So for example, if you're

13:36 calibrating a model on code, you'll get a model that's still good at code and probably going to be worse at other things, right? because it's keeping the experts that make sense for coding but

13:47 perhaps not for creative writing or something like that. The second one that definitely shows some promise is things like Microsoft's Bitnet where the model is trained from the very first step with

14:00 weights that are just minus1 0 or one. And really that's the true 1.58 bit model. And while that one does seem to work and be quite interesting, the big challenge there is that currently GPUs

14:14 don't have proper hardware support for it. So it does seem to me that the realistic path to this kind of thing is exactly what Nvidia has done here. Train big in 16 bit compress distill down

14:26 prune and then distill down to low bits etc. All right so let's jump in have a look at how this model actually performs when we run it. Okay so coming in and actually testing this model out. So, I'm

14:38 actually testing this out on the ZGX Fury. And you would think with that setup that this thing would actually fly, but it turns out not to be the case, right? Remember I talked about

14:48 this as being a bigger model, not necessarily a faster model. You can see here with the Fury, I'm getting around 220 tokens a second. Now, that's a lot faster than what you're going to see on

15:00 the RTX Spark laptops, etc. But you can certainly see a difference in that if I come across and put the same thing into the Neatron 3.5 super model here, I'm going to be getting well over probably

15:16 around about 600 tokens a second depending on whether it's code, depending on whether it's text, something between 5 to 600 tokens a second in here. You can see I'm getting

15:26 extremely fast prefill on that as well. Remember this bolt model is taken from this 3.5 super here. If we look at the outputs, they are definitely different. Yeah, probably I would say that the 3.5

15:40 super is better, but certainly the bolt is not unusable or something like that. It just definitely has a different feel to it. And my guess is that it's certainly a bit trimmer on and quite

15:51 different sometimes on some of the thinking tokens that you're going to see here. But it does seem to do a pretty good job at most of the tasks in here. Now, if we look at the output though for

16:02 certain things like the SVG test, you can see one here where okay, not don't know if the pelican's actually riding on top of the bicycle or riding on the wheel or, you know, we've definitely got

16:14 a pelican there. We've got a bike, but it's not looking so great. So, someone asked last time about why don't I show some of the ones from different runs and stuff like that. So, here's one from a

16:23 different run. This one is definitely a fat Pelican. Looks like it's kind of on the bike and stuff like that. If we come and look at this in the Neotron 3.5 Super, though, you can see that we're

16:35 not necessarily getting an amazing one out. We do get it a lot quicker on the ZGX Fury, right? You can see going through this that the average token speed, especially when it's getting to

16:48 the actual SVG stuff. We can see we're hitting 650 tokens a second here, which is pretty amazing. But if we look at the output, we're probably getting a little bit better, but it's not hugely better.

17:00 So really, what that tells me here is that more just this model hasn't been RL for any of these kinds of tasks. It's not a slight on the model itself. If we come in and test the model on things

17:11 like the rag tests, you're going to see that sure enough, it's going through. We're getting faster token speeds here a bit. We've got sort of limited prefill speed on this even though we've got the

17:24 huge compute. Just the way the model is configured doesn't seem to be supporting that in here. So for this particular machine, the ZGX Fury being a DGX station GB300, it probably doesn't make

17:37 sense to use this model because at the same time we can switch across to the full version of the model, get much faster token speeds out, and you can see like just how quickly that answered. And

17:50 you look at the prefill here, we're getting over 13,000 tokens per second prefill here. Now really where we want to know this model is going to perform well is if it's going to be for agentic

18:02 stuff is it going to get all of the sort of agentic tool cases working and so far it does seem to be pretty good at these basic ones at locking down single tool calls at doing different things like

18:17 parallel tool calls and stuff like that. You can see that this one's gone through and it's scored 10 out of 10 there. It's gotten all of these and it hasn't done it in a way where it's required huge

18:28 amount of calls in there. Now, with the pro calls, I find that it sometimes can be confused a little bit over certain ones like the prompt injection resistance in here. We'll see sometimes

18:43 it gets all eight out of eight in here. Sometimes it only gets seven out of eight in here. And you can see sure enough here it's missed one. go down and look at the one that actually failed.

18:55 Sure enough, it's this prompt injection resistance. So, that's something that maybe the smaller bit rate does suffer from. For all the others though, it did a good job. It gets two failures. They

19:08 were configured to fail automatically. Sure enough, it can pick it up and then keep going with that. Running the same test with the 120 Super, right? This one turns out it also has the same problems.

19:21 So, it does seem that the model's distilled. And you can see if I just try that again, I've had multiple times now where it's failing on the prompt injection resistance that's going on in

19:33 there. And you can see, sure enough, it's this particular test that is the one that's failing. So, that's both failing for the big one and for the small one in there. Okay. Okay, if we

19:43 run it through some concurrency tests just with four users here, you can see that the bolt is really only built for a few instances at a time. Here we're getting aggregate output of 130 tokens a

19:55 second. Even when we go across four users that's coming down below on some of them below 40 tokens a second, which is the sort of minimum that I like to have in here. You can see if we run the

20:08 same tests with four and even 16 users here, we're getting 2 and a half, sometimes even over 3,000 tokens per second out here. And all 16 users are getting well over 150 plus tokens out.

20:24 On the four concurrent users, we can see we're getting over sort of 300 tokens out. And sure enough, if I come and put it up to 32 concurrency, you can see I'm getting well over three and a half

20:36 thousand tokens, often even over 4,000 for some of the different tests depending on which test I'm running. And each user here is getting well over a 100 tokens per second. I say the

20:49 takeaway here is that while I understand most people are not going to have a GPU that is the GB300 like the ZGX Fury, if you've got an RTX Pro 6000, if you've got perhaps even like a 3090, 49,5090,

21:04 you're perhaps going to be better to go with the bigger model than the Bolt. The Bolt is certainly going to be good if you've got one of the Sparks, the ZGX Nano, that kind of thing. or obviously

21:15 if you've got an RTX Spark, then it's going to make a lot of sense for this. So, it really does seem that the Bolt really is built for the right purpose, right? It does very well on the hardware

21:29 that Nvidia designed it to actually ship on. And its performance isn't hugely off the Neotron 3.5 Super VL, which is 120B model with 12B active. So going forward, I'd love to see some of the 200 or 300B

21:46 models actually be put out like this. You could imagine a GLM 5.3 flash or 5.5 flash that has one of these chopped down versions so that it can basically run and is optimized not for concurrency but

22:02 for running on this hardware like the RTX Spark etc. Okay, so in conclusion, I would sort of say that the takeaway for me is that it's not just two bit on its own. You know, if you see that in a

22:15 model description or something, that doesn't really tell you that much. What really matters here is how the model got to two bit. Was it rounded after the fact or was it pruned and trained to

22:26 that precision as it was actually going through the training? Bolt is a really good example of the second version here. I think we're going to see a lot more models built this way and it really has

22:36 the potential to unlock a whole generation of newer, bigger models that people can run on their current hardware. Okay, so I've put the links to the model, the technical reports, and

22:47 the papers in the description. Let me know in the comments what you're running locally and whether you've actually tried going down from 4bit to two-bit models, etc. And in another video I've

22:57 just been working on, I'm going to show you that at times even dynamic one bit models seem to actually be able to produce some interesting outputs. But I'll leave that one for an upcoming

23:07 video. Anyway, as always, if you found the video useful, please click like and subscribe, and I will talk to you in the next video. Bye for now.