Qwen3.8-27B & How to Serve it Fast

0:00 Okay, so last week Quen dropped the weights for Quen 3.8 Max and it's undeniable that this model is pretty amazing at what it can do. But the key problem with this is it's 2.4 trillion

0:13 parameters. So there are very few people that are going to be able to run this locally. What everyone was actually waiting for was the model that dropped on Friday and this is the Quen 3.827B.

0:26 So in this video I'm going to look at the model and the benchmarks etc. But more importantly I want to talk about which version of this model you should be running and how you should be running

0:36 it. Now I've talked a lot on the channel about the Quen 3.627B that really in many ways that was sort of the darling model of a lot of people doing local coding running local agents

0:50 etc. And we've even looked at things like thinking cap which was a fine tune of that 3.6 six model to basically get it to be more succinct in its reasoning tokens but still keep the same amount of

1:02 intelligence. And even last week we looked at Glimmer 30B from Meta which was benchmarking against the Quen 3.627B model in a lot of their examples. And I made the comment at the time that they

1:17 were possibly just trying to rush this out knowing that the 3.827B 827B was on the way and was likely to drop any day after that. Well, sure enough, that is pretty true. If we look at the

1:28 benchmarks that actually came out for this from Quen, we can see that this model is doing substantially better than Meta's Muse Glimmer model in here. And not only that, obviously, it's doing

1:39 quite a lot better than the Quen 3.6 model. Now, I mentioned at the time that Meta was probably rushing that out knowing that the Quen 3.827B 827B was going to drop any day now. And while I'm

1:51 going to go through the benchmarks and have a look at this, most of this video I want to look at which version of the model should you be running because already there are lots of different

2:00 fine-tunes of the model. There are also lots of different quantizations of the model and perhaps more importantly, how should you be setting the model up to get the right balance so that it

2:10 actually runs locally on your system. All right. So, if we come in and look at Quen's own benchmarks, you can see that the comments that I made back then about the Muse Glimmer were pretty much true

2:20 that this model pretty much surpasses Muse Glimmer. Now, I wouldn't totally dismiss Muse Glimmer. I think it's still a very good model. It's been trained a lot more for generalization than the

2:30 actual benchmarks. So, it's still worth testing for your actual use cases. And like I said in the video, I feel that that team is just getting started. So, we're going to see a lot more versions

2:40 of that going forward in the future. But back to Quen 3.8. So if we look at this, it's a substantial bump over 3.6 in pretty much every area here. So if you were happy using 3.6, you're definitely

2:53 going to get a nice drop in replacement here. On top of that, they seem to put a lot of effort into the vision performance in here. And we can see even things like computer use and stuff like

3:02 that. This is actually beating out OPUS 4.6 6 Max, which to be fair, those previous OPUS models weren't heavily focused as much on computer use and browser use as the newer versions have

3:14 been. All right, so just as I've been recording, Artificial Analysis has released their intelligence index scores for this Quen 3.8 27B model, and I got to say, the results are pretty insane.

3:28 This is scoring 52, which puts it not far behind models like the GLM 5.2. 2 at 53, DeepSk V4 Pro, and also just way ahead of the Quen 3.627B model and pretty much any other open

3:47 model that you even have a chance of running without a serious investment in the hardware. The other thing is when we look at the Agentic Index here, this is actually beating out GLM 5.2, beating

4:00 out some of the GPT 5.6 models. Again, this is pretty insane for a model that you can run locally and get a decent token speed out of it. But the key thing I think is really interesting here is

4:13 which version of the model do you actually pick? Now, if we come into the Quen 3.8 family on HuggingFace, we can see the 2.4 trillion parameter models there. But under the 27B, they've both

4:25 got a Bflat 16 version. So, that's kind of like their full resolution version. and they've got an FP8 version. On top of that, they're already 4-bit versions out from Unsloth using the NV FP4

4:38 quantization. And you should remember that not all the GPUs can actually run this quantization as well. And on top of this, there are already lots of different fine-tunes of this model

4:50 already out there where people are both training it on different data sets to get new results out of it and then also where people are making sort of uncensored versions of the models. So

5:01 there are some examples where people have done that through fine-tuning and there are other examples of where people have done that through obliteration. So the Black Frost AI team was one of the

5:10 first groups to do this. But since then, there are also multiple obliterated versions out there. And of course, if you don't have access to an AMD or an Nvidia GPU and you're running it on a

5:21 Mac, the MLX community has already released multiple versions of Quen 3.8 for MLX. And that includes things like the BFLOT 16 ones, 8bit ones, 4-bit ones, as well as MLX versions of the

5:36 different kinds of quantization people have tried as well. So, what I'm going to do is go through a few of the different versions I've been testing over the weekend. But more importantly,

5:44 I'm going to show you that actually how you set up the reasoning is going to be one of the key things that determines whether this is a good model or a bad model for you. Okay, so I've been trying

5:55 out five different versions of this model. And I'm running them on a Dell T2 Pro Max, which has kindly sponsored the compute for this. So, all of these are actually running on an RTX Pro 6000 GPU

6:08 from Nvidia in here. And that's a pretty beefy GPU. So, I've got 96 GB of VRAM here to play with. So, in my particular case, I have no problems being able to load the model, right? Obviously, that's

6:19 a luxury. Many people are not going to be able to load the full 16bit Bflat resolution version of this model, but I'm going to try out that version. I'm going to try out Quen's own FP8 version

6:32 in here. And then I'm also going to try out unsloth quantization and I'll talk a little bit about some of the obliterator models that I tested out as well. So first up, whatever model that you pick,

6:44 you're going to realize that it's not just about the quantization that determines whether this model is good or not. So let me show you a little simple exercise. So here is my standard

6:54 sort of HTML test where I ask it to basically make a website. The joke is that it's Daario Wellness Retreat. I think a number of you can remember where that came from, but you'll see here that

7:06 this is a version of the website that was made with no thinking at all in here. So, it's done a pretty good job at just basically thinking turned off completely. And you can see when we come

7:21 in and look at the results here, there are no thinking tokens, right? It just goes straight into generation for this one. If we look at the same thing for reasoning on with low thinking, you can

7:32 see we get quite a different website, right? We're getting something that's still very nice. There's a lot of perhaps effects in it. And you may actually even like the version that had

7:42 no thinking at all in here. And this is sort of what I'm trying to get at is that the different levels of thinking consume hugely amounts different tokens. So here, if we look at the code, we can

7:53 see we got 512 thinking tokens. And this seems to be reasonably consistent when I've run it a few different times. Occasionally it gets quite a bit less, occasionally it gets more. And it will

8:04 depend on what sort of task you're actually asking it to do. But here low is getting 512. So if we come in here and look at the medium thinking, you can see that this is kind of similar

8:16 actually to the low one, right? So you would presume here that this is going to actually be using more thinking tokens, but actually it ends up using less. And I've run this sort of four or five

8:28 times. I'm not sure why on this particular task, medium thinking seems to be using less than low thinking here. Now, the question should be, well, what about high? Well, they don't really have

8:40 high, they have X high thinking. And this is where the model goes nuts. You'll see here that I don't have a finished website to show you on this one because the issue here is that while

8:51 I've limited it to 32k max tokens out, I've actually run out because I've spent 17,500 tokens on thinking. Now, I've run this multiple times trying to get one that

9:05 actually finished and I've had the thinking tokens be as high as 22,000 thinking tokens. And if we look at them, it goes into insane detail in here. So, while it actually produces a lot of the

9:19 similar sort of colors and stuff as the other ones, it just talks to itself a lot about this. And this is a general consistent thing that I've noticed is that if you're going to run this model

9:32 on X high, you've got to be prepared that one, you want to make the context window very big, but two, you're also going to be sitting around waiting for the thinking tokens to actually process.

9:45 And this seems to be true whether I'm running on the FP8, which I've done these examples on, but I've also tried this on the 16- bit model and on the Sunsloth model. And you'll see that for

9:56 each of these on XY thinking you get insanely long amounts of thinking. Now if we look at the same thing for the SVG test, right? So this is the generate a Pelican SVG test. I've taken this from

10:10 Simon Wilson's test and you can see this is on X high. So we've got 11,000 tokens of thinking to get this and it's not a bad version out. But if we look at the medium thinking here, we're down from

10:23 11,000 down to just under a thousand tokens. But we can see that here we've still got a pretty good pelican going on here. If we look at the low one, actually again, we're over the number of

10:36 thinking tokens of the medium one. And we've got a sort of similar one, although we've definitely starting to miss some of the details there. And sure enough, if we turn the reasoning off and

10:46 just have a look at the straight one out with no thinking, we've got a pretty ugly pelican. Although at one stage, this probably would have been considered really good. So, the sweet spot for me

10:56 seems to be having the reasoning on medium. And just to show you quickly how much it's overfitting on the Pelican example here, I've changed it to a red dragon. And you can see that it still

11:09 can get the bike right on no tokens out. But our red dragon is kind of childish. If we go up to low thinking, we've got something that's a bit better. If we go up to medium thinking, you could even

11:23 say that it's kind of regressed a bit on the dragon. Still pretty good on the bicycle, but not on the dragon there. And by the way, this time the lowing actually is quite low. It's 420 tokens

11:35 as opposed to medium where it's actually doing 2,000 tokens here for the thinking. Now, you can see when we go to X high, we're using 35,000 tokens of which 21,000 were for thinking. Is it

11:49 better? Yeah, it's probably better. I don't think I would say it is great. We've got a moon. We've got some sort of city backscape. We've still probably got a better bicycle and stuff like that. So

12:01 yes, the the X high will get you better results, but the amount of tokens is just kind of insane. So you can see that overall I had to extend the token window, and it's used up 35,000 tokens,

12:14 of which 21,000 here were thinking tokens just to get the dragon, which is, I guess, is okay looking, but it's certainly not great looking here. All right, so now that you know, setting the

12:27 number of thinking tokens is really important. The other thing is which version of the model do you actually go for? So first off I tested the basic Bflat 16 original version of it and that

12:41 was getting me somewhere around sort of 30 tokens per second and that was just pure generation. Now at that point I had no speculative decoding going on. Right from there, it became pretty clear that

12:53 you want to basically have the speculative decoding on and you probably want to set it to MTP equals 3. So, it's doing multi-token prediction for three tokens there. And that alone got me a

13:05 good speed bump in actually running the model. Now, for running these mostly, I've been running them on VLLM. And the next version I went to was the FP8 version of the model from Quen

13:16 themselves. And this with speculative decoding was getting me around 80 to 120 tokens per second. And it works very nicely. I didn't really notice much of a quality difference compared to the Bflat

13:27 16 version. Again, this was using their builtin MTP model for speculative decoding. And if you've got that set up right, VM can just take care of that for you. Now, around this time, I also tried

13:40 out some of the obliterated versions of the model. And while I found them interesting and yes, they sort of open things up, I find that these versions often get into repeated loops on the

13:51 thinking and just get stuck in that repeated loop. So honestly, probably at the moment, I wouldn't bother with using the obliterated ones. Next up was the unsloth model and this had really good

14:02 token rates. With the MTP, I was getting around 120 tokens per second. Here is very nice quantization of the model. I found it worked for a lot of the different tasks and I kind of thought at

14:14 this point 120 tokens per second was doing pretty well and that's when I came across SG Lang's version of serving this model. So they are not supporting all the different hardware. They're really

14:25 focused a lot more on the Blackwell GPUs and you can see here that I've got it configured for doing the RTX Pro 6000 FP4 and the DSpark. And it turns out that taking everything else off the GPU

14:37 and loading that in a Docker configuration was able to get me very fast speeds. And if you look at their X account, they're kind of showing this on their account, right? They're claiming

14:49 here that you can get up to 206 tokens per second. And to do that, they're using a very particular NVFP4 model. I found that when I used their setup with the Unsloth quantization, it

15:01 didn't work as well. But with their model weights, sure enough, I'm getting much faster tokens per second. So, you can see here, this is 36,000 tokens with an average of 173 tokens per second. And

15:14 you can see that for things that are not too long like this, I can often actually get around 200 to 220 tokens a second. Now, I would say the average is probably a bit under 200 tokens per second.

15:26 around 200 maybe a little bit below that but at that speed that's flying. I've got it running in a Docker container which is using their particular image from their cookbook. It's the NVF FP4

15:41 quant from them with the G-Spark speculative decoding in there and it's got the full 262K context window running beautifully in there. And you can see even for things

15:52 where it's using a lot of different tokens, I'm still getting average speeds of 150, 160, 170 tokens per second. So overall, just to finish up, I would say you really need to experiment with

16:03 different versions of the model to see how it's going to work for your particular use cases. And then once you've got that model, the different configurations of the amount of

16:13 reasoning tokens that you want to set, and it even turns out what inference library you want to use. So I tested VLM in here. I tested SG Lang. SG Lang was the winner for this final configuration

16:25 in here, but if you've got a GPU with a lot lower VRAM, you probably also want to test out Llama CPP. And I got to say on whatever you're doing the testing, give these pages of where people are

16:37 talking about how to set it up and stuff like that to your coding agent and have it test out different configurations to see what works best for you. For me currently, if you can serve this model,

16:51 this is the model to beat for local AI. It really is doing so well. The fact that the artificial analysis intelligence test shows it to be so high just confirms for me a lot of things I

17:02 felt using it and testing it out. And I'd finish up by saying that I think it's going to get even better. I'm really looking forward to see do we get a thinking cap version of this where it

17:12 uses less tokens for the same amount of intelligence? Do we get other refined tunes that actually improve the intelligence? And do we get things like fuse kernels where this ends up being

17:23 able to be served at even faster tokens per second? So, let me know in the comments how you're running this, what your current setup is, how many tokens per second. I may revisit this model

17:33 with some tests on the DGX Sparks or the GB10s and show you with one, two, maybe even three machines what you can do there for this kind of thing. But clearly this is taking a huge step

17:44 forward both not only in open weights but in local AI where you can run these things on sort of proumer hardware. So anyway as always if you like the video please click like and subscribe and I

17:57 will talk to you in the next video. Bye for now.