Which is The Best Qwen3.8-27B?

0:00 Okay, so this is the same question going into two versions of the same model. Same weights underneath, similar answer at the end, but one of them has used a lot more thinking tokens to get there,

0:12 and the other has used hardly any in comparison to get there. So on my machine, that's a difference between 18 seconds and 90 seconds. And if you think about it, that's sort of the problem

0:22 with reasoning models now, right? They get the answer, but they take a long time to get there. It's almost like one model flies there to get the answer as fast as possible and the other one

0:32 drives around the coast to take a scenic route. And you'll see this in ways that are often not totally bad. They'll do things like they restate the question, they check. They check again. They say

0:42 to themselves, "Wait, let's reconsider." And every one of those tokens is time that you're spent waiting for this answer. And of course, money that you're paying for the inference for this. Now,

0:54 in the past, I've talked about Open AI focusing on reducing the number of tokens between the 5.0 model and the 5.5 model. Well, they're not the only ones that have been focused on this, right?

1:07 Most of the major labs have been looking at how do they take these long chains of thought and reduce them but still keep the accuracy. And the cool thing is that there are also teams trying to do this

1:17 with open-source models. So, in the past, I covered the thinking cap model for the Quen 3.627B 627B. And no surprise, I'm going to cover their new model in this video, but also I want to

1:27 look at some of the other teams that have taken on the Quen 3.827B and fine-tuned it to think less. So on top of thinking cap, we've also got Quenpi, which is sort of like a

1:40 specialized version with less thinking for coding agents. Got Swift 1.5. They all claim fewer tokens for about the same accuracy, but it seems they all go about it in really different ways. And

1:53 it also seems that they're probably really good at different things. So, in this video, I want to look at how each one was made, then I'll run all of them against the base model and see if the

2:02 claims actually hold up. All right, so before we do the fine tunes and look at those, you need to know a bit about the sort of thinking and reasoning on Quen 3.8 that it already gives you out of the

2:14 box. and they call it reasoning effort. It has three settings, low, medium, and X high. And amazingly, out of the box, X high is actually the default. So, if you're serving with VLM, you actually

2:25 set this in the chat template. And you can see that when we set that reasoning effort, that's going to affect how long the chains of thought are that's coming back. Now, I talked about this when I

2:35 made the video about serving Quen 3.827B, and I got to say that model is still one of the best open models that you can get. and serve on reasonably small

2:45 hardware. For some uses, the 3.8 next is probably better, but that's going to move you into the realm of needing a bigger GPU or needing multiple units in there. Now, back to the 3.827B.

2:57 The whole reasoning effort there is you are basically telling the model to try less hard. So, yes, you can get less tokens out of it, but when you actually turn that reasoning down, it's often

3:09 going to cost you inaccuracy. And if you're running this locally on 1GP, you're really going to be decode bound. So the thinking or reasoning tokens there match pretty much directly to

3:20 seconds for you getting your final result out. If you can have those tokens, you're roughly having the weight that you're going to have to wait for your final answer out. Secondly, these

3:29 models have multi-token prediction heads for speculative decoding. And that actually works better on shorter traces than longer traces. And we actually see this in some of the reported results.

3:40 So, thinking cap's model card actually reports that speculative decoding goes from about 2.6 tokens per step at X high to over three tokens per step at medium and low. So, you can actually think

3:54 about this as thinking less doesn't just mean fewer tokens. Each token comes out faster as well. All right. So, the big question here is what are these fine tunes actually trying to do? And are

4:04 they going to get you the savings without any sort of accuracy hit here? All right. So here's the family tree. Everything starts from Quen 3.827B. On one branch of the tree, you've got

4:16 Thinking Cap from Bottlecap AI. Their first thinking cap model was the one that I covered a couple of months back and that was built on the Quen 3.6 model then. And you could say related to this

4:26 was the UKIS AI swift model. So when they built their first version of Swift, part of that recipe was a transfer component from the thinking cap model. So you can sort of think of Swift of

4:38 having some of the thinking cap DNA in it and being related. Now on a completely separate branch here, you've got Quen Pi which is playing a different game totally. I'll get to that as we go

4:49 through each of the models, but let's start with Thinking Cap. So Bodcap AI is a lab in Prague and one of the co-founders there is Thomas Michelov. So at the time that I recorded the original

4:59 video about the first Thinking Cap, I didn't actually know that until one of you pointed it out in the comments. And so for those of you who don't know who Thomas Mikolof is, he was actually the

5:09 first author on a very famous paper word tovec. So this is one of the key papers that really changed natural language processing. The whole idea of having vectors that could represent words and

5:21 then could be used for things like classification like a whole bunch of these things. At the time Michelof was working at Google from memory he was also working on a lot of the ideas for

5:30 seek to seek which all of these were very early precursors to the transformer idea of sequence in sequence out. Now after one of you had pointed that out. I actually reached out to him to send the

5:40 video and message him where they're going to do one for 3.8. And I found out they're working on an interesting project that hopefully I can cover in the near future. But clearly I wasn't

5:49 the only one giving feedback that the community wanted a 3.8 eight version of the thinking cap models. So I think they reallocated some of the compute that they were using for their other project

6:00 to this. And just a very quick side note here, I would say that we really should be grateful for startups like this. His startup is not some billion-dollar startup in San Francisco that's training

6:09 their own frontier models to compete with OpenAI or Anthropic, etc. Bottlecap is a startup that's raised a tiny fraction of that money compared to some of these bigger ones and most likely has

6:21 a fraction of the compute that those more famous startups actually have. So the fact that they're allocating compute to do something like this for the community, I think is very cool. All

6:30 right, so how was this actually made? Well, Bodcap didn't publish the actual training recipe, but they're clear about the objective here. The target was one thing, how many thinking tokens it takes

6:40 to reach the same answer. And they mentioned that the aim was deliberately conservative here. They didn't try to make the model smarter, to teach it anything new, to change how it talks.

6:51 They haven't tried to add any knowledge in here or even improve the instruction following or safety or any of those things. Those are meant to come through totally untouched. So stream to X high

7:01 and this time they put more focus on much harder sort of benchmarks and tasks. So things like math, reasoning, long context, agentic tasks, etc. You can kind of think of this is teaching

7:12 the model when it's done rather than teaching it to be brief. So the headline here is 37% fewer tokens on average across 12 benchmarks with the average accuracy going from 86.6 to sort of

7:26 85.8. So it's just a tiny percentage less accuracy there. So, for example, on the GPQA diamond, this goes from an average of 12,800 thinking tokens on the original version down to 7,300 on the

7:40 thinking cap one. And interesting on some of them, like the long context result is a really interesting one. On their long context retrieval benchmark, it uses 39% less thinking, but the

7:51 accuracy actually goes up a couple of points there. The chart I think that is most useful in their blog post about this is this one because it ties straight back to where we set the

8:01 reasoning effort. If you take the base model and just drop it to medium, you remove about half the thinking, but you give up about nine points of accuracy. Thinking cap at X high removes 37% for

8:15 under a point of accuracy. So suddenly that X high level gets really useful there. All right. So what's this model actually good for? It's basically just a drop-in replacement for the Quen version

8:27 at X High. So anything that needs general reasoning, knowledge questions, long documents, etc. All of those things, you can basically just drop this model in. Perhaps the only downside I

8:37 would say here would be the agentic traces. And not because they're not shorter. They're shorter, but it seems that the reasoning only drops by about 11% there. Now, interestingly, the other

8:47 thing here is that this version is made for X high. If you want a version that's made for the medium or lowinking and optimized for that, that's something that they're actually selling to

8:58 enterprises. And along with that, the other change here is the actual license. So, it's gone from being Apache 2 for the 3.6 version to this one being a small business plus personal use kind of

9:10 license, which totally makes sense if they're trying to make some money to actually cover their compute costs, etc. All right. So, next up is Swift 1.5 from Yucus AI. And this one's quite a lot

9:23 more aggressive. So, this one is going for up to 58.5% fewer thinking tokens on the Quen 3.827B. They built it in two steps. So, for

9:35 Swift 1.0, they were really looking for specific tokens in Quen's reasoning that seem to trigger overthinking. And you could think of some examples of those are where things like it says wait let

9:46 me ask what the user wants hm and then you know asks itself something let me reconsider and then they fine-tuned the model to penalize those tokens rather than just penalizing the length and then

9:58 they added that to the original version of the thinking cap model for 1.5 though they took the model and scaled up the reinforcement learning and on policy distillation with their main focus being

10:09 on long horizon agentic and coding ing tasks. Now, the cool thing here is that not only have they published the models, the training data is also on HuggingFace, which is really good to see

10:20 here. You can see it in the coding numbers here on their model card. Live Codebench goes from just under 77% for the base model with around 11,200 tokens out to actually an increase of accuracy

10:32 of 81.7% with only 8,400 tokens out. So, the other interesting thing here is that they also claim that the savings that they're getting hold at every effort

10:43 level. At low, it actually scores slightly above the base model with about 29% fewer thinking tokens. Practically too, this one has probably the most ways for you to run it. There are standard

10:54 Lama CPP GGUFS for this. There are small ones to fit on small GPUs. There's MLX. There's an NVF4 version and even an AMD build here. And they even have like a free research API for this with no API

11:08 key if you want to just try this out. Now, that headline of the 58% fewer thinking tokens probably turns out to be a bit less when you start looking at the mean thinking reduction across this, but

11:19 it's still very impressive here. Another thing that they've focused on is that they claim that the base model loops and fails early, which they're claiming that their version doesn't do. Now, the

11:29 license for this one is their own custom license, which is free up to a million dollars in revenue. After that, you need to basically contact them. All right, so that brings me to the third one, which

11:39 is kind of the odd one out here. And incidentally, this is the only one of the three that actually has an Apache 2 license. So, this is Quenpi. To understand it, you first need to know

11:49 about what PI is. Pime is an open- source coding agent that's very deliberately minimal. It has a very limited set of tools for doing read, write, edit, and bash. And last time I

12:01 checked, it even has a system prompt that's under a thousand tokens. So Quen Pi is Quen 3.827B fine-tuned specifically to work inside of the Pi harness. And I want to cover

12:12 this one because I think this is something we're going to see a lot more of where people fine-tune a particular model for a very specific harness. And it totally makes sense. If you can get

12:23 the model and the harness to work together, and the model can be primed for what kind of inputs it's going to see, what kind of inputs it should generate out, you're probably going to

12:32 end up with a model that's a lot more suited to a very particular vertical harness there. And this idea is something that I've been running tests on recently with the number of vertical

12:40 harnesses in the co-scientist area. So, I feel like this model shows a little bit of where that direction is actually going. So, how was this made? First, it's basically supervised fine-tuning on

12:51 real PI sessions, but only the successful ones. So, the model learns the whole workflows, how to look at code, how to make a change, how to run tests, how to fix what's broken, etc.

13:02 Secondly, they make sure in those tasks that they actually check for working code versus broken code separately themselves. Thirdly, they're doing reinforcement learning with GPO and a

13:13 custom reward for reasoning efficiencies there. So obviously the goal there is to basically reduce the long chains of thought. Interestingly that reward is aimed at the low and medium settings

13:23 while X high is left to focus on getting it right. The other interesting thing that's mentioned in here is that they actually pick the checkpoints based on actual agent results not just based on

13:34 training loss. So you could imagine that as the model gets better and the RL kicks in to reduce that training loss that doesn't always lead to the best agent runs. So, they're much more

13:45 focused on getting this to actually work for agent runs than purely just for some LLM training goal. Now, my guess here is a big part of why this works is that PI is a minimal harness. When it's only got

13:57 a few tools and a really tiny prompt like that, every session looks reasonably similar. And you can imagine that that generates really nice data to actually fine-tune on. So, the idea with

14:08 this one is different. Thinking Cap and Swift are trying to make your X high shorter. Pi is actually trying to make the low and medium effort settings actually good. On their numbers, Pi's

14:18 medium setting matches the base models X high on Terminal Bench with about 41% fewer output tokens. They're trying to reduce the number of tokens you're using by actually getting you to use the lower

14:30 and the medium levels by increasing their intelligence up to the high level. Now, obviously, the weakness for this one is that it's a very narrow use case. All the evidence comes from using it

14:42 inside of Pi. From what I've seen, nobody has really shown that this carries over to other harnesses. If you wanted to use open code or you wanted to use other things, etc. And on the

14:51 general sort of reasoning tasks like the GPQA, the base model still actually wins at the medium level. So, in many ways, I'd say this is sort of like a community testing effort that perhaps isn't

15:02 totally proven yet, but is certainly very promising here. So, let's jump in and actually take a look at some of the outputs that we've got here. Okay, so each of these models is running on a

15:11 Dell Pro Max tower here and they're running on an RTX Pro 6000 with standard VLM here. I haven't really optimized any of that in here. So, thank you to Dell for sponsoring the compute for this. It

15:24 really makes it possible for me to be able to benchmark these models out. Okay, so let me set up the demo here. What I've basically done is I've gotten Claude Fable and Opus to generate a

15:35 bunch of questions or prompts that that are going to create a large amount of thinking tokens or reasoning tokens etc. And then what I've done is I've gone through and actually rendered out the

15:47 inference for each of these models and saved it with exact timestamps so that we can look at all four of them playing back at once here. So you can see here we've got the original Quen 3.827B.

16:01 This is technically the fine-tuned model that they released of their base model, but this is sort of the standard version of the model. I've got the Quen Pi one, I've got the Thinking Cap one, and I've

16:14 got the Swift one in here. So you'll see that if I come up and I pick something like this, I'm actually playing back at 16 times the speed that it was recorded. And you can see sure enough here this is

16:27 now thinking cap has done. It's completed. It's got the correct answer. We've now got swift 1.5 done completed with again with the correct answer. We've got the quen pi model done with

16:40 the correct answer. And finally at some point we will get the original 3.827b looking at this one also with the correct answer there. So in this case, each of the models has given us the

16:54 exact same answer back. But we can see very clearly in this case that thinking cap was way shorter, right? With only 4,000 tokens. You can see that even the inference speeds are roughly the same.

17:07 Second is the swift model and third is the pi model and then finally is the long version. So all three of them are doing much better than the original version. Let's try it with a different

17:21 prompt. Okay, so again you can see these are all off and we can see in this case the swift model actually was fastest followed by the pi thinking followed by the original model and in this case

17:35 thinking cap actually came last. All right. Now, I will point out that this one is more of a sort of coding task with this. And remember that the pi model is made for coding tasks and tends

17:48 to specialize in that kind of thing. All right. If we look at another one like this, we can see that sure enough, in this case, the Swift model did really well followed by thinking cap. The pi

17:59 model took longer than both of those, but still was way shorter than the original one. All right. If we give it sort of a logic puzzle to go through, you see in this case the thinking cap

18:10 one, swift came second followed by the base model and then pi. If we give it another mathematics one, we can see in this one that thinking cap 1, swift came second, pi came third and then the

18:22 original model was quite a long way behind that. And you see if we translate these into time that we're getting huge savings with some of these models, the thinking cap one is just a lot faster, a

18:34 lot more concise in what it's actually producing there. Another example, again, thinking cap first followed by Swift. And in this case, the base version actually beat the PI version. So just to

18:46 finish up, I'll show you one that is kind of a bit surprising. In this one, you've got the Swift model actually doing slightly better. We've then got the base model coming second, followed

18:59 by thinking cap, and finally the PI model in here. And for me, the interesting thing here is that even though we've got the same prompt for making the SVG, each of them is making a

19:09 slightly different SVG, you know, coming out of it. And I would say this is sort of the perfect example of where you probably should try each of them out for your specific use case because just as

19:21 the SVGs change in quality in here, you will find that other answers that don't necessarily always have an exact answer also can change quite a lot from model to model in here. While in my test,

19:34 thinking cap seems to be winning most of the time, the variance can be really different on different kinds of questions. And I think that's got a lot to do with just how people are doing the

19:46 fine-tunings here. Whatever they make the model focus on and use the RL on, it's going to get good at that. And I think for each of these three fine tunes here, they've gone for slightly

19:56 different data curation and perhaps curriculum learning in the way that they've actually put these together. So if you remember back, I also talked about that the Quen Pi model was

20:05 actually trained to do better on Medium. And the thing I find is that sure enough, this model actually is a lot faster out on medium than the original base model. It doesn't mean though that

20:18 it's always going to be faster for all of them. Often the thinking cap one will still be faster for some of these. And I find that even the thinking cap one tends to be a little bit more accurate

20:30 at a lot of the different tasks and stuff when you have it actually set to medium. In this particular use case here, you can see though when we compare this that the PI model is actually being

20:41 faster than both of them when we've got the setting on medium there. So anyway, let me know in the comments if you've actually had a play with these, how you found them out, which one of the three

20:51 is your favorite. I would say that all three of these are actually very good. In general, they all tend to do better than the base model for a lot of the particular tasks. Now, the Pi 1 seems to

21:03 me to be definitely more suited to coding kind of tasks and it really shines at those. The thinking cap and the Swift one are much more sort of generalist models. So, anyway, as

21:12 always, if you like the video, click like and subscribe and I will talk to you in the next video. Bye for now.