ThinkingCap - The Local Coding Model

Okay, so in this video I want to look at a replacement to this model, the Qwen 3.6 27B, which has been the darling model for a lot of people doing coding with local AI. In this video, we're going to look at a model that is actually a fine-tune of this, which has got some really special properties, not aimed at making the model necessarily smarter, but improving it in some other ways that you'll see. But first, I want to look at how we actually got here. So, one of the things that Daniel Kahneman had in his talk at the recent AI Engineer Summit was this plot. And this is basically showing the ability of models measured on a time horizon for software tasks. And the really interesting point that he made here was that pre-O1, we were sort of on this one trajectory. And then after the O1 model, basically the first sort of reasoning model, we went on this whole other trajectory. And really a lot of the advances that we see from models recently are because of this ability to do long chain of thought reasoning. So, when we're talking about long chain of thought reasoning, a lot of this goes back to the paper of let's verify step-by-step. And the idea is that you can think of these long chains of thoughts as containing sort of sub-chunks in there, the steps-by-steps in there. And while this all sort of started out with simple sort of ideas for chain of thought, as the chain of thought gets longer and longer, it turns out that you can then sort of coax the models to almost sort of talk to themselves to get better results. And those step-by-step components are actually sort of very specific kinds of chunks. So, often it'll be something like rephrasing the problem, it'll be decomposing what would make a good answer, and what needs to be the components of each of those before it goes on to think about those and then come up with a final answer out. And we see that quite commonly when we look at the models and we look and see like that they've got these sort of step-by-step elements in their chain of thought here. And often they will be things like starting with understanding a user request. You can see number two here is deconstructing the process step-by-step. Then they'll be drafting parts of it, but there are chunks of these chains of thought. And this is what made the models get better and better over time. It started out with just longer and longer chains of thought. It also then started out doing sort of parallel chains of thought. And then over time we've seen the shift for how do we actually get better quality steps so that we've got just the right amount of steps to get the right answer. And a good example of this has been the GPT models. If we look at sort of GPT-5.1 to 5.5, you can see that one of the main things that they were trying to do there was to get the chains of thought to be higher quality but shorter so that they could come to the right answers, perhaps even better answers, but in fewer numbers of steps in the chain of thought. Now, the challenge has always been for the proprietary models, you don't see the raw chain of thought. So you don't really get a chance to actually sort of see the quality of the chains of thought changing over time to be able to give you either the same answer in a lot fewer tokens or a better quality answer in the same amount of tokens. And this is something we've seen both in sort of good ways and bad ways of a lot of the frontier models over the past year or so. That sometimes we'll see successful versions of it where a model will just get better generally overall, but then at other times you'll see things like the Gemini 3.5 flash model where the model certainly got more intelligent, but unfortunately it needed an insane amount of tokens to be able to do that. Meaning that the way that the model was working, those thinking tokens just weren't efficient even though the model was more intelligent than the previous models. And we saw this very clearly with something like the Gemini 3.5 model where you can see here that the 3.5 flash model just uses a lot more tokens than the Gemini 3.6 model. So, the main difference in those two models is just trying to keep roughly the same level of intelligence there, but consume sort of 25 30% less tokens than it previously did. So, this trade-off between getting high intelligence, but getting low token usage has been an ongoing battle for pretty much all of the labs. In the end, you've only got so many levers that you can sort of pull here. You can try to make the model more intelligent by making the model bigger, you can make it more intelligent by having longer chains of thought, or you can try and have better quality chains of thought. Each of these things has got their trade-off. And that brings us to this model that I want to look at today. So, Bottle Cap AI have taken one of the models which has been sort of the darling of the local AI coding people. And this was the Quen 3.6 27B model. So, this is obviously a dense model, not a mixture of experts model, but it's a size that a lot of people are able to fit this on their GPUs and then are able to actually get really good results out of it. And what Bottle Cap AI have focused on here is to focus on this idea of a thinking cap. So, here that the whole idea is that they're aiming to fine-tune the Qwen 3.6 27B model to keep the same level of intelligence, but to require a lot less thinking tokens to be able to get to that level of intelligence. And you can see that the result that they got out here is 46% fewer reasoning tokens on average. They've still got comparable benchmark performances to what they've got. They've got fewer sort of reasoning loops where it just gets into these sort of loops of just repeating different steps of the long chain of thought. And then obviously once you're using less tokens, you're going to get lower latency, you're going to get lower inference costs, and ideally you're going to get the same level of intelligence just with a lot shorter chains of thought in here. So they benchmark this themselves on 12 different benchmarks, and they claim that this model produced nearly identical accuracy, but using almost half the amount of thinking tokens that the previous version did. And you can see here that they've got an example of where they're comparing their fine-tune getting to an answer much quicker than just the straight Qwen 3.6 27B in there. So their goal is very simple, just reduce the amount of thinking tokens, reduce the long chain of thought, yet be able to preserve the quality of the final outputs out. And you can see that when we look at their training approach, unfortunately they don't give us a data set. That's what I would love to have here is the data set that they actually used so you could use this for other models that are out there, but you can see here that they talk about their training objective as being more efficient reasoning rather than just simply rewarding correctness. Now they don't really talk about whether they've done this with reinforcement learning, whether they've just done it purely with supervised fine-tuning, but you could imagine that it's probably a bit of both in here. So they talk about that their evaluation method required multiple runs of this obviously and then using multiple seeds in the runs to reduce noise. They've done it on a number of different benchmarks and they've been able to put all of this together. So when we come in and look at the benchmarks here, you can see that when they're comparing the different benchmarks, these were the original benchmarks and these are the thinking cap versions in here. So it's pretty easy to see on most of them, they're getting very similar scores but just using a lot less tokens. So you can see this is the tokens on the base one, this is the tokens on their post trained version in here. So I think let's jump in and actually have a play with the model and see how it actually does for a variety of these different tasks. All right, so if I come in here and look at using the model, I've got two models here that I've set up endpoints for. So one of them is just a standard Qwen 3.6 27B instruction tuned with a quant four of it. This is from the Unsloth quants and then the second one is the thinking cap model. Again, this is the 27B with the four-bit version of it. So you can see here we're using the standard Qwen model and on this algorithms question, we're getting 3,000 tokens of thinking and then we've got our answer. If we look at the exact same question with the thinking cap one on, we've got quite a bit less of the thinking tokens. In this case where it's 2,200. And that's really what this model is all about, right? Is that it should get you sort of the same level of intelligence but just with less thinking tokens in here. Now, it's really interesting to sort of look at this at the thinking tokens. So you can see here when I was talking about the sort of step-by-step stuff before, the first step here is understand the user request. Second is identify the core algorithm. Third is formulate the rhythm and then it'll go on. So, if we look at the thinking cap version, how different are those things? Well, it turns out that they're not that different at the start. Understand the user request, identify the core algorithm, formulate the algorithm. And so, you'll see that the idea here is each of these steps, the model ideally, if it's going to get smarter but not need as many sort of steps in its change of thought, that it's going to basically prune out the ones that don't help it get to the final answer in there. So, that's an example of doing a sort of algorithms one. If we look at the long essay one, you can see same sort of thing. Here we've got 1,800 1,900 tokens for the thinking. We've got analyze user request, identify key constraints and challenges, outline structure. If we come to the thinking cap version, the thinking cap version is about 500 tokens less on the thinking, but we've got similar kinds of things. Analyze user request, identify key constraints and challenges, outline structure. And you'll see this, and you can sort of go down and actually look at what's going on for each of these as you go through. Now, really I kind of feel there's not a lot of point in me going through lots of these examples too much. I have tried it with the 16-bit model. The 16-bit version of the model seems to work very nicely. I kind of feel that for a lot of people, they're going to need to just go through these things themselves and see, okay, how does this actually perform? You can also test it to see how many tool calls is it going to need for a particular task? So, here we can see some thinking. It's This is the sort of standard Quen model. It used two tool calls. And again, here we've got the two tool calls, which is kind of what we'd expect, one for search, one for fetch in here. And I've got basically the tools I can turn on or off in here. Here, with multi-tool, it's done three tool calls for this. Let's see, for the same thing, we've still got the three tool calls. But you can see, in this case, we actually used more tokens on the Thinking Cap version. And this is something that I think you've got to sort of experiment for your particular use case, is it going to be shorter or not. A lot of the times for me, it seems to be, especially in things related to coding, hard math, logic puzzles, those sorts of things. It seems to have been trained to respond to those, which is probably good because a lot of people are using this as the sort of local coding model. I definitely find though for some of the sort of long essay stuff, it can be a bit hit and miss. And the challenge is you really need to be able to run it multiple times on the same prompt to get a sense of how these things actually work. So, overall, just to finish up, I would say if you are actually using this Qwen 27B model for coding or for doing any sort of local stuff like that, you should try the Thinking Cap model out as it is just basically a drop-in replacement for the other one. Everything else about it should be just the same. And like I showed you that even the thinking is going to have a lot of the same steps in there. And in my testing, haven't seen that it's any less intelligent than the Qwen version of this model. So, it's certainly an interesting idea that they've got going here. I do wonder though, if as we see Qwen 3.7 27B come out or 3.8 27B come out, if it comes out as open weights, of course, that we actually may see a lot of these improvements baked in to the future dot releases of the models as they basically improve the post training and make the model more efficient. So, check out the model on Hugging Face. They've got a GGUF version and an FP8 version so that you can try these out very quickly and just drop them into a project where you're already using these. >> [snorts] >> Anyway, as always, let me know in the comments what you think of this. I'd like to hear back from you. What are you using for your local coding model? I've yet to see anything that's sort of in this size that really gets close to the frontier models, especially the recent frontier models. But, certainly we are getting to a point where some of these local coding models are actually getting very usable for a lot of tasks that people want to do. Anyway, if you found the video useful, please click like and subscribe and I will talk to you in the next video. Bye for now.