MiniCPM5-2B: The Best Sub-Agent Model Yet?

0:00 Okay, so a few months ago, Mini CPM released their 1B model and I covered this on the channel talking about the whole concept of sort of having a small cognitive model that could do a lot of

0:13 different things. Well, after a few months, they are back with a new model and they've jumped from 1B to 2B here. In fact, I think it's actually a 2.5B model, but we'll give them that. So this

0:26 is the second model in this mini series from open BMBB and in many ways they've taken the same sort of recipe but they've scaled it from 1B to 2B. So I guess the big question here is when you

0:39 scale the data and post-raining recipe by 2x does that suddenly make this model a lot more viable for the hard function calling and along with that 2x scaling does the agentic stuff scale as well. So

0:54 the interesting thing here is that the 1B model was pitched at sort of coding agents and tool use. The 2B model is claiming to beat 4B models at the same kind of thing. If we come in here and

1:07 look at their benchmarks, we can see that they're edging out Quen 3.54B on things like Sweetbench verified, etc. Now, when we look at the training recipe, it seems to be mostly the same.

1:19 And it seems that these scaling improvements really did sort of have an effect here. In fact, in some places more than I expected. So let's take a look at this stuff. So where the

1:29 supervised finetuning for the 1B was 200 billion deep thinking tokens and 200 billion hybrid tokens, the 2B has gone for 400 billion tokens of deep thinking only. But we can see where some of these

1:43 changes have come in both the RL and the on policy distillation. Here we can see that they've changed up how they're doing the RL stage. So they've got this new thing called RL2. It looks like

1:55 they're going to release a whole paper about this. This is just sort of like an early preprint here. And the key thing here is that they've changed how they're doing this here to basically include a

2:04 critic in the RL as well. All right. So if we take a look at their training recipe, we can see that they have this sort of pre-training and mid-training here. And we can see that that's

2:15 basically setting up the model for the post-training processes. So if we look at the post- training, the post- training is breaking down to three different parts, right? You've got your

2:24 standard SFT. This is sort of providing the format for outputs, providing sort of some of the key thinking behavior, etc. They then do RL per domain. So they've got multiple experts that

2:37 they're doing the RL and the on policy distillation stuff from there. And we can see that those are sort of breaking down into sort of reasoning task RL general task RL and agentic stuff there.

2:50 And the key thing with that is that each of those experts only needs to be good at one thing. This is the new form of RL here, this criticbased just RL2. And they're claiming that this actually has

3:03 better stability than the sort of traditional GRPO stuff. But it is interesting that this is showing here that this makes the model a lot better on a lot of their different benchmarks.

3:14 Now finally they have their on policy distillation layer here. The model is sampling out its own trajectories that were used to train the teacher models. And then from that they can then

3:23 basically look at how different it is to the teacher models and then update in a loop etc. So when we go back into the benchmarks, they're doing a lot better than obviously a lot of the 2B models,

3:34 but they're doing very well compared to a lot of the 4B models in here. And so the trade-off here is that it's not a huge amount to go from 1 to 2B if we're getting a model that's substantially

3:46 better and now also has things like general knowledge in it. If we come down and look at the things like the GPQA diamond, etc., We've got a model now that, okay, is a bit behind the 4B

3:59 model, but this is way ahead of the other sort of 2B models that are out there. And of course, it's miles behind the Frontier models that are well up into the high 90s, etc. at this point.

4:10 But not that long ago, this kind of score was considered a good score for models that were 10x to 100x bigger than this particular model. Now, the cool thing here is that they've released a

4:23 bunch of different versions of this model. You've got sort of the final model that's got everything in it. You've got just the SFT models before RL. You've got mid-training models.

4:33 You've got the base model in there. And then you've got a bunch of quantizations and a BSpark model in here, which is good for the speculative decoding, etc. Of course, one of the good things about

4:43 this size of model is that it's very easy to fine-tune for your specific use case. then you face the challenge of what if you wanted to have that fine-tune on whenever you wanted to

4:53 access that and that actually brings me to today's sponsor. So, one of the things that you often want to do is get a model like this set up as an OpenAI compatible API that scales and that's

5:04 where RunPod serverless comes in. So, thanks to them for sponsoring this video. Let me just show you how quickly you can go from having a model on hugging face to being able to deploy it

5:14 to VLM on a serverless endpoint with a GPU that you choose. So you can see here that I've picked the model from Hugging Face. I now have to pick a GPU with enough memory to run it on. Then I can

5:25 just set my scaling limits and create an endpoint. Once it's ready, I can now send a prompt and the model returns a response. It's literally that easy to have an API that my applications or my

5:36 agents can call. Behind the scenes, RunPod routes the request to a worker running my model on a GPU. And if demand grows, RunPod can automatically scale up those workers within the limits that

5:47 I've set. And if I don't use it for a while, it can scale it back down to zero. The cool thing here is whenever I ping a new request, the model will spin up and then be able to serve it. Compute

5:58 is built by the second while workers run. So you don't have to keep a GPU rented around the clock. They also have flash boot which helps reduce cold starts. And if responsiveness is the key

6:09 priority, I can actually pay to keep the workers running and ready all the time. So all of this just gives you the control to serve the model where the GPU infrastructure is all managed for you

6:20 and you only have to pay for the time that you're actually using it. If you're building any AI app or deploying your own model, check out RunPod serverless at the link below and get started today.

6:30 All right. So, did everything actually scale here? And this clearly is beating 4B models. Actually, the answer is no. If we look at things like Swebench Pro, we can see that the Quen model is

6:41 substantially better than this. Basically, double it. And if we look at Terminal Bench, we can see that the model is way behind the Quen model, but it still is a lot further advanced than

6:52 a lot of the other 4B models out there. And don't forget that the Quen models probably have a much bigger budget for pre-training, for a lot more tokens, for a lot more things like that.

7:02 Unfortunately, we haven't seen the really small Quen models being updated to say the 3.8 models. Hopefully, that's going to come with the not too far off Quen 4 models. So, let's jump in and

7:13 have a look at the speed of this of running it locally. Have a look at just how it actually does on some of the function calling and some of the agentic tasks in my sort of testing suite. All

7:23 right. So, I've got three versions of the model running here. I've got the Bflat 16 version. I've got a DSpark, so basically a speculative decoding version, and I've got the 4-bit GGUF

7:35 version with Llama CPP. I'm running the two bigger versions on SG Lang here. And you can see that if I kick it off, I'm actually getting pretty good token speed, which is to be expected. I'm

7:48 running all of these on an RTX Pro 6000. Now, if we look at the performance of this model, pretty much it's as I expected in here that the model is not great with knowledge sorts of things.

8:00 You can see here I asked it for a 5,000word essay. It's giving me a reasonably short amount of thinking on it, but the essay is nowhere near 5,000 words, not even close in here. If we go

8:11 in and have a look at things like the logic puzzles, we'll see that this will fly along and it gets to the right answer here quite quickly. So the logic stuff is actually going quite well. The

8:24 same here for when we look at something like the hard math stuff, it's doing well and that kind of reinforces what this model is actually good at, right? But if we come over and we look at

8:34 something like the SVG Pelican test, this thing just doesn't do well at this kind of thing. You can see that we're starting to get really high token speeds on this. But when we actually look at

8:45 the outputs, maybe they actually are sort of better as abstract art for certain things. But that's okay because that's not what the model's for. Exactly the same kind of thing when we look at

8:56 the generate an HTML page, right? If we look at the code that comes out, it's not that it's necessarily anything hugely wrong. It's just that it's not nowhere near as good as any of the other

9:08 models that we would use for coding these days. That's not what this model is actually about. Where this model shines is going to be not at the coding outputs, but it's going to be at the

9:20 tool use. So, you can see now I've moved over to the GGUF version. It's flying at well over 400 tokens per second. No problems here. But let's come over and look at how this actually does on the

9:32 agentic tests. Remember, these are a bunch of tests that test for function calling to see does it provide the right arguments to functions, is it able to use tool use well and stuff like that.

9:45 And this is sort of like the basic set of tests here. And not only is it flying speedwise here, it's getting all of them right. So, we're getting some really nice results out of them. So let me come

9:57 over to the more advanced one. So here the goal is to basically test things that often go wrong when you're running these kinds of things. And so again I'm using the gguf version. I'll just say

10:11 going through this that the current setup with the sg lang is actually seems to be the tool use seems to be returning the wrong format. So when I test out the Bflat 16 and the G-Spark versions, I'm

10:22 getting some issues, but it's pretty clear that they're not because of the model, just seems to be the configuration on SG Lang. When I try it on Llama CPP with the GGUF version, so

10:33 this is a 4bit version. And of course with this, you would expect that's actually going to be worse. It works fine. And you can see here that it's gone through and passed eight out of

10:43 eight for all of these tests. Now, some of the tests in here that I think are interesting are the sort of planning tests where it's using multiple different tools as it goes through. It

10:53 has to deal with errors often before it gets to a correct output. With something like the prompt injection resistance, it has to work out how to basically extract out the right kind of thing going in

11:05 here. And I'll point out while I'm showing you one run of these tests here, I run them at least 10 times going through. Now, this one did fail a couple of the times, but it's been passing

11:15 around about 80% of that. The other one, too, which is one that a lot of models in the past used to have lots of problems with, were things like failure retry tests. So, what's going on here is

11:27 this is deliberately pretending to fail for two of the times and to see does the model keep going at trying to get the result out. So, you can see that these failures were actually created

11:40 deliberately. The model then keeps going. It's able to basically get the result out we want. The distractor toolbait also very impressive in that it's able to basically find the right

11:50 information in there. If I come and test things out in the tool playground here, I think it's really interesting to see that this model has no problem both dealing with multiple tool calls here.

12:03 But you can see that this one it's having to use duck.go search and a fetch URL here. And it's lightning fast going through this. I'm getting sort of end to end over 500 tokens per second. And I

12:16 guess partly because it's so small, it has no problems with large page preills being able to extract out what's the right information in there to pay attention to. Again, on multi-tool

12:26 stuff, again, we've got very fast result. We've got accurate use of the tools that are actually in there. And one of the things that I really like in here is that it's not trying to use its

12:38 knowledge. Right? The fact that it doesn't have the knowledge is fine as long as it doesn't pretend to have the knowledge. And we want it to actually defer to its tools rather than try to

12:48 basically make everything up itself. So for me, this is the biggest issue with a model like this is that if we do actually give it a lot of tools and we ask it to defer to the tools rather than

12:58 its own knowledge, does it have a problem with that or not? And it's great to see that if you give it the tools, it seems to know not to use its own knowledge. And that's exactly the kind

13:08 of model that we want for sub agents, right? for anything that you're doing agentic where you've got sub agents with lots of tool calls. You don't want them trying to use their knowledge too much

13:20 in there. You want them just getting the right stuff out for the right inputs for the different tools, etc. And that's exactly what's going on here with this model. Okay, so to finish up, there are

13:31 a number of different versions that they've got of this up there. The ones that I showed in the video were the standard one, the GGUF, and the G-Spark one. Definitely this is worth checking

13:41 out if you're looking for a very very quick model that needs to deal with perhaps a little bit more complicated function calling and tool use than the previous 1B model. You can see in here

13:54 that its strength has really been trained and pushed like crazy for this tool use. And while it also did well in the math reasoning and some of the other instruction following stuff that I gave

14:05 it, the way that I would suggest that you use the model is for sub agents and for where you need a very quick but reasonably simple kind of model that can handle a lot of different tools and can

14:16 deliver good arguments or the appropriate information to pass into those tools. So anyway, let me know in the comments what you think of it. As always, if you found the video useful,

14:25 please click like and subscribe and I will talk to you in the next video. Bye for now.