Bonsai 2 27B First Test – Is THIS the BEST Single-GPU AI Model?

0:00 I don't like the way that it made me look like a blob. But aside from that, the Quenbear looks pretty good. Today, we're going to be taking a look at Bonsai 22 27B from Prism ML, which is

0:11 essentially simply a very, very reduced size version of Quen 3.827B, something that can realistically run on single GPUs with 12 gigs of RAM or 16 gigs of RAM. So, this really can open up

0:25 a lot of options for folks who want to run local AI that don't require a ridiculously powerful system like a lot of recent open models have. So, before we get into it, do feel free to

0:35 subscribe and let's just start out by taking a look at what exactly this is. Now, this may be somewhat familiar as I have covered some of their previous models. Most recently, they had a bonsai

0:46 version of the Quen 3.6 27B. However, the 3.8 model is such a leap in capability over the 3.6 6. It does seem like this Bonsai version of it is going to also maintain that leap, at least

0:58 compared to the previous Bonsai. Now, there's kind of like a lot in terms of the technical specifics of these models. And I've been thinking, what is the best approach to showcase these? And to be

1:08 honest with you, we're going to be doing a sidebyside test with both of them using Pi. And that's what we have running right here. We'll get into what both of them means in a little. And I

1:18 think the best way to really test these is to just run them through some of the benchmarks that we have a ton of reference points to compare with to see like, oh, this actually made a really

1:27 good Subway FPS or something like that. Just a lot of comparisons that already exist, I think, is the best way to test these on so we can see if basically they're good or bad. So, with that,

1:37 let's start out just by taking a look at some of the interesting things about these and then we'll jump into some benchmarking. Now, we can see right here in the highlight section that this is a

1:45 reduction in size from the 54 GB FP16 version of Quen 3.827B down to 5.9 GB at the absolute smallest. And it should be mentioned pretty prominently that there are two versions

1:59 of these and that's why we saw two sidebyside agent windows. So, we have this PTQ 1.0 which packs the weights in a smaller way which represents the 5.9 GB file. We also have the PQ2_0

2:14 which stores them with a bit more information I suppose to put it simply and that comes in at 7.21 GB. So compared to the FP16 version of 54 GB we have a reduction in size bringing it

2:26 basically to 6 gigs or 7.2. Really a gigantic reduction. And what they prominently list here is that 98.2% of FP16 intelligence has been retained. Now, I don't want to say that's

2:39 misleading, but I do kind of want us to temper our expectations a bit. So, if we go into the linked white paper that is listed right here, they actually have a section talking about how Bonsai 2 is

2:50 representative of the work that they wanted to do for these Bonsai models to get them better at longer context and agentic coding. And at the bottom of this paragraph here in the white paper,

3:00 they say that these retain roughly 3/4 of the full precision performance on some benchmarks related to coding and agentic performance. So don't come in expecting 98.2% of the retained

3:12 capability, but they say 3/4, but still even 75% capability retained with such a significant decrease is the exciting part of these models. As Claude would say, that is really the loadbearing

3:24 thing here. So I am currently running these on FP16 KV cache. Now this is mentioned as experimental here and this will really get you to a lower lower memory footprint. However, as they say,

3:37 it's experimental and the quality may kind of degrade as one would expect it to be. However, the 100K token context window here would use about 1.8 gigs of VRAM instead of 6.3. So this is a pretty

3:49 significant reduction in the overall memory footprint. And if you are memory constrained, it would also probably be prudent to have this enabled in your test setup because then you'll be able

3:59 to have longer context windows and can have it do like agentic runs and things like that. So, I just want to specifically mention this because it can be also a bit confusing looking at this

4:09 and then if you go to download and load it in and realize, hey, why is this using like 10 or 12 gigs of VRAM? That's kind of the reason why. And some additional memory saving considerations

4:18 just are the vision capabilities are optional here. So, if you don't really care to use these with their ability to see things, you can also discard those and save some overhead. So, there's a

4:28 few other additional ways here just to get the absolute lowest footprint if you're kind of dead set on running it on something that is pretty small. And then they just have a bunch more about like

4:38 things to get it set up and running on your system, things that like an agent can take care of for you, as well as benchmarks of speeds here on specific cards. Now, for today's testing, and

4:48 that's where we'll probably just swap into actually taking a look at how these perform. I am using both of these. They're both being served on the same exact card. It's a 6000 Pro that I

4:57 bought when it was like less than half of the price it currently is. Just to point that out, cuz GPU prices have been crazy, which is why models like this are so expensive. So, we have, of course,

5:06 begun with the browser OS Test V2.7. We see both of these are working right here. And in the lefth hand pane we have the 1.75 BPW version. So the smaller 5.95 GB one. And then in the right pane

5:20 we have the 2.13. So the larger more performant but also bigger I think 7.2 GB version. So we'll see how these do. It's something that previously these did have some trouble where like okay they

5:33 don't really know when to stop. They can kind of just keep rambling on and on maybe like I'm doing right here. So, we'll wait for these to finish and then we'll take a look at the results side by

5:42 side and then we'll get into some more difficult things. I want to try these with Blender and GDO because why not? So, something right here I just kind of want to point out that we're noticing is

5:52 both of them have been working for quite a bit and we're like, okay, the first one was not good. I'm going to redo it from scratch. This is something that is definitely a concern at least in terms

6:02 of like them functioning agentically is are they actually capable of writing out the files and making the changes or do they just overthink it? Both of them are still doing like the second or third

6:14 rewrite of the browser OS results. However, they did also both just create something. I almost have half of mine just to look at what we get here in case they stay in a loop. Okay, so this right

6:24 here is is the more quantized or smaller one. We can definitely see there's something not quite right here. So I suppose the rewrite was a prudent choice in that regard. Let's try the Q2 one

6:38 instead. Maybe we'll see a little bit more functionality. Nimbus. Okay, we have a blinking cursor right there. Usually that's followed by some form of like loading animation.

6:48 Again, this is very I would like file these under experimental. And neither of them did fully successfully finish writing, so we'll just allow them to go. But I do

7:00 have concerns that they could like kind of just do this a bunch. Yeah. So, it's going to try to rewrite the entire thing again, which is just not what we want.

7:12 All right. It seems like the larger of the two has finished writing the browser OS now after I'd stopped it and told it to just write it and don't overthink. I at this point I'm not feeling very

7:25 bullish about this and may have to just swap to a much simpler task. However, let's just Okay. Yeah, this is trying to write the whole file again. I'm stopping these.

7:36 This task is a failure. It It was too difficult for them. So, let's try something simpler, maybe. All right, for the smaller of the two, I'm giving it a significantly simplified prompt. Just

7:46 saying, "Create a beautiful website for Steve's PC repair. Do not overthink." I suppose we should just do this for both of them. I really want to see if they can spit out a game. When I previously

7:56 tested the version of this that was based off of the Quen 3.627B, I did get a partially functional Subway FPS from the larger of the two models. So, I'm kind of hoping that we get

8:07 something better than that today. However, the browser OS test was just not good. And I don't even know that I would really go out on a limb and say that would reflect a 75% of the way

8:18 there coding capability. So, there's definitely some claims here that may stack up in very specific niche benchmark scenarios. However, I believe that this is a very optimistic

8:29 generalization of the performance. And I want to just specifically call that out because folks may see this and be like, "Oh my goodness, my card like I can run something now." and then be heavily

8:39 disappointed. So, that kind of just rubs me the wrong way. All right, I'm happy to say very quickly the larger of the two got this done. It did notice there were some issues. This is actually I

8:50 can't remember exactly. So, just interesting to see that in the chain of thought. This did finish and produce something that passed its syntax check. So, let's actually take a look at this.

8:58 And this is a much simpler task. I do believe that telling them not to overthink specifically in the prompt is probably a good thing to do. Okay. I mean, very simple. Yes, nothing special

9:09 here at all. However, the point is it quickly did this. It didn't overthink. It did not get stuck in a loop. So, consider this almost some form of like verification that it's safe to proceed

9:18 with a little more complexity. And overall, not bad. We'll keep this up just so if the other one does finish the smaller of the two, then we can also just sidebyside compare them. All right.

9:29 I'm a little worried cuz this is like flying past like 5,500 lines of Yep. Okay. So that was that's what I was concerned about. And yeah, so okay, let's just uh I didn't

9:44 really like nothing else came to mind. So we'll just give this one more chance. All right, I put in an extra closing HTML tag there. Let me just click on it anyway just to see because it did

9:58 actually end up writing. Your PC is slow. Bend it back. Okay, it's interesting because in some ways this is a little better because it's got more content on it even to the point

10:14 people who fix it themselves still call Steve. Oh, okay. My daughter's laptop wouldn't boot for 3 months. I already bought a new one. Steve pulled the old drive, fixed the BIOS, and had it

10:23 booting in 40 minutes. She's still using it. I'm still annoyed I wasted the new one. Okay, good. So, it did show that like the extra purchase was a waste of money. Just some good semantic

10:32 understanding there. Interesting. So, here's the larger one. Maybe a nicer contact form. And then this one also was I mean not horrible all things considered. So, I wanted to just verify

10:47 both of them were capable of actually outputting something and then we can move on and have them do a C++ skateboard game. So, next up, we're just giving these a very simplified version

10:56 of the self-contained C++ skate game prompt where it's create a simple 3D skateboard game in C++. Do not overthink. I don't care if they use Ray in this case because as we saw with the

11:07 browser OS, were I to give these the full complicated prompt for this just based off of what we've seen so far, it would likely just be a waste of time. So, I want to try to give them things

11:16 that I believe they reasonably have a chance of doing. All right. So, we're on to the third rewrite of the file now with the larger of the two and then the smaller of the

11:25 two in the lefth hand pane is like tried writing it a few times and then just starts thinking midwrite. All right, I got to stop the the smaller one here.

11:37 It's it's freaking out about like how it's going to write text and neither of them opted to use RayB which is interesting. So, I had to stop it and say, "We don't need text. Just make the

11:47 3D game." And we'll see if that helps at all. I'm going to give that same feedback to the larger one. All right, the bigger one compiled cleanly. Let's see if it pops up right

11:58 here on the screen. We may see like a a teaser of Okay, good. Okay, but it opened real quick and then it closed. So, that's actually Okay, the other one compiled cleanly, too.

12:14 Seems like no text was the key here um for this. Even though they could have used RAID, but I didn't tell them not to. But let me just kind of try to snipe these real quick. Let's just look at

12:27 this one real. That's it's something. That was the the smaller of the two. Let's just look at the larger of the two. So, that one just

12:38 opens and closes real quick. the smaller one actually. Oh, okay. It just opened it itself there or I clicked it too many times. So, we at least have one reasonable thing here now. Oh, that's

12:52 okay. We We'll wait. It opened and it did it all in like raw C++. So, that's actually okay. I'm going to stop this one because it's just freaking out. All right. It's still unfortunately bad. And

13:06 we got one working one. So, I'm going to stop it here. At least we got the this. So, you know, something. Next up, I want to try something with having them use Blender through the MCP. So, they

13:21 both have their own Blender instance set up. Right here, we can see they're running on separate ports. And the prompt I'm giving them is to create a scene of a city block, render it, and

13:30 then save the rendered image directly to the directory that you're working in. All right, this is actually cool. So, the larger of the two did render it. It saved the image and it's using its

13:39 multimodal capability to look at it. And it said the camera is inside a building. It's rendering a close-up of one wall of the building and then it gives itself the coordinates of the camera. I am

13:48 actually just somewhat curious to take a look at what specifically the image does look like before it Oh, okay. Yeah, that looks I mean, we see some sky there or some other form. So, it did render

14:00 something and create a photo of it and then look at it and realize, hey, this isn't right. So that's a good step actually. All right. And we also got a rendered image for the

14:10 Oh, okay. So the smaller one rendered something as well, but it's saying the renders a flat gray. The camera is looking at nothing. So there's definitely some issue there. Although to

14:19 be honest with you, I am curious to take a look at it. Okay. Yeah. So that's what our smaller one received for its rendered image. So both of these have completed their task. I was not watching

14:28 them towards the end. I had to step away for a second, but we have renders. So, okay, here's the smaller of the two. It did create a city block and it put a camera in it and rendered it. So, that

14:39 is a successful completion of the task, albeit perhaps a bit simple. However, we didn't necessarily denote much detail. And then here's the larger. We can see it has chosen some different colors.

14:50 Unfortunately, the buildings are floating in midair, but there is also some nice green foliage here with some like visible triangles, which is nice. I like that aesthetic. So overall, they

15:02 did actually both successfully complete the task and use Blender to do this. So not super complicated, however acceptable. All right, next up, I'm going to be giving them the full Subway

15:13 FPS prompt. I have not included anything saying like don't overthink or do it simply. I'm just giving them the same exact one that I've been giving all the recent larger models. I would imagine

15:23 this is probably going to take quite a bit of time. So I'm basically just going to leave them running here and hope to come back to something that's playable. This is the one also where the enemies

15:32 get delivered by train as the waves increase, which we'll hopefully see. All right, so hypothetically, we have a completed version of this from our smaller version. So, I had stopped the

15:44 big one right here and said like, "Stop trying to test it. Just give me the game." They've been going for like an hour and a half. So, I figured like at this point you, you know. All right, we

15:54 have a completed thing for this. Oh, I forgot to remove the city block. That's okay. and it just gives me a command. So, it just says run it on server. All right, let's take a look at this is the

16:04 smaller of the two. That's actually a decent front end. Okay, we have the commands and things like this. We also have multiple different weapons. The big issue, of course, is the fact that we're

16:16 not actually able to enter. All right, let's check the larger of the two. I'm a little more bullish on hoping that that one will work. So, okay, we do have our script right there. Very similar. Click

16:27 to start. Please do something. All right. I tried fixing the smaller one. I used claw just to be like, fix this script. Don't play test it. I will show like specifically what it assessed

16:43 was the issue. So keep that in mind. This is a failure. But after waiting for like an hour and 40 minutes, I really just wanted to see what it at least had put together. So this is what we have

16:54 here. The gameplay loop doesn't work at all, but we actually see it loads into a relatively competently made 3D scene at least. So, I am actually okay with that. So, none of the keys really respond. But

17:05 now, I'm going to do the same thing with the larger of the two because perhaps we'll have a little more playability with that. But keep in mind, both of these are failures. I'm using Claude to

17:14 fix the syntax that was produced. All right. So, on the left right here is what Claude did to fix the script that was created by the larger of the two bonsai models. So, this is everything

17:24 that was fixed prior to looking at that. We still have not yet looked at it, but I want to showcase what needed to be fixed after the fact. On the right here is what was fixed with the smaller of

17:34 the two where we saw once this was done, it did actually load into a simple subway scene. There was no playability, but I just want to specifically showcase what was needed to just get it across

17:44 the finish line. So, with that, let's actually take a look and see if this was properly fixed, at least the larger of the two. If we click to start. Okay. So,

17:56 unfortunately, there was perhaps something additionally wrong and and because, you know, it took such a while, I do kind of just want to All right. Apparently,

18:08 that's been fixed. So, let's Oh, okay. And then we just get a bunch of other errors. So, at this point, I'm just going to call this a failure because it's too much additional work that would

18:19 need to be done by a model that is not the Bonsai one, but we did get something with a little bit of assistance from Claude for the Subway FPS for the smaller one. All right, so the next

18:30 thing I'm going to do, and depending on how this goes, maybe the final, is the beautiful watch website test. This is one that I think is at the crossover of complexity, but still realistically able

18:40 to be done by these. I have also included at the end, do not overthink just because that seems to produce better results that are actually finished quicker. So, we'll see. This is

18:50 the watch website. It has to create a 3D model of it, a cinematic panning effect in the hero section, and this also denotes an exploded view of the watch somewhere down the page. All right, so

19:00 both of these finished the watch website. Unfortunately, we basically have two decent looking websites. I will say the structure of either of these are much better than the

19:11 Steve the PC repairman one. So what we're looking at right here is the one from the lefth hand pane. The smaller of the two. Everything here actually looks okay. The hover effect isn't bad. The

19:21 problem is and what I had suspected may be the failure point for this test is neither of them had the actual 3D model show up. So this right here is from the larger of the two or the one in the

19:31 right hand pane. And unfortunately, we have some nice UI elements that would have corresponded well with the 3D models, but they just didn't appear. And that was just something that I kind of

19:43 suspected would probably trick it, but I wanted to try anyway. So, these are respectable websites in terms of just some of the aesthetic choices, but kind of not fully functional. I do want to

19:54 try both of these on medium reasoning effort. up until this everything we did on X high the maximum possible one it is a potential that maybe these tend to overthink on that and you can maybe

20:05 sometimes get better performance in a weird experimental model like this if it's on a lower reasoning level so for this task I've given them both a photo in this case the photo of me holding the

20:16 Quendall and I said make an SVG of this photo and I put it in either of their respective folders so I'm almost a little concerned and nervous to see what they come up with, but it should be

20:27 entertaining. P. Okay. Install image to vector. Interesting. We'll we'll see who again. Okay. So, they're just I will say it's clever the way they're going about this,

20:41 but it was not my intention. So, I'm going to stop them and clarify. All right. So, that'll get them to like make it themselves. This is actually a pretty interesting test of its vision

20:53 capability though as well because everything it's describing right here is based off of actually being able to see the image in the repository and the Quen the base model the vision capability of

21:03 it the model this is based on I should say not the base model is actually quite competent hair slightly receding in front okay so these are both still working but we did get a render of the

21:15 larger of the two and I'm just out of curiosity you You know what? That's I It's not wonderful. However, all things considered, this is on medium

21:28 reasoning effort. It is a bonsai version with vision capability. This is actually not half bad. It has the microphone boom. It has the pop filter, whatever this piece of mesh is in front of my

21:40 mouth. It even has the line of the boom going to it, the wood slats on the wall, some other things. The Quen text is actually written. It even tried to get the logo from my shirt and it's still

21:50 working on this here. It did give me a pretty favorable hairline as well, which I am quite pleased with. So, I have to say this is actually, all things considered, this is not half

22:01 bad. I wonder if we have any results so far for the smaller of the two. Wow. Yeah, we do. And that's a significantly significantly more impressive result. Interesting. So, that is the smaller.

22:14 That's the 5.95 GB version of the model. That's downright like decent. This is actually pretty impressive. I made a mistake at the end of the last right. All right. But the good thing is that

22:25 the bigger version seems happy enough. It renders correctly. I'm going to stop this one because we already saw like basically 90% of what we would see anyway. So, let's do a definitive look

22:38 at that one while the other one hopefully finishes up. Oh, that's the wrong thing. That was when we first started it and it was trying to just like do that. Okay, that's definitely

22:48 much better. And this is the larger of the two. Well done. Well, I don't like the way that it made me look like a blob, but aside from that, the Quenbear looks pretty good. It seems very serene.

23:02 It mistakenly put a window there, but there was art. Everything else here is all right. This is actually for vision capability assessment as well. I'm impressed with what I'm seeing. Again,

23:11 pretty well done. I would have to go out on a limb here and say that I think the smaller one did a better job with this to be honest with you. So make of that what you will. Both of these were done

23:20 on medium and I think I know I've said this probably a couple times now. There's one final thing I want to try with them on medium medium effort. So the final thing I want to do here is

23:30 just on medium give them a simple 3D skateboarding game task that just needs to be contained in a single HTML file. I have specifically said a couple of things. Do not try to play test it. just

23:42 deliver the file and also don't overthink. Good. And we can see right here it's really quickly transitioning into writing. Whereas before on X high and without adding in some of those

23:54 things reinforcing simplicity, it would think for a lot longer. All right, they're both now into the second rewrite or the third. I think the second. Awesome. Both finished basically right

24:04 at the same time. We haven't looked at it yet, but I am pretty happy with the fact that they got this done pretty quick in the scheme of like what we've been used to. So, here's our All right,

24:15 this is the um the larger of the two. Yeah. So, unfortunately, we get nothing. However, I will make note that when I first moved when we had initially loaded this in, we did collect some coins. So,

24:29 there is some form of a game state registering here. Let's just take a peek at the smaller of the two. Again, it's interesting that this one seems to be producing some superior

24:42 results in certain scenarios. So, I don't quite know what to make of that, but it's just kind of what we're noticing here. At least it gave us something initially off the bat. I think

24:50 the final thing to do here is just say I'm going to give it the one piece of feedback, a description of the issue. We'll see if it fixes it simply or not. Nice. Good. So, it did fix it. I will

25:02 say this is again this is the larger of the two after a bit of feedback. Look at that. That is actually, you know, I'm okay with this. I'm pretty pleased with this actually. And we had this on medium

25:13 effort as well. So it made this little 3D game. I think really including and driving home like make it simple. Don't try to worry about like play testing it too much. I think that may be the

25:24 difference a lot of times in using these between success and failure. And I will note that this was done on medium as well. So this actually this is a respectable job and I am quite pleased

25:35 with what I'm seeing here. I know it's simple, but when you consider what this model is, I think this is pretty darn impressive. I mean, not so long ago, this would have been a result made from

25:45 like a Frontier state-of-the-art model. So pretty cool to see the progress. And we can collect coins. It even put in like some form of gameplay elements beyond just the skate game. So I think

25:56 that's very cool. Now the last thing I wanted to do is I've just gone here for this model being served by Alibaba cloud. So they'll be serving it correctly on open router medium thinking

26:07 effort and I gave it the same exact prompt right here. Keeping in mind this wasn't working aentically in a harness but I just want to see what the difference is. So here's the one that we

26:16 just got from open router. Okay. And we see that this is definitely a bit more competent. However, I will say it didn't have the cool like coin playing style and also the board doesn't

26:29 actually seem Oh, okay. It does do tricks. I just didn't figure out what the keys were for some reason. I can't seem to get it to do it again, but it did. We saw it kick flip. So, there's

26:39 definitely some differences. This is actually, I would say, a pretty strong comparison result for Bonsai compared to what we see right here from the full version of the model on the same medium

26:49 reasoning effort. So, overall, I don't really have a lot to say. I just wanted to do a test of this because I know there's going to be a ton of interest in this. I've seen online already there has

27:01 been. I bet if I refresh this, this community number will have gone up a bit. Definitely. So, this is very cool. I would say my only kind of disappointment with it is it seems like

27:11 they're so advertised as like this is basically like a stone throw away from the capability of the full model but this much smaller. Based on the tests, I don't know that that's necessarily the

27:24 case, but I think that does a disservice to the actual coolness behind this model. This is a massive reduction in size. There's even more optimizations to go further like the 4-bit KV cache. You

27:35 can omit the vision components from this as well. So, this can really get down to a tiny footprint. And I would say the fact that these models can actually produce lucid code, can make tool calls

27:46 and function in an agentic harness over a longer context is fantastic. It is so darn cool. So, I don't like seeing like, oh, this is like an Opus at home because it's not. But that does a disservice and

27:59 takes away from the fact of what it is, which is pretty darn cool and incredibly impressive from a technological standpoint. So, I wanted to test these. I know there will be a lot of interest

28:09 about these. I will definitely continue to test some of these Bonsai models. I believe someone had asked here, hey, you guys should do this with the Quen 3.8 Flash Next, which would be extremely

28:19 extremely interesting to test. I have it running in front of me like over there right now building something. So, that's a great model and I would be very eager to see that from this lab. So, if you

28:29 have any questions, feel free to leave them in the comments and thanks for watching.