Bonsai 2 27B and Hermes Agentic Coding Local Ai Test vs Qwen 3.8 27B fp16

0:00 you don't have a lot of VRAM, but you want the best model possible. One of the biggest questions I get on the channel is, "What can I run in just one 3090 or maybe a couple of 3060s?" Today, we're

0:10 going to test this out, and we're going to be using the Turnary Bonsai 227B. Now, if you're not familiar, Turnary means negative 1, 0, and positive 1 as representation states instead of just

0:21 negative 1 and positive 1. This should give a little bit better quality to the smaller size that we're looking at here, which is nine times smaller. So, it can fit in 24 GB of VRAM, the full 27B with

0:35 a very, very minimal loss in quality. So, they claim. We're going to put this to the test today, however. So, we're going to jump right in. You I'll I'll put the link. You can learn more about

0:44 all of this stuff, but I know everybody's like, "Let's get to the uh meat and potatoes here." So, I'll just run you through the steps to get this up and running. I'm going to assume that

0:52 you have followed along with the guide on digital spaceport, which is the Proxmox 9 LXC guide. That's what I'm going to be using here over on Prox 2, and we're going to be running in

1:00 container 118. So, I've already sshed into this container. We're going to paste this in and run it. This should go ahead. It's going to ask for your hugging face key if you want to give it

1:12 that. I'm just going to skip it because this is a small model. It'll still download very fast, but basically this should be able hopefully with this demo to run a really high quality with

1:24 amazing retention of precision at really, I hope, decent speeds. See, speed has been one of the biggest problems for people running, you know, smaller GPUs and sparks and stuff like

1:35 that is that is a pretty big impact that you have as far as running a dense like this. So, if we can run this and get decent speeds on it, it really opens up a new frontier possibly for people who

1:45 have 13090 or maybe two 3060s even. And today I tossed in two 3060 12 GB. So, we're also going to get a speed test on those. We're going to do the in-depth testing on the single 3090. However, and

1:58 if you're interested in what the prices look like, you can find those out on our new feature on digital spaceport.com. And my wife is updating this quite frequently. We are seeing $1,400 is the

2:08 lowest used price right now for 3090s. That is up actually over the past couple days quite substantially. Of course, 5090s have gone through the roof this past week. And 4090s and 3090s are kind

2:19 of following in that trail. But I will say the 3060 we've got down here is $300 the lowest used price for that right now. And so we're going to test that one out also see what the kind of

2:29 performance is. You can find that in the links below. That is uh digitalspaceport.com/gpus to get there. also. So, let's see if it's got that all downloaded yet. Okay,

2:41 so it's got it all downloaded. Perfect. The next thing that we're going to do is pull up one of the commands from their git repo. So, there's a couple things you want to check out here. So, like the

2:50 environment variables is something you want to pay attention to. So, we're going to use bonsai host as a variable to override the local host. Now, if you're running this on a machine and not

2:59 serving it over your own local network, you can just leave it as local host and that'll be fine. But we're going to override the local host address here when we set ours up and bind it to

3:08 0.0.0.0. That means we'll be able to access it from any location on the internal network here. As well, there's speculative, the KV cache set to four if you need to really crunch it down. And a

3:18 contact size. I'm going to leave it at auto for the contact size. And I'm not going to set speculative or KV4 for this test. And we're going to give it kind of the same test that we have seen with

3:27 this exact same model when we tried it at full precision from Quinn. So, we're going to CD into it looks like scripts and yeah, that's it over there. So, you've got start llama server.sh. So,

3:42 get our override in there. And that should get us up and running here. And we can see our 3090 down here. Hopefully, we get that loaded up just really quickly. and we'll take a look at

3:59 what the memory footprint looks like as we're running this also here. So, it looks like it is loaded up. It is currently occupying 16.772 GB worth of space on the GPU of the 24

4:10 GB that it has available. So, that gives quite a bit of extra headroom there. It should be accessible also on 192.168.1.61. And I believe this is going to be port

4:21 8080 is the default. Okay, so we can see that it is loaded up. Howdy. and see what kind of performance we're getting. Okay, so like 69 tokens per second is what we're seeing right off the bat

4:31 there. And we're going to give it a very open-ended make an animated SVG of a cat walking on a fence at night. And I told it to make it awesome. You can see we're

4:47 holding pretty steady here at about 69 uh tokens per second. Very much faster than what we would be looking at if we were, of course, this is decode. If we were looking at running the full 27B at

4:58 FP16, which would be about 35 to maybe 40 with four 3090s running at the same time on that in VLLM. Of course, what we're running here is Llama C++. This does not have upstream support yet

5:11 integrated into the core of Llama C++ for right. So, for right now, you need to run their fork that you can uh grab just like I'm grabbing here to be able to get correct outputs from it. still

5:23 holding at about 68 tokens a second. Okay, let's see how much context it gave us. So is it set our context at 131. That probably won't be a problem, but definitely if you're running a

5:34 workflows, you might want to go higher than that. If you're doing something like Hermes agent, setting that up a little bit higher, especially at these speeds. I don't know what the prefill is

5:41 going to look like, but we'll get to checking out what the prefill looks like pretty soon here. I'll have it maybe do another revision or something like that. should retain vision support. Like I

5:50 mentioned, it does have the Dlash speculative decoder also built into it. So, it should be for certain types of tasks really, really good. But if you're doing like structured code and stuff

5:59 like that or math, it should be actually pretty darn decent at going through that and getting a very serious speed increase. Okay,

6:32 that's a bad fence, but the cat walking across it was actually pretty good. Uh, [laughter] it's not the best fence that I've ever seen. Let's Let's take a look at this

6:42 larger. So, I think the cat is one of the best cats walking that I've ever seen uh one of these make. There was Okay, the shooting star, it looks like it's hitting the moon like direct

6:52 impact. Um, but the fence is pretty bad. The cat walking pretty good. Even has a little shadow. I really think this is a creative expression that we're seeing out of it. I I would have to say I like

7:05 the cat a lot. I think this is actually one of the better ones that we've seen, but definitely definitely I'm not sure if this reaches quite the same quality as the FP16 would, but it did it so fast

7:16 that, you know, I've got to say I'm going to save this for posterity. Maybe put it up on the website on digitalspaceport.com. So yeah, this is

7:28 this got here and let's just take a look at kind of like what it took. So it took 25,000 tokens. It really didn't do very much prop processing at all for the most part. Hit about 64.4 tokens per second.

7:39 So it works pretty good. I mean it is functioning pretty good. Let's go ahead and toss up our Hermes agent here. Okay. definitely has kicked up and started

7:57 thinking now. Also, it looks like and it looks like the prompt processing is kicking in right about 1,250 tokens per second. Okay, so we're going to run this against head-to-head what we got

8:12 out of the outputs for Quinn 3.827B at FP16, which was pretty stellar for the arcade results if you remember that video. And if you don't, you have the chance today to go play this at

8:23 arcade.digitalspaceport.com and you can headtohead the versions yourself and you let me know in the comments below what you think about the two variants. And we're going to have it

8:32 run the full suite that we had generate before. This should create a complete video game arcade console style and actually have everything up and running pretty pretty good as soon as it's done.

8:46 Good luck to it. This is a hard one. It took quite a bit of time in the past. So let's see. I think I need to change the text here because that should update. So I need to change that one thing

9:05 and it is off. Okay. So it has been a really really long time. It is now the morning. I got about four maybe five hours of very light sleep there and this is what we

9:22 have I would say. Let's take a look at this versus what the full FP16 produced. And there is a quality difference here. I would say just in the landing page we've got Neon Laner, Space Racer, and

9:38 Pixel Breaker. So, we're going to go through play these. It doesn't sound like there's an audio track here, which uh sometimes maybe not that bad of a thing. So, we'll we'll figure it out

9:49 here. But let's let's get Neon Laner running. Okay, so there's already kind of some font issues here. I would say this pattern is This is crazy. Okay. Okay. So, pretty simple.

10:11 Shooting the things. Hey, what what game was this exactly? I don't know. You tell me in the comments below. I would say this is definitely not as fun as what we ran into with the other one. I'm not

10:22 sure that I can die. Hold on a second. Let me see if I get hit by stuff. If I No, nothing seems to actually harm me here. I can't move forward or back, but just side to side. Nothing seems to be

10:33 causing any problems. And this is an impossible [music] amount of enemies or whatever is going on here. Feels like there's a few problems. Although, now that I've got a faster

10:43 shooting stream, wow, look at me take them out now. The score is very, very small up there, but it's in the upper left. Very far, 16,000 or so. I guess I won or something. I guess I've got a

10:54 boss up here. The boss is shooting in the wrong direction, though. So, the boss is definitely shooting in the wrong direction, it does appear to me. So,

11:04 that is a bit of a fail. Not as good. It It's playable. Hitting escape here doesn't seem to be able to do anything. Got that. So, neon laner. I'm going to give it a um I got C minus

11:18 D plus. Uh not really that great. Next, let's check out Space Racer. Thread and thread the lanes. Dodge the pack and burn the boost to carve checkpoints out

11:31 of the void near misses. is refill your turbo. So maybe this will be better. Hopefully this will we still have the weird font thing going on. So definitely uh not as good. I mean that's definitely

11:43 not as good. Uhoh. Uh I think I got killed there. Uh oh. Okay. This does not have very good movement controls. Uh oh. Okay. So, Lane

12:00 Racer is pretty impossible to play, honestly. So, let's give the last one a try here. Pixel Breaker. So, definitely [music] definitely a quality difference.

12:16 That's not looking good up there. I don't think that's supposed to be going like that. No. No. I think pixel breaker also. So,

12:27 we've got two Fs and a D+ essentially uh on the on the challenge here. So, I think that is a fairly decent evaluation of is this 98% of what we ran into with the other one. And I've got to say, I

12:44 really don't feel like it is. Granted, there were some things about this one that I thought were really, you know, like hard, but this was very very true

13:00 to like what I was expecting kind of the music was kind of like corny and like it it it was better gameplay. Granted, it was not full screen, which I would have preferred it had it been a

13:13 larger screen size. Definitely playable though and if you hit the backspace, it worked. The same was true also when you would look at the star breakers. This this was actually Okay, let me just miss

13:27 that right off the bat. Uh, this was actually pretty darn good. And wow, I'm talking and doing this is not working out for me right now. Like I said, not a whole lot of sleep. So, even when it

13:39 bounced up there, it didn't get trapped. It kind of went bounced around. Still did the thing that you would expect it to [music] do. So, much better quality overall. the difference between the FP16

13:49 [music] and what we're seeing. However, there are some things that really went right with this. As far as the chat, as far as the cat, I've got to say both of those

13:59 actually went very smoothly. The chat speed is very good. The tool calling, I did not see miss tool calls. So, I thought that was also a really positive. On the negative side, the quality is

14:11 definitely different. Not as good. Definitely not in my opinion 98%. Is it still really pretty awesome though? Yes. Is it substantially faster? Well, yes. It is a

14:26 absolutely substantially faster. Is it potentially something that is able to get better? That is a good question. Is it something usable? Yeah, if you want to have a chat and you want to have a

14:37 single stream agentic workflow with something totally usable for that. if you want to do agentic code development, probably not going to be a good time. So, that's my feelings on it and I look

14:49 forward to reading yours in the comments below. So, huge hat tip to all of our channel members and everybody that likes, shares, and subscribes. You can do that by hitting the button down

14:58 below. And thank you everybody for getting us across the line. We are past 100,000 subscribers now. That is a really awesome milestone and we've got a lot more really fun and great things for

15:10 you in store after that 100,000. Don't think we're stopping. And if you're looking for more information about getting up and running, maybe following along with the Quinn 3.8 video that we

15:20 did in FP16, you can check out the video we've got here. And if you're interested in some hardware videos around how you can get up and running with your own local LLM, check out the videos that

15:31 we've got here.