QWEN 3.8 27B Local AI Review

0:00 I just produced the first outputs with my Quinn 3.827B. This is running in VLLM. I'm going to go through the script that I'm using for the run block, all that stuff. But I

0:10 wanted to give you a look at the first output. It did take 2 hours and 40 minutes. I had it generate an arcade box. We're going to see what this looks like here. It does say that it is up and

0:21 running. Let's uh just see what this looks like. So 192.168.1.215 215 8088. All right. And this is looking pretty

0:35 much like I was expecting. So I gave it a prompt that basically told it to create three retro games emulator style 8bit synth wave as a feature and with sound. And it does appear that the sound

0:48 is working also. So it should have all the things like leaderboard and everything else. And this is all in JavaScript. Nothing in Python. Let's start off with Void Raiders, which is uh

0:59 basically an invasion defense game. Hold the line against descending alien grids, destructible bunkers, mystery craft, waves that only get angrier. All right, let's see if this plays or not.

1:15 Oh, this is very hard. Actually, don't shoot your own defenses. Okay. One. What? Dang. Two dead. Uhoh. Oh. All right. So, uh, DSP and my score was 900. So, let's see if it saved that

1:51 to the global leaderboard also. Okay, cool. So, void raiders DSP. So, it kind of has a global leaderboard. Let's try Nebula Drift. Oh, this one's going to be hard, I bet.

2:11 Oh, this one's very hard. So, I just got to fly around. No. and avoid getting hit by the asteroids. Oh, this one's Okay, that one's

2:26 that one's almost impossible. Okay, the inner key being a part of the dynamic here. The inner is the fire key, so that makes it very hard. And space is thrust. So, hm, that one needed some

2:48 little bit probably refinement on it. This was all a zero shot. So, just this is just what I was able to come up with right off the bat basically. And this was all using Hermes agent as

3:05 well to orchestrate everything. And I've got to say the Oh, yeah. The quality of the code is substantially better than what I've seen before. And that this is just really

3:21 good quality though for a zero shot. Literally zero shot. I did have it do full review on it. I told it to do full review on it. One prompt and this was the output. That is impressive. And it

3:34 did it. I I know it doesn't look like it was very fast here. Let's take a look back. It is possibly still doing some compaction. And I there's a lot of things that I'm going to talk about,

3:43 especially with MTP, that you're gonna want to consider here. But I mean, holy cow, really good. Honestly, I am super impressed because playable right off the bat. Overall,

3:57 this is better than 3.6 could ever have produced, even hooked up to Hermes agent on a zero shot. Like very, very much different. Quinn 3.8 8. Huge difference versus 3.6. I saw zero tool calls fail.

4:13 Zero tool calls fail. Like if we go up here, no tool tool calls failing. Just no tool calls failing. By the way, I'm going to put this all up so you can access it, check it out, play it if you

4:26 want to uh run your own if you're interested in doing that. Full context window fits as well. Full context window fits. hit its compaction right at the end of this entire generation which

4:38 super glad that it didn't happen in the middle of it. And I think there's yet another game paddle breaker that I did not play yet. So it keeps best score. It has kind of a running tally of high

4:49 scores. Here it is. Budo number five. By the way, do you like budo number five? Budo number five is four 3090s and one 4090 over there connected at full 16 width. And this is on the WRX80 Thread

5:05 Ripper platform. Definitely definitely a speed improvement I'm noticing already in this. Let's uh give this one a play here. We have not played Starbreaker yet.

5:20 Arrows. Okay. That was kind of crazy. It went up around and stuff. I'm not sure if I'm supposed to be catching those or not. I like this dynamic where it bounces off

5:59 the top. also that that's actually quite helpful. M does toggle the sound, so that's good. P does look like it. Oh, okay. So, P does not pause. P resets. So, there's a

6:18 bug. That still is pretty damn good for a zero shot. I mean, this is just fun. This is just a lot of fun. Let's see. Once you mute it, also it does not want to appear to come back off of mute while

6:32 you're in it. So, I was in here and I muted it. Now, let's try going back and see if So, there's sound toggling it looks like on some buttons that is not quite

6:45 working out right. But, yeah, this is uh this is really good quality. 4 Quinn 3.8 27B. Let me mute that so I'm not talking over that. I don't know if I'm going to be able to mute it. Oh, okay. So, I'm

7:02 gonna have to mute the tab. So, there are some audio toggling issues that you could go back and have it refine here. Terminal bench, the improvements that we saw from Quinn 3.6, I have got to say,

7:15 are substantial. As a matter of fact, like no tool calling. The terminal operations just worked every single time. Now, this is a brand new fresh Hermes installation. So, sometimes

7:27 that's probably the best way to get a basis on a agentic install is you can bring in a lot of crust and a lot of broken stuff and a lot of stuff that's awesome. So, I did a completely fresh

7:37 Hermes installation. This was of course in our Proxmox instance over here and this is in DSP Hermes that I've got this up and running. Definitely, it feels like this is insanely better code

7:47 quality. The code that I was able to generate with 3.627B 627B was very much iterative back and forth. Fix things and this is through Hermes. Fix things, review things, things are still broken

7:59 that it should have caught. Inability to use things like playright really succinctly was another thing that 3.6 had. 3.8 nailed playright just insanely well. So

8:13 I am incredibly impressed, incredibly incredibly impressed with the improvement as well. It did tons of vision that I told it that was part of the instructions was use your vision

8:23 skills. So I basically started up and primed it by giving it the giganto text file and saying you're using Hermes agent and I said you're also running Quinn 3.827b fully review both Hermes

8:36 agents document and 3.827b 8 27B understand fully your capabilities and utilize your capabilities in the development of this game which is a very good way to start off an A gentic

8:46 session if you are using a new brain in it and it doesn't have that kind of requisite knowledge already. That was it. That was basically it. And I'll post the prompt and everything like that at

8:56 digitalspaceport.com as well as all the rest of this and you can find a link to that in the description below. But this is just a really great first look at this and I am really really impressed at

9:08 the output that we were able to get from it. I'm going to be playing with it more this weekend. I can guarantee there'll be more videos. We're almost 4 hours in and this is I can tell already it is the

9:20 improvement that we were really hoping for. I mean the tool calling especially just being almost flawless on it. The ability for the full context window to fit in. Excellent. Excellent. Excellent.

9:31 So, a couple things here. Let me quickly show you. We'll terminate out of our VLM instance here. And let me go ahead and cat our run block. So, if you want to, you can copy this run block. I would

9:44 recommend disabling speculative config. As a matter of fact, I'm going to do that right now for myself. So, instead of cat, we'll just go to nano. And I did not think that the speed up

9:59 that I was seeing was and the workload really wasn't like if you're doing code review and stuff like that, working on the same project over and over, you're going to really see MTP kind of start to

10:09 shine there. But looking through this, I did actually go back and change some things that in the prior 3.6, do not use your 3.6 six run block that I showed you how to uh basically get some really

10:20 optimum tuning if you have quad 390s or quad GPUs in general that this will fit on. Don't don't reuse that. I'll I'll show you the things that are different here, but uh there are a couple things

10:30 that are different. In particular, the enable thinking is now enabled and true and functioning. Prior in the 3.6, six, you really did have to disable this for really good agentic workflows to not

10:43 just have really bad loops, all sorts of problems that would happen in them. So, we've got our tensor parallel, of course, set to four. The max model length was set to auto. Hermes does a

10:53 great job of autodetecting that. Fully pulled it in at 262, 144, 256K. The GPU memory utilization, I bumped this from 0.90 to 0.93. It worked fine. And that maybe even could go up a little bit

11:08 higher like 0 n5. I did not see that there was contact runout or anything like that. And especially with I think I will actually push that especially with MTP disabled. That's a little more space

11:18 for the KV cache. So that should actually work out just fine. Port 9876. Our host is 0.0.0.0. And a couple other the rest of this looks the same. Mm process cache type

11:30 SHM. We've got our max num batch tokens at 8192. That will give you big fat batches. Enable prefix caching. Mamba cache mode align. And our API key. Our

11:42 CUDA visible devices are 1 2 0 and four. Good to explicitly set your GPU order. I've seen that in Quinn 3.6. So I made sure to copy that over here. The the order could probably be changed around,

11:54 but setting an order is what I would recommend. Our opnum threads we set to four. That's a very good recommendation for you. So that means you would probably be wanting to run your VLM

12:04 instance on like an eight core kind of count in a virtualization instance or a a 8 core minimum CPU. And the ML MKL num threads is set to four. Also, our CUMIM enabled is disabled. And we also have

12:21 our expandable segments true. Like if you're using multiGPUs, you definitely want that. And also, you won't actually be able to set your CUDA visible device order until you actually set this as the

12:31 PCIe bus ID up here. So, if you want to copy a run block that really works good, here's your shortcut also to running this. It does occupy pretty much almost all of your memory. So, let me save that

12:44 out. Reset that. And let's hit that run block again. And we'll jump back into Hermes really quick. and I'll ask it if it can save a uh skill based off of what it just did, which I think it did as

12:58 such an excellent job that I definitely would want this skill saved to my own internal repository of skills. Again, when you're setting things up like this, I know I've mentioned it several times,

13:08 use Giddy internally for your own Git repository. It is an excellent way. Give your agent its own credential set so that it can make its own commits to it. And that will give you a much better

13:19 insight and a much better roll back point if there is problems on things that it is working on. You can even have it create things for its own skills. You can have it updates its own skills in

13:29 there as well. Yeah. And you can see the speed of loading also dramatically faster on the Thread Ripper. 4 GHz versus 3.325 max single stream. And it went down from

13:41 there versus almost all the threads are running at four. I think all the threads are running at four on the 3945WX. That video that I did when I did that, I did pick up one of the 3945WXs.

13:52 I already had a WRX80 motherboard though, so I just reused that. And I sold my 5955WX, which not that big of a difference in performance that I can see between the

14:02 3945WX and the 5955WX in what I'm doing right here. Also keep in mind this actual desktop, this recording that's being done right now is all happening in my Cache OS VM instance

14:14 that is right here and our Hermes instance also is living on this. And then we'll get a better idea of what the actual tokens per second looks like as it's creating the skill. Of course,

14:24 we've got our MV top over here. Budo number five. Budo number five is alive. Do you get the reference? Maybe. Got to be old to get that reference. Do like the new rig though. Insanely heavy.

14:39 Insanely heavy with all the GPUs in it and dual power supplies, a 1500 and a 1000. That entire build video is going to be a really long build video. I give so much meta in that and that is coming

14:51 out probably sometime very very soon. We've been working on that one. But I did film it while we were putting it together out here sweating to the oldies in the garage because have not gotten no

15:02 mini split system yet. But the good news is I did have to turn it off for recording this. The little mini AC since we're able to confine all this in this area has been okay enough to keep things

15:14 not super hot. But definitely definitely need a mini split. Definitely going to go with a mini split. And thanks for everybody that dropped the recommendations. I did read a lot of

15:24 those. And we are started up here now. So let's get back over to the terminal. And you can see we are parked at right about 94% of memory utilization. 23 gigabytes or so on each one of the GPUs

15:41 over here. And uh I'm just going to tell it yes please create that skill. And I'll move that skill over as soon as we connect it back up over here to its own giddy credentials. I haven't taken a

15:58 time yet to give this brand new Hermes instance giddy credentials and stuff yet. So yeah, you can see it now 100% on this. Definitely have one of the GPUs here. Since I'm using Envy Link on two

16:10 of these, that pushes two of them together closer. That's always going to make one of them hotter. Usually the one that is kind of uh on the bottom of the connection. So, fan number one, this one

16:21 here is going to be hotter on that NVLink because there's just not much space for the air to get in there. And it's basically pulling heat off of the back plate of the other one. So, whether

16:31 or not I keep the NVLink on it or not is a big question. May end up selling the Envy Link. I don't know. I'm going to test the Envy Link out, play around with it a little bit, and we'll see. I guess

16:41 I'll just say that. We'll see. Oh, you can hear it spinning up also over there now. So, prompt throughput at 7,214 tokens a second. Jin at 15 currently, and that should get a little bit more

16:54 refinement as it starts running through. So, now we're seeing the gen come in at about 41.2. So, that's pretty much where you're going to expect to be on a Quad 3090 rig.

17:06 But seriously, super hats off to the Alibaba Quinn team that produced this. Excellent. This is this is this is amazing. This is amazing work. Definitely there is a FPA and an

17:21 NVFP4. Those are essentially quants also. But if you are maybe running Blackwell GPUs that NVFP4 if you got two 5060 Ti 16 GB GPUs 32 GB probably can fit that in. If you've got two 4090s or

17:37 something like that, you probably can run that FP8 and get uh pretty similar performance. I'm not sure what the quality loss is going to look like amongst those. So, at any rate, check

17:47 that out. digitalspaceport.com. Link to the prompt, link to also probably the playable if you want to play this also will be in the description below. And thank you

17:56 everybody that signs up for the members on YouTube. Uh it really has helped out the channel quite a bit. And if you're interested in learning more about how you can get up and running and building

18:07 rigs that are doing things like this, well, first off, the updated build video for this is coming. And man, this is a great build. This is just this is a great build. But you can find out more

18:19 in the playlist here. Have a great rest of your day.