DeepSeek V4 Flash Vision Is INSANE – Tested With DeepSeek Harness!

0:00 Max City. Oh, what? All right, this is actually kind of sick. So, Deepseek has released something that a lot of folks were very, very excited

0:12 for, and that is adding vision capability to one of their models. In this case, it comes to V4 flash. Now, this is still experimental as denoted just by the EXP tacked on the back.

0:23 However, this is something that really will benefit this model. We tested this a few weeks ago when it came out. Honestly, I went to look for the video of when I tested it when it came out. I

0:32 thought it was like a few days ago. It was a couple weeks ago, so things have been moving very quickly. However, giving this multimodal capability seems to have one given it the ability to see.

0:41 But two, apparently the scores have increased over a number of tasks, specifically ones that we like to run on this channel, such as like coding related tasks and things like that. So,

0:50 for today's video, and because we did just test this fairly recently, I may do a set of somewhat different tests that are designed solely around testing the vision capabilities of this model and

1:01 seeing if it benefits from the ability to actually look at something it does, check its work without having to rely on just making intelligent tools to compensate for not having vision. So,

1:11 before we get into it, please do feel free to subscribe and let's get into it. So from the benchmark chart we see right here and they even mentioned that this is bringing the capability closer to

1:20 Opus 4.8 deep sEore and I'm more interested in just seeing the boosts in capabilities over its previous version which was oh 0731. Okay, my timeline's a little off. So

1:31 this is actually a pretty big jump in capability at least in a few of these. Now I always tend to find deep S.E scores reflect my personal experience with models pretty well. So seeing a

1:41 jump in capability there of around five points and then nudging this over Opus 4.8 is pretty darn exciting. Now speaking of exciting, something else that has been getting a lot of attention

1:51 lately is the Deep Seek harness. So for today's video, we're also going to be kind of testing that as we will be using this model exclusively through the new Deep Seek harness, which I do actually

2:01 have open somewhere. Yeah, good. So right here. And I've already run a couple of tests through this. I've not looked at any of these. So, these were just things that I knew would take like

2:09 about an hour or more each, and I wanted to have them ready for when I did begin to test this. However, we're going to start with something probably a bit closer to what we're used to, but with a

2:18 twist. Now, one thing I do want to quickly bring up is I don't see any mention here of this ultimately becoming open weights. Now, I have no reason to believe that it won't being that all of

2:26 the previous models are. However, there's no mention of it here, and it is not currently on Hugging Face at the time of this recording. So hopefully that will change soon because this could

2:35 be a pretty darn cool option to have on some more like enthusiast grade local AI setup. So for this model in specific, assuming the most expensive scenarios, a million tokens in is 44 and then per

2:47 million out is $1.32. These prices, I don't know how competitive these are just given how much more competition has been appearing lately in the landscape. Additionally to that, this is not

2:56 necessarily a very large model, so it'll be interesting to see how it performs. And on that for cost, I have run two tasks as we may have saw in the deep sea harness when it was open here. I did the

3:07 3D wrestling game and then the full-on cinematic Steve the PC repair game prompt. The total cost for both of those running for over an hour each was $12. So keep in mind as we began this test

3:19 right here, we've used a dollar two so far for everything we're going to see in this video. And then I will just showcase how much this goes up after each additional task that has not yet

3:28 been run. So, we're going to begin, of course, with the browser OS test v2.7 where it needs to do all of the traditional things and additionally to that include procedurally generated

3:37 wallpaper that has movement. Actually, for this specific one, I am going to delete that because this is more of a test of its multimodal capability and I am just adding something else in the

3:46 bottom. The result must completely match the reference image in terms of style. All apps should be designed to look as if they were natively included with the reference image, which is just a

3:55 screenshot from the Mac OS 9 desktop. So I basically want our traditional browser OS. However, the style must completely 100% mirror what is seen in this reference photo. So in lie of just doing

4:06 a browser OS which doesn't really test its vision capabilities to a high degree, we'll be able to see how well it can accurately mirror a reference design photo. All right, so in 26 minutes and

4:15 47 seconds, we've received our results. Uh something that's just totally distracting me here is the special feature is Moose the desktop resident. A small autonomous moose that lives on top

4:26 of your windows. He walks, sits, blinks, mutters random observations, comments on your actions, and reacts to clicks. Okay, let's see how much this cost, and then we'll take a look at the result.

4:38 And moose. So, we were at $12 used up prior to running this task, and it's gone up to $1.26. So, 24 cents in total for our browser OS equipped with Moose. Um,

4:52 let's just take a peek at it. Okay, that's okay. Our desk does cop pet just threatened us. Okay. Is there a right click? No, but I

5:21 No, OS9 had one. This did actually I think this did a pretty respectable job of capturing the essence of the reference photo here. The amount of things contained in this. Okay, so most

5:34 of these we can't actually click on which I guess makes a bit of sense. Let's close that. Oh, it said type moose. Okay. Oh, it responded to typing moose there.

5:45 Okay. Can we like maybe there's a settings app where we can banish this thing from existence. Nonetheless, Mac OS good.

5:56 Single file browser OS. Memory sizes are decorative. The moose is real. Yes, sadly. All right. Full screen looks good. Minimize to Oh, nowhere. Okay. And the actual ways that the hover

6:10 effects appear over these menu items are more or less pretty decent. All right. File about this computer. Love it. Do we have about moose? Yes, we do. Special empty trash impresses him. He is easily

6:23 impressed. This is really clever. Oh, okay. I was trying to drag him to the trash, but chooser. No printers were found on the Ethernet port. There is no Ethernet

6:33 port. Keycaps. Keycaps board slit under your desk. Under the real desk. This is really, really, really put in a lot of Easter eggs and clever things which were not denoted in the prompt. So,

6:46 apparently this moose thing likes it if we empty the trash. Yeah, it liked it. Okay, let's just try the calculator. 54 * 6 324 / 9. All right, let's take a look at some of the icon things here.

6:58 Okay, desktop. Very cool. Documents. And these are our apps. All right, I'm going to open probably just the Grand Theft Auto first cuz why not? Mac City. Oh, what? All right, this is actually

7:14 kind of sick. These buildings are leaning toweresque, but they do. This is really good. Oh, look. Oh, we're getting a look.

7:47 This was something I probably should have full screen, but that's okay. Wow. I did not expect this. My pad. Oh, we saw that. We can type Moose. And this might be the

7:59 best browser OS result ever. Power mail. Moose. I live here now. Starhog 3D. Yep. Okay. And I think it's doing it like in the style of what you would expect.

8:15 All right, network. This was really just local talk network. That's a thing. Power mail gateway GTA server. This was incredibly incredibly well done. Very, very impressive. And we have

8:28 things up here. We have the time and then the trash finally and the moose. Moose mode activated. I do believe this warrants one more look and we can take a car. The way it

8:42 actually tried wrapping the textures around these cars would have been pretty darn cool had it accurately like this. Oh, there are mesh colliders on them. Okay,

8:55 this is getting insane in the title just from this response. All right, so before I just like start really getting excited about this, I did run the 80s wrestling game. It basically needs to make a 3D

9:05 wrestling game that looks like something from the mid80s in terms of style. So, it named it Slam City86. And apparently it's running on local host at 8080. I've not looked at this yet. I can tell you

9:15 right here, this took 37 minutes and 54 seconds. So really not horribly bad in terms of like compared to the Steve the PC repair game, which we'll probably look at as the last thing. So let's just

9:26 take a look. Okay, this is pretty actually decent. Macho Mike, Baron Blaze, Johnny Jet, Iron Ivan. Which has the most like flamboyant

9:45 outfit? I think we'll go with Macho Mike. All right. Oh, this is clos line. He done hit the post. This is if this thing

10:04 gets open weighted and can run on two sparks. I think that I would be quite concerned if I was serving like lowcost. How many people have two sparks? All right, never mind. But the point I was

10:15 trying to get across is that this is really like I've only run this test with a few models, significantly larger ones, and I can tell you this is pretty much beating out the GLM53 result, which is

10:26 surprising. Oh. That's That's That's BS. Stop hitting him. Okay, that angle was

10:41 That was a bad That was a bad cut. All right, I'm done with this. No, I'm not yet. I need to get this yellow guy with the the yellow shorts. Get up. Oh, this is

10:55 Get up. All right. All right. Well, that was really good. Very, very good. All right, so next up is going to be something that I would imagine is going to be pretty difficult.

11:07 I've essentially given it a few reference photos of the space that I'm currently in right now. And I'm telling it that using these reference photos, I want you to replicate this scene in an

11:17 isometric view using Blender. Ensure the textures match that of the photos and the overall scene is wellmade. Now, to give a bit of an example of what specifically that entails, this is an

11:27 isometric scene. So, just keep in mind that's what we're expecting to see just based off of the reference photos of the actual room that we're currently in. This will, I would imagine, take quite a

11:37 bit of time. The files are Mac OS metadata. I will ignore them. Okay. And we can see right here it's now using its multimodal capability. And as I said, I would imagine this will take a little

11:46 bit of time. The Blender session is set up with the MCP, so it should be good to use that as well. All right. In 74 or so minutes, we've received our result. I have not yet looked at it yet. I don't

11:57 actually know how to. Okay, good. There's a render so we can take a look at it. There's also the blender scene. In the meantime, let's just see how much that specific task costs. So, we were at

12:06 $126 so far total. And now we're at $1.97. So, 71 total for that task. Just looking at some of the actual pieces that it picked out here. Black 3D panel, acoustic wall,

12:19 that is correct. Worm oak wood slat panel with black gaps. That is correct. White radiator floor. Procedurally generated oak. Then dark walnut desk along the panel wall. Curved monitor.

12:29 Okay, that's not true, but that's okay. With glossy abstract art. These are actually some of the few non-curved ultrawides I could get. I hate curved monitors. They make me infuriated. VR

12:39 headset, blue handheld papers, white keyboard corner, gaming PC, racing rig, lime green office chair, tall pine retrofit shelf with beige max style desktop, red drawer rack with helmet

12:50 floating shelf rack. Okay. I mean, that just shows a decent amount that it pulled out from these reference photos. Now, in the meantime, let's just take a look.

13:04 That's like I like this. I don't actually have any reference point whatsoever with which to judge this, so make of that what you will. But I would say like the way it

13:16 actually got the poster on the wall behind me, the Apple history one. And then also the wall textures. There is a bit of white behind the painting. It accurately got that. The Toshiba Quasmio

13:27 laptop that's right there. The screen. It even rendered the old beige PC. The the VR headset. The tower. This is like I think this is a cool scene. I don't know how. Oh,

13:41 there. That's the driving sim. That's a little odd. All right, let me figure out how to use Blender. And then but like look at these textures and things like this as well. This is just like this is

13:53 so cool. How do we look at this? Oh, okay. Blender is installed and open. But I uh Yep. Okay. So, look at this. Here's the scene in Blender that it made. If I can figure out how to move this without

14:06 throwing this computer out. I'm not a I've never really been able to to get Blender and that's that's on me. So, here's the actual scene. I would say this is I think pretty good. Again, I

14:19 don't really have much frame of reference for this, but I mean like that's pretty awesome. I think like even like the seats are actually made of individual parts. Look

14:29 at I think this is really good. I just Yeah, this this is 100% getting insane in the title because that's awesome. All right, so our next test is just the typical Subway FPS

14:46 test. However, we have included one that reference photo that we generated using Nano Banana. I did opt to go for the grimy looking first one that we got. Additionally to that, I have just

14:56 enhanced the prompt a bit just to make sure that it's a little more full for a more difficult test being that it also needs to use Blender to create the assets, which we have told it to do, and

15:06 then GDAU to create the game. I would imagine were I to make an educated guess, this will take 40 minutes and 27 seconds. So, we'll see if I was right. All right, so in 40

15:18 minutes and 27 seconds, just as I had guessed, this game is finished. That was actually pretty quick, I think, all things considered. So, let's take a look at what our usage went up to. It's at

15:28 $2.32 right now. I don't entirely recall what it was at before. I think it was like a $1.90 something, so maybe 40 or so. In the meantime, I am very, very, very eager to see what this has done.

15:40 So, let's just run the command here to check the game out. I'm going to full screen. All right. Oh, no. Okay. I think we may be outside of the station. Oh no. All right. So,

15:57 something has gone deep deeply wrong here. However, it's uh it's quite troubled. Let's just pretend like it's not for a

16:12 second. Watch the reload. It even has like a um you know the other thing that's required to reload. So yeah, this is uh something's gone quite

16:29 wrong here. All right, so I'm giving it a follow-up just saying basically something is deeply wrong with this result. I've also given it a few reference photos just of what we were

16:37 experiencing where we spawn outside of the station. Things are like floating in midair like that and then there's just a bunch of odd issues. All right, so a little over an hour later, we now have I

16:47 believe a fixed result. Let's just take a look at the pricing. So, we were at $2.32 prior to getting our fixed result and now we're at an additional 31 on top of that as it has swapped into the next

16:59 day. So, let's just take a peek at this and see how it is. Okay, definitely not as sideways as it was before. I will say though, overall, I'm not as impressed with this as I was with

17:13 the previous ones, just based off of judging it on how the reference photo looked compared to how the real implementation looked. However, it did make all of these models. This used

17:23 Blender and GDO to do this. This isn't a single file 3JS game, so it is a more complicated implementation. And we gave it a reference photo. So, sound is good. The textures on the

17:37 zombies are Oh, okay. Kind of interesting. We do have tracks and things like that. So, spaces to jump. Oh. Oh, those are the ones we just had got. Okay. So, there is a subway car

17:48 there. There's some effects. There are like flickering lights to make the scene creepy. I don't know where the additional enemies are. Next up, I'm going to give it the

18:04 self-contained C++ skateboarding game with the 2000's New York City block as the map. This is a test that really will show us a difference, I would imagine, between having vision or not. Something

18:15 that often times really would have benefited from these models having competent vision, and this model in specific, I don't recall off the top of my head how it's done on this specific

18:24 task before. However, the fact is it does have overall improvements across the board in some coding related things. And then the ability to actually be able to look at what it's doing may assist it

18:34 in giving us something better than what we've previously seen. Apparently, this um glitched out at 22 1/2 minutes, which would have been good for me to have realized about an hour and a half ago.

18:46 However, I would imagine that there's probably still some form of like saved progress things. So, we were at 31 cents there and we're now at $1.16. So, I do have some hope that perhaps maybe it did

18:58 finish up in the background and maybe just the gooey kind of stopped responding. All right, so thankfully that is what happened. Um, the gooey froze up, but fortunately in 45 minutes

19:08 we did receive RC++ skate game result. So, let's take a peek at it. That is it's much better. Shaded City. Is there any audio? I can't imagine there is, but this is quite respectable, especially

19:27 considering this is a flash model. I would actually, truthfully, now there are some, of course, issues with the pedestrians. The buildings actually look pretty darn good for a C++ result. Now,

19:39 it's been around for a while, so make of that what you will. However, this is actually jump onto rail and ledges to grind. Space pops off. Well, yeah. I need to pop on before I can pop off.

19:52 All right. Are we going to get stuck at the end? Oh, no. We didn't. Okay. Let's try just nice. The board moves independent. The board moves independently from the

20:04 person. This is honestly pretty respectable. I'm going to say and forget about even comparing this to other models. I know that the previous version of this model, Sans Vision, would not

20:14 have done as nice of a job as this. not like by a long shot. So, this is one of the more competent ones to this date. I believe Quen 3.8 Max popped out the best one of these by a large large

20:26 amount. However, this in and of itself for the size of the model is really really quite competent. Even just putting larger skyscrapers in the background like that, it's not something

20:35 we see very often. There's Let's see. Can we interact with these shaky pedestrians? No, that's all right. Are we grinding on a curb? Yeah, we are.

20:50 I don't. All right. This This is definitely just justifying the insane in all caps in the title. Very good. This was the previous skate game result and this was the one that had the California

21:00 Boardwalk aesthetic. So, this is what the Flash model from 0731 did. So, basically 3 weeks ago, this is what the Flash model received for the same general prompt. However, not the New

21:12 York City boardwalk aesthetic. So we can see here this is actually arguably pretty strong as well. However, there were some issues just basically like that with the text and things of the

21:22 sort. So good and bad in both. However, the New York City one I think was definitely more detailed. Now I had also run the very long and drawn out cinematic Steve the PC repair game as a

21:33 test. We've not yet looked at that. And being that I'm just like I want to see this. I've been waiting long enough. So, all right. This should have sound and everything and this should be a pretty

21:44 cinematic game. All right. So, sound is working on Oh, good. That's very good. Maybe we'll just The sound is definitely not quite working. It's possible the

22:01 speech transcription didn't work, but we can kind of see. Okay, so it's still that Oh, look at the cat tail sticking out of the box. actually don't like cats. I'm allergic to them, which is

22:12 just horrible. But don't they like going in boxes and things like that? So maybe that's actually a purposeful choice. The scene overall looks good. Miss Ellis is actually blinking and there's some

22:22 slight back and forth or side to side lean movement. The big issue is like it's just really slow and the sound there's no voice. So we'll just see what we get. Look at the raindrops outside.

22:33 Should I voice act this? The SMOS battery. It's like a pacemaker for the computer's memory. Dying at this age is odd. I don't know that I'd use that lingo around Miss Ellis, but it is

22:43 1998 tech. It has been through a lot. 60cond job. Then your receipts are safe and that email from Danny will open again. Danny is a norw walk. I'm not I'm not I'm not voice acting

22:55 this. All right. Yeah, whatever. Come on. There's There's no like I can't actually All right. Here's what's up. I essentially told it to just turn this

23:07 instead into a movie that we can watch. And there is also going to be a pace to it as well, so we can fast forward and make it go faster. We can see here it was definitely pretty slow. So, I'm just

23:18 almost more interested in getting a feel for how this would have been. So, okay. So, we're at three times speed here now, and it's still like pretty darn slow. So, we'll just see.

23:30 Keep in mind this is a movie and the issue with the sound is if I turn the speaker on. Yeah. So, let's just take a look at the visuals.

23:48 Okay. So, now it's just autonomously in a movie style fixing this. It's opening the case. Some interesting things to note are that Miss Ellis, who's in the shop with us, actually has like some

23:58 static but like side to side leaning. So, there's some movement. That clock is a bit odd. However, it's just interesting to actually There's blinking as well, which is

24:10 pretty cool. I've seen that in like Fable, you know. So, I saw a very faint thing on the screen. So, it actually showed the screen reflecting the repair

24:24 worked. Interesting. So, we're supposed to be seeing another person right here in the shop. Uh, it seems like the camera didn't catch that or something of the sort.

24:41 Oh, there was a small small like snippet of the airplane. Interesting. Somewhere over Greenland. Look outside the window. You can't see what I'm pointing to. Right here, you can actually see clouds

24:54 moving by in the window. That's kind of cool. I'm glad that we just opted to do this movie method instead because um it was, you know, there's some Z fighting there just in the windows. This is

25:05 inside the airplane. So, there's I do believe one of the flight attendants. The Alps went by under the wing, then the lake, then everything got small and clean. That's the thing about

25:14 Switzerland. Even this. Okay. Oh, I think that was the landing that it started getting shaky for and at three times speed. Okay, so now we're at the building and it's raining.

25:30 This is kind of cool, I think. Hopefully now we're at the dock, like the loading dock, service elevator. That was cool. There's a security guard walking around

25:53 the IT department. Or maybe that's an IT worker. I don't know. These are definitely security guards, I think. Then there's like one of these micro

26:07 games that would infuriate me. Had we had to play it. So, I'm happy we did this. What was Okay, so this is the quantum computer and I'm pretty sure he just has

26:23 to shut it down or something. We don't really have much in the way of sound. So So we had a choice basically to either shut down this supercomput or let it live.

26:45 I would go with number three, but this is in movie mode, so maybe good. Oh, good. We see the cat, which is perhaps something we could have done

27:00 without because it looks like slices of bread loosely put together. All right. Well, that was actually pretty interesting. I'm glad that we got to kind of watch

27:14 it, even if it was without sound, just to see through what it would have implemented for the game. And I'd say overall that was actually pretty darn cool and the price for like the fix of

27:24 that because that's something I had run prior to beginning to film this video. So, it was part of the introductory like $12 for the 3D wrestling game and the Steve the PC repair game. So, we were at

27:35 $116 before I told it to turn that into a movie instead. So, it had a lot of assets there. It went up to $1.64. So, 48 cents more to turn that into a movie, which I still think was pretty darn

27:47 cool. So, for the last task, I want to experiment with this on a different thinking level. So, I've swapped it down to low. Every prior test has been done on the maximum thinking effort, but this

27:57 is a pretty simple task. It needs to just replicate this GPU rental website from this reference photo, which was just AI generated. So I've told it simply create a pixel perfect

28:06 replication of this website and on this lowinking mode I'll be interested in seeing how long it takes to do this and then how true to life of a replication we get for this source. So this was

28:16 significantly faster than any of the other results that we've seen so far. It took 2 minutes on the dot. Now again it was a simpler task but on low thinking mode. So I want to just take a peek at

28:26 it. Okay did actually I think a pretty decent job. Let's look at the reference photo. Okay, so Velocity Rentals, we have this all on the left here and then we have some things on the right. Okay,

28:39 it did decent enough uh in terms of a job. However, okay, key features. Let's see how well it replicated these because these would be easier to replicate

28:51 key features. Y velocity. Okay, better use cases. It's simple and it kind of overall just got the scope and things of this, but without the actual source assets, it's

29:04 definitely going to be a bit more difficult. I would say I think the SVGs probably could have benefited from actually having some gold in them or just to look a little more like some of

29:12 these reference sources as well as the header photo up here. However, this again was just on low reasoning mode. And I basically wanted to do one more thing just to test the vision

29:22 capabilities again because that is the entire point of today's video seeing how this does with vision capability. So I don't actually know the specific figure for what that last test cost but it will

29:32 become more apparent in watching the video. So the entirety for all of the tests we did today. So that would bring us to $3.96 basically for every single thing that

29:42 was seen in this test which is significantly more expensive than these models were to use like a month ago. But with the increased price, it's still not bad at all. I mean, some of these were

29:51 working for north of like a 100 minutes. And then we did follow-up things as well to the PC repair game and then the subway game test. And really overall, I don't know that I'm going to do like a

30:01 traditional results overview because I filmed this over a few days. And I just think like the culmination of this is it seems extremely impressive. And part of this was also just being able to test

30:11 the Deep Sea harness with this model. It seems quite good. I don't have much to say in terms of a reference point because this is the first model I've tested and the first time I've tested

30:21 this harness in, you know, so I think overall I'm very impressed with this. It's 100% getting insane in all caps in the title. And the final thing I'd like to touch upon that I will just check

30:31 again because it's been like a day or two since I started filming this. So the weights still are not on hugging face. I am hopeful that they will appear here. I would imagine they will, but it's uh

30:42 just something to keep in mind that this is not currently open weight. So, if it does become so, it will be a pretty darn good option for dual DGX sparks as I had said. So, this was overall a very, very,

30:53 very impressive model. Having vision capabilities definitely makes it perform better, just as we saw with the Blender only test where it needed to make the isometric scene of the room we're in.

31:03 That was absolutely fantastic. And overall, it was just a lot of fun to play with. It had a lot of bits of cleverness displayed, whether it be the Mac OS 9

31:11 browser OS, which had the moose, which was funny, and some other Easter eggs in there, or in the Steve the PC repair man movie. Some of the lines were clever in the story line it came up with itself

31:22 and things of that sort. So, that is going to conclude our first look and test of Deepseek V4 Flash with Vision Experimental along with the new Deep Seek harness. If you have any questions,

31:32 please feel free to leave them in the comments.