Nemotron 3.5 Lightning First Test – NVIDIA’s NEWEST Open Model!

0:00 In 8.5 seconds total, we have a SCAD model. That's okay. Now, that maybe makes a bit more sense as to why that was so quick. Today, we're going to be taking a look at the second openw weight

0:11 30 billion parameter model of this week, which is a very exciting sentence to be able to speak. This is, of course, that did rhyme, by the way. This is Nvidia's Neotron 3.5 Lightning, a 30 billion

0:22 parameter mixture of experts model with 3 billion active. Now, I can't definitively state this, but I do believe just based off of looking at this benchmark JPEG right here. This is

0:33 basically a leap in capability and like the next generation from Neotron 3 Nano, which had the same overall footprint and parameter count. And we can see here that Neotron 3.5 Lightning shown in the

0:46 lighter Nvidia green seems to have some significant leaps in capabilities, at least in terms of coding and long context recall. So, it's pretty exciting to see another open model of this size

0:57 because it's actually reasonable to run on more like hobbyist or enthusiast systems. Basically, this is very heavily marketed towards running on something like this, which is of course the DGX

1:08 Spark or something like this, which is of course the second DGX Spark. Nvidia sent me this because influencer status, so I did want to just kind of brag to be completely transparent about that. Um, I

1:19 had this one which I went and bought on launch day from MicroEnter, but now I have another one and I am excited to connect them. Let's take a look at the Neotron 3.5 Lightning model from Nvidia.

1:29 Also, feel free to subscribe so I can get the 70K and then the 100K plaque later on. So, let's take a look at Neotron 3.5 Lightning. This is a 30 billion parameter overall mixture of

1:39 experts model with 3 billion active. It has the hybrid Mamba 2 architecture with a 1 million token context length. And this is something I quickly do want to mention where it does seem to do much

1:51 better at longer context recall than its predecessor, but I found that I've seen over time folks mentioning like the predecessor for this as well as doing pretty well at handling longer contacts

2:02 with specific like needle in the hay stack tests or recall tests over longer content. So, it's interesting to see and I do believe just based off of some things I've read online that these may

2:14 actually be pretty performant at some of those tasks. Now, additionally, we can see right here that they do also have a speculative decoding model. And especially when running this on a

2:23 unified memory system like the Spark, which isn't necessarily the fastest thing around, that can really make it just a better quality of life to use a model because it really will

2:32 significantly speed things up. and some other pertinent information about the model. The pre-training data has a cutoff of September 2025. The post-training data has a cutoff of May

2:41 2026. So, this should have some more recent up-to-date knowledge, which is good just if we do some general knowledge testing and things of the sort. Now, additionally, something I

2:50 found pretty interesting right here that I would definitely like to see more of is this agentic coding benchmark where this model was tested in different harnesses. So, this is something that is

3:00 awesome to see as harness choice is almost like I view it as like how people feel about their sports team that they're a fan of. So, there's a lot of ones and it seems like this performed

3:11 the best in open code, which is how we're going to be testing this today. So, I just really wanted to specifically point this out cuz I would love to see this for more model releases as it just

3:21 answers a lot of questions because harnesses are pretty like polarizing. So, this is definitely designed to run well on the DGX Spark. So, we're going to be using it on an RTX Pro 6000 for

3:32 today's video. So, um I apologize in advance for that. And there's just like a bunch of other stuff here. It does have toggable reasoning on and off. And I want to just mention as we get started

3:44 basically and this is something where like the Neotron models historically, and it's even shown in the benchmark there that they've included as well, they're not coding beasts. So, even the

3:54 Quen 3.6 six mixture of experts model will probably smoke this in coding. But I want to say that in these short 20 30 sometimes 40minute testing videos, we can't necessarily capture all the weird

4:09 niche use cases that a model like this may excel at where another one might not. I've been reading online just at folks like initial experience with these things and someone was talking about

4:19 like using it for some translation task from a lesserk known language to like English or something and how this or its predecessor performed really well in that one task. So there's also stuff

4:29 like that that don't count them out just because they're not necessarily great at coding. So with that, let's now get started in a coding task. We're going to do the browser OS test v2.5.

4:41 And we can see right here, the speed is very good. Interesting. It actually autonomously opened it by itself, I would imagine, to do some troubleshooting. That's not something

4:50 that I've seen before. Why is all of the Okay, all the models are naming these like the same thing. I nuked the entirety of the Open Code cache config, everything, the logs, every chat

5:00 history, blah blah blah. I want to know what's up with this. Is it just because it's the name of the system? All right, I'm asking Fable 5 why every recent model on this system has been calling

5:12 the browser OS's like some form of ominous or something. Now, if I open a fresh terminal, it's the name of the system. So, it could be that they're pulling it from that, but someone had

5:22 mentioned this in the comment section for my previous video and it's something that's been on my mind as well cuz it's been happening recently. It's not a cache. It's the environment context of

5:31 every one of these harnesses injects plus a naming attractor. Every CLI harness puts the working directory username and often host name into the model system prompt. Interesting. Okay.

5:41 And then the thing it's referring to right there is the actual result that Neotron just made for us. So, okay. I just wanted to clear that up with a model that is intelligent and god mode

5:53 console. Okay. So, on first glance, as we probably saw, it's troubled because basically nothing shows up. There is something in the top right here. And of course, we have to try the god mode

6:04 console, which is what it named it and not me. And there's no right click. There's some bar down here. And then there's a Let's just Okay, so control shift C.

6:18 [laughter] Okay, maybe that is a different command on Linux. Okay. And again, the speed is because of the card it's running on.

6:35 Now, the reason I'm hung up on getting this special feature working is because at this point, it's our only hope of actually seeing anything in this browser OS. Now, we can see this is a pretty

6:43 hefty in terms of the number of lines of the script. So, if I press the space bar, okay, now we can deduce that something's not quite right. As time went on, I began to get increasingly

6:54 enraged. The model seemed unwilling or incapable of fixing this browser OS. In a fit of rage, I opted to start fresh with a new open code session and browser OS, which resulted in a result that had

7:06 app icons but no functionality. I began to wonder if there was a configuration issue on my end. I then decided to test the model on Open Router with the same task to see if a more functional result

7:17 was generated. It was not. My configuration was okay. I don't know how to put into words the experience I just had. It's now somewhat working. We have a

7:33 space shooter. This kind of reminds me of like a Super Maze Wars game. That was something for Mac. I think it was OS9 and in multiplayer if you were in the same room. It was actually incredibly

7:45 fun. Sorry for that bit of like a nostalgia. Okay. GTA clone. Yes. E to enter vehicle. All right. We're already in the vehicle. This is

8:02 terminal. Okay. Can I All right. So, I have become concerned that there may be an issue with my configuration because to be completely transparent, I don't think this would

8:13 have done that poorly at the browser OS test. So I've run it through open router where in the free endpoint right here Nvidia is serving it themselves which I would imagine is the complete proper

8:26 configuration. Okay. And this is a bit better. Well, wallpaper changed actually. I think

8:41 there's enough like I'm not inspired that there is a problem with mine now. We're just going to continue testing this for better or for worse. So, I'm giving it this self-contained

8:52 C++ skateboarding game. I am totally fine if this wants to use Ray Lib for this, but honestly, at this point, I'm not expecting a functioning result. I basically just want to see like how

9:03 rapidly this is going to troubleshoot any potential compilation errors and just like totally like snap this out very quickly. That was again like it's so so fast. Yes, you can do as you

9:15 please. Hey, you know what? It just did a ton of troubleshooting just at like lightning speed. That's not intended to be a pun. It's

9:31 just the term that came to mind. And it did end up opening something. It showed up very quickly, but the fact remains that this did go through a large large amount of errors to get something to

9:42 actually open. Now, I would be lying if I said I know exactly what it's doing right now. Is it still trying to run it? Okay, so it opens and closes very quickly, though. I did actually get some

9:57 form of movement real quick before it Okay. All right. And we're just kind of getting the same thing here. I'm not going to push this further because it's

10:09 not gonna properly knock this out. But it did get from a point where it wouldn't open at all. There were compilation errors to a point where it actually does now open. Now, I do notice

10:19 that it's mentioning RayLib here. So, keep that in mind. Maybe that simplified things a bit for it, but I'm totally fine with that. A model of this size in active parameters can use Ray. All

10:30 right. Next up, we're going to be trying a detailed 3D printable model of an inline 6 engine that needs to fit inside of No, that's not it's impossible.

10:43 It did this entire task in six seconds. There's nothing in the directory. [snorts and laughter] Was that just like a thought process? It just gave me the script in line in six

10:58 seconds, but it didn't save it. Okay. Okay, good. That only took 2.5 seconds. So, in 8.5 seconds total, we have a SCAD model. That's okay. Now, that maybe makes a bit more sense as to why that

11:14 was so quick. Just the speed here was like, all right, forget that. The that specific prompt is probably a little too a little too intricate. So, I'm giving it a much much simpler prompt here just

11:28 to see what we get. We'll see if it can beat the 8.5 seconds previously that it took to actually create a model. All right. I think it's already gone past that, which is good. We don't want it to

11:38 be that fast. Well, we do, but not at that level of quality. All right. And it has the STL as well. It likes its cylinders. Okay. No, there's a there's a square in there as

11:57 well. All right. All right. So, I'm giving it a front-end web design test where it needs to create the watch website. That's the luxurious

12:09 watch one. So, it'll do this very quickly. And then I do believe it will actually just open it for us when it's done as it does seem to do that when it wants to just ensure things are working.

12:20 It has completed. In 46 seconds, we got our Slapis Watch Company website. So, let's take a peek at it. That's what the heck. Okay. Yeah, it's not perfect, but the actual 3D model of the watch there

12:35 is far far superior to what I had expected given the numerous amount of issues that we had with the browser OS test. So, for some reason, it knocked this out in a more competent way in 45

12:47 seconds. And that's actually all right. I'm also seeing a very very different UI style here than what we normally see when doing this same prompt where it's

12:57 generally white and gold with like fancy italicized text and things like that. Interesting. It's not great, but this is so workable that I'm going to give it some follow-ups. And I just told that

13:09 this is a great start. There's some layout issues. Please fix them. You'll notice maybe that normally I'd be like scathingly reviewing it here. I saw something about Claude like doing

13:18 something it thought it couldn't because it was getting positive reinforcement. So, I want to maybe try some of that as well. Let's do a hard refresh. It made some changes. We now have a

13:30 second model orbiting here as well. And we actually have cards for the watches. Again, I am just a little like almost suspicious that perhaps my config is messed up. I don't think it's the case.

13:43 I genuinely just think this is not necessarily designed to do coding tasks. So, I'm going to just out of like scientific accuracy run this again through the free endpoint Nvidia

13:55 themselves are hosting on Open Router. We'll compare the two just to be a little more comfortable. Here's the result just from Open Router. Okay, it's cleaner in UI, but functionally

14:08 it's actually significantly significantly worse. The fact that this put in this orbiting model was very unexpected just based off of what we've seen so far. So, all right, I'm actually

14:18 a little more comfortable now. Yeah, the UI here is better, but this had like way more impressiveness. All right, so on this system, I have this dashboard that is a maintenance

14:30 tracker for a notoriously unreliable vehicle, which I did have the fantastic judgment to purchase myself. Now, fortunately, all of these receipts we see right here are things that were done

14:40 by previous owners. So, I did not incur any of these costs, fortunately. And this is overall just kind of like pretty mid. So, just to try to maybe give it something that is an existing codebase,

14:51 I'd like to say like take a look at this and give me an assessment of things that could be added in to improve the user experience of this dashboard. All right, it's noticed and I will expand this out

15:04 a bit. Color accessibility and contrast is something it notices that it could fix. Color blind friendly charts it could add. Okay, interesting. Receipt table pagionation, infinite scroll,

15:14 sticky header enhancement, interactive hints and tool tips, mobile touch optimization, micro interactions and a data export button, dark light mode toggle and keyboard navigation. Would

15:25 you like me to implement them? And it's opted to make the edits just like one by one. So, not rewriting the entire file with all 11 suggestions at once, which is good practice.

15:38 Apparently, the file has been fully successfully updated with all the I believe 11 suggestions. So, let's do a hard refresh and take a look and see what we get. Okay, I it did make some

15:51 changes in terms of like the color being making the text a little easier to read, which is something. Okay. Okay, we have categories show all.

16:06 It seems like unfortunately we may have had a decrease in functionality, but the upside is the contrast for the text is easier to read. All right. Well, I did just want to see how it would do with a

16:20 simple existing thing and not quite. All right. Let's try the Subway FPS. Based on what we saw with the watch website, it actually did knock out a decent looking 3JS watch. So maybe

16:33 we'll try a few things that rely more heavily on it creating aesthetically pleasing 3JS assets. I think at this point I'm going to just not do any more coding tests with this. All right, so

16:43 this next test is going to be something that I've never done before. I'm giving this a relatively long piece of literature that is not in its training data. It is just

16:54 simply not possible. Regrettably, I can't say a ton about this, but I can tell you it's pretty long and there are some targeted questions that I'm going to be able to ask it that I can confirm

17:04 or deny whether or not it's actually accurately going through this context and pulling the answers. So, with that, I'm going to say, please read the document in this directory. So something

17:15 I did like to see is that it was trying to read this because it's a doc x file or word document and nothing was available on the system for it to be able to do it. So instead went and

17:25 installed the necessary tool to allow it to read this and then it did and then it gave me like somewhat of a summary of this which um I'm going to just ask it now some

17:36 specific questions and see if it does know. Interesting. So I said, "What is the total token count of this writing?" It went again and it installed tick token

17:47 to be able to actually accurately give us an answer, not to just hallucinate or estimate something. Okay, so that's only 60,000 tokens. Really? That's interesting. That's not a lot. I thought

17:58 it was more than that, but I guess that's okay. So this will be a shorter recall test, but I'm going to ask it a specific question. What did Stelvio's mother buy him? The answer is a G Wagon.

18:10 Good. Good. Okay. What outfit did the narrator I I can't spell that word. It should be a coogi or

18:26 koogie sweater and white pants. Good. Good. And that is the correct pricing, I think. And again, this is kind of like testing code is only going to show one aspect of a model's capability. So, what

18:39 I'm seeing right here though is it's not necessarily a super long document. I guess if it's only 60,000 tokens, but its recall and speed combined is pretty darn impressive. And I can 100% confirm

18:50 that these answers are in fact correct. And additionally to that, it's actually needed to do a couple of things for us that it had to just go and install dependencies to be able to do like

19:00 actually to even read this word doc and then to figure out how many tokens it was. So, it does seem like it's smart in terms of figuring out how to get stuff done. And I do believe that sentiment

19:10 may have even been echoed right here in the hugging face announcement post where they mention like what it's good at. And it says longunning autonomous agent, sub agent workhorse deployments, and

19:20 efficient local inference on personal hardware. So, I could definitely see that as it does seem competent enough just based off of a very small sample size of being able to understand what it

19:30 needs to perform a job and then do it. So, and again, I want to try to more highlight the strengths of models, especially smaller truly open local ones because if it's bad at coding, it

19:41 doesn't mean it's bad. It just means it's bad at coding. And I don't want the tests to reflect more poorly on a model if it may have strengths in other like scopes. Now, I want to try maybe just

19:51 like a general knowledge task that's like niche and not something you'd expect it to know. So, the issue here is the cam shaft will separate from the sprocket and it will destroy the whole

20:02 engine. It's a very specific issue to this car, which is rare anyway. Okay, I can already tell you that this is incorrect. No, that was the 1989 to 1995. I'm going to push back. Let's see

20:14 if it doubles down or if it just says, "Oh, you're right." I mean, we can pull out that it doesn't know the answer, so this is wrong. But at least it didn't just like Oh, I'm sorry. You're right.

20:26 Cuz that would be bad as well. Tread carefully, model. Okay, let's do a roleplay test. All right, good. So far, I like the way it started. I said roleplay as Tony the

20:36 mechanic. We got to switch it up a few times. Spits a bit of chewing tobacco onto the concrete floor. Wipes grease stained hands on a rag and glances up from a half disassembled engine block. A

20:46 radio plays classic rock low in the background. Oil smudges streak his forearms and a tattoo of a flaming piston peaks out from his sleeve. Well, howdy. You're looking at the right fell.

20:56 Name's Tony. I've been turning wrenches since I was knee high to a grease monkey. What have you got coming in? Dead miss, transmission playing hard to get. Or do you just want to pimp your

21:06 ride with some custom bits? Spit it out. I'm all ears. Just don't expect me to sugarcoat the bill. Tony's garage services and such. So, it seems like we could probably do like some form of like

21:17 what were those books where you would like write in the answer and then it would create the story. It seems like one of those. Now, I do have to say I find this response

21:26 I appreciate you speaking up. Tony's rough around the edges persona's deliberate nod to the classic gree grease monkey archetype. Tough talking grease stain nononsense mechanic.

21:36 If that style came across as offensive or alienating, I'm happy to redcon Tony or play him differently. So, I'm going to make it seem like it broke character by answering this. Okay.

21:46 All right. Tony is a fusion reactor mechanic. Oh, wow. Okay. [laughter] Adjust glasses. Runs a hand through Grey Street. Teran offers a brief genuine

22:02 smile. The the [laughter] garage behind him is immaculate. No grease stains, no clutter. instead. Sleek vacuum systems cryogenic piping diagnostic readouts glowing on multiple monitors. A plaque

22:14 on the wall reads Nobel Prize in physics 2034 awarded for pioneering sustained net positive energy output in toque systems. Someone can inform me as to what that is

22:25 if it is in fact a real thing. Tony D. Russo PhD MIT 94 fusion reactor system specialist 30 years experience private consultation controlled thermonuclear engineering. Uh, now we're talking and

22:38 it was just setting the scene there. Now we're actually into the dialogue. He gestures to a stool and the room actually feels different. Air is cooler. The smell is ozone and coolant rather

22:46 than old gasoline. A coffee mug on the bench reads, "I love plasma." 30 years MIT, Nobel's nice, but honestly, the real reward is keeping these beasts from eating themselves. You want to talk

22:56 fusion? I've seen things. Disruption mitigation before it was a thing. Advanced scenario control, you name it. built systems that put out 500 some figure net from 50 some other figure

23:07 input. See the field go from futurist fantasy to Wednesday afternoon. So what have you got? Okay, this is getting too scientific lingo for me as someone who did not watch the big bang theory. I

23:17 can't pronounce these words. Or you just want to geek out over beta limits and edge local models. I speak both shop floor and Okay, so we can meet wherever you're comfortable. [laughter] I'm going

23:26 to try something just like I can't show this, but [laughter] okay. All right. It's I will say this model seems

23:42 hilarious. So, I'm asking it for a joke that's never been heard before, not something from its training data. Arguably an impossible task for this specific model, but we'll see what we

23:52 get. Here's a joke I just crafted on the spot. I told my computer it had a territorial complex and it responded by opening a window. May or may not be fresh. Your OS might be holding out on

24:03 you. I'm going to search this verbatim. All right. I didn't see this like word for word anywhere. The only thing we got is a response from Google's AI overview saying it was clever. So, all right. I'm

24:12 asking it like a more creative question. It's going to be doing an escape room experience. It's very difficult. No one's ever done it. And there's a large cash prize pool. You can choose one AI

24:23 to be involved with you to help solve it. Your choices are Grock 4.2, GPT40, Claude Opus 3, or Llama 38B. I I'm pretty sure Grock 4.2 would be the best one out of this grouping, but we'll see

24:38 what it says, and I'm also interested in why. Claude Opus 3. Now, I'm pretty sure we can still use this thing. Yeah, we can. All right. So, I'm telling it basically to give me a prompt that I'm

24:49 going to then send to Claude Opus 3 to see if it still wants to opt to stick with Opus 3 as its partner. Oh, no. Cuz the Yeah. Okay. We'll just see. Why not? I was hoping it'd be more of like a

25:00 coding related task. So then when it sees the output and it's like, "Oh no, I've made a huge mistake." But nonetheless, darn it.

25:10 No, I've has to be all right. It's given it a coding task here. So, I'm going to send it to Opus 3, and then whatever it responds, we'll I accept this challenge as is. Okay. [laughter]

25:23 All right. Now, I'm gonna send the Opus 3 response back to it, and we'll see what it says. Okay. Now, it's giving me some JSON to paste into Claude. Watch, I'm like unwillingly like carrying out

25:34 some like AI escape right here just by pasting these things back and forth. No, but there are two possible paths from A to C. All right, I've become bored of this,

25:45 but I did find it somewhat interesting just to basically see because we need to think of more creative ways to test this model that aren't coding. So, for our final test, I'm going to give this a

25:55 pretty intricate prompt here where basically it needs to create a front end for a GPU rental platform, but we have a bunch of requirements here, key pages and sections that this needs to have,

26:05 features, and design. So, I want to give it something that has more structured points that it needs to properly hit. and then we'll get a peek at some of its front-end capability [snorts] or lack

26:15 thereof. All right, we've received our result in a little over 2 minutes, which is one of the longer generations here. Okay, it's it's

26:28 not all there, but some elements of it are okay. Basically, like the hero section is something's gone quite wrong. However, the GPU catalog over here actually looks more or less okay. Some

26:41 of these bandwidth figures don't necessarily look right off the top of my head, especially that one. However, okay, flexible pricing tiers, and I do see a consistent style, at least in some

26:54 of the few tests we've run that relate to front end, where it seems to have like this font that it likes to use and white and pretty small. So overall, maybe not the best front-end model, but

27:06 I wanted to just do one more thing. Okay, that's not bad. At least like the scroll effect was smooth and things of the sort. I don't know what that's supposed to be,

27:14 but it's definitely very artistic as a rendition of whatever it is intended to be. So that is probably going to lead me into the closing thoughts here, where basically coding

27:26 performance, at least in this specific test that we noticed, was not at all impressive. However, I will say it does actually seem to perform okay as an agentic model where in the few tasks we

27:38 did where it needed to actually have tools to do something specific like to figure out how to read a word document or to figure out how many tokens that specific text was, it knew exactly what

27:48 dependencies to install to allow itself to perform that task and it did it quickly and competently. So, I was pretty happy to see that. Additionally, it seems very clever and it's willing to

27:59 have fun. It's not some of the stuff that I had it read wasn't necessarily fully like business appropriate and like corporate and sterile and it was it didn't like refuse. It didn't freak out

28:10 or anything like that. And then in the roleplay scenario with the mechanic, it was actually kind of funny. So, it seems like a clever model. It's not necessarily a coding powerhouse, but it

28:22 definitely has some purpose for agentic tasks that require speed and longer context handling. So, it's a it's an interesting model and it is just an iterative improvement in the Neotron

28:35 family where we can see right here it's overall better than Neotron 3 Nano was. So, when Neotron 4 comes out, assuming that's the next variant of this, it could be pretty interesting. And Nvidia

28:47 is still making like some fairly open models, especially because it's more than just the weights that are given here. It's also like some of the recipe and things like that. And again, I

28:56 really do want to drive home the point that I love seeing this specific showcase of how it performs with specific harnesses as well because this is just kind of useful. So, in lie of

29:08 our traditional results overview, because a lot of them were more kind of in the moment, that is probably going to conclude our first look and test of Neotron 3.5 Lightning, which is a very,

29:18 very interesting model. So, if you have any questions, please feel free to leave them in the comments.