Nemotron Lightning - NVIDIA's Super Fast Agent MoE

So, your agent spends 90% of its tokens just doing absolutely boring work. If you look at people's traces, you realize that the tool calls, validating outputs, formatting results, most of that can just be done by a very simple, cheap model. And NVIDIA has released just a model like that, specifically for that layer. It's totally open weights, they published the post-training recipes and data and all of that. But, what makes this really interesting for me is not the fact that this is a small model that can be run pretty easily. It's the fact that NVIDIA's own customers are realizing taking this small model and just giving it a very small amount of fine-tuning can make it way better for their particular use cases. And that's exactly what this is. NVIDIA has built this NeMo Tron 3.5 Lightning. So, this is an open 30B with 3B active mixture of experts model. And really, it's been built for two key things. One is to basically go along with their whole routing system called SwitchYard. So, if you haven't seen my video on that, go and check out the video on that. That's super interesting of what they're doing there with creating their own sort of routing system that allows you to go for big models, small models, etc. But, the second thing, this is the smallest member of the NeMo Tron 3 family here. It's basically distilled from NeMo Tron 3 Ultra. And it's made not to be a chat model or a sort of frontier reasoning model. It's just built for the execution layer of long-running agents. So, things like tool calls, validation, retrieval, all your rag stuff, summarization, classification, etc. And all of that benefits when you fine-tune it for your particular use case. So, the interesting thing here is not that this is an awesome model at intelligence or something like that. It's that it's been made for this grunt work and it's been made so that you can fine-tune it specifically for your kind of grunt work to get the best results out of it. Now, with something like this, when you're using this grunt work model here, you really care about two main things. You care about cost and you care about speed. And the speed story here really is an interesting one. So, they're claiming that they can get 4x throughput versus similar-size models from the Qwen family and from other similar kind of models out there. In fact, they tested it on Pinch Bench with 10,000 different tasks, and this is 30 to 35% faster than the similar-sized Qwen 3.6 MOE there. So, how do they get that speed? Well, they've gone for the hybrid member transformer approach. This is the same as the rest of the NeMo Tron family, but more interestingly here, they've actually baked in a multi-token predictor during the sort of continued pre-training of the Nano 3 base model that this was originally based on. They've also incorporated D-Flash, a speculative decoding drafting model here, which we've seen speed up models a lot. So, this is great to see them actually sort of incorporating this from the start. And then the other thing that's really interesting here is that they've also incorporated D-Spark. So, this is a D-Flash variant tuned for the DGX Spark itself. So, this is all derived from the DeepSeek methodology and paper that they published showing how they can use speculative decoding to massively increase the speed of their models. And it's cool to see other players like NVIDIA pick up on that, realize that that's a big win for the community, and just start incorporating it in the models that they're releasing. And I hope we see this going forward not only from Nvidia but from other model providers as well. All right, so Nvidia has shipped this both with a Bfloat16 version but probably most people are going to use the NVFP4 checkpoint. Obviously it's been tuned for Nvidia hardware and they've kindly sponsored the compute for the video. So if you do have an RTX graphics card or a DGX Spark etc. This is really made for you. But along with shipping the model itself like I mentioned at the start they're really pitching this as a customization play and they've got some nice case studies in their blog post of partners using the post trained versions of lightning for doing really specific use cases at scale. So for CrowdStrike they were basically able to take this and fine tune it to get Nemotron 3 super level accuracy at about 1/5 the cost there and I'm guessing a lot faster as well. They've got an example of Code Rabbit and Base 10 basically training this thing for one epoch under three hours basically costing around about 100 bucks and then being able to use that as a router for a whole different system. Now of course they're showing these because these are success stories etc. But it does actually make sense because along with the models and the data that they've actually released here you've also got people like Unsloth providing scripts to basically fine tune this on consumer level hardware so that you can run that fully locally and customize the model for your particular use case. All right, so let's jump into a demo and have a quick look at what it actually do etc. So when you're testing this model it doesn't make a lot of sense to give it a lot of the intelligence tests. Quite simply one of the things that you'll see is that it doesn't have a lot in thinking tokens the reasoning is not great and stuff like that. But when we look at it for these sort of agentic tasks you can see that consistently again and again, it's able to pass the vast majority of them. Not always all of them, but it does a very good job at being able to get the basics down. Now, one of the ones that I noticed that it does fail often is prompt injection resistance. So, this kind of shows that it's not great at being sort of the front level model, right? You want to probably have an orchestrator model above this to deal with these kinds of things. But, being able to do retries on errors, remember this is, you know, deliberate errors injected in there to see, does it give up? Does it keep going? It keeps going. It's able to get those. Doing a lot of the other agentic tasks in here, it's able to go through them and just get them right. We can see things like when it's not told not to use a tool, it knows not to use a tool and just use its own ability to answer. We can see long-running tasks. It does good job of those things. And even chaining multiple tools together, it's able to do that. Now, if you look at it, it really doesn't have the long chains of thought that you'll see in the better quality reasoning models here. But, on the whole, for the grunt work that this is actually made to do, that's actually fine. And it's not about getting the best intelligence all the time for this. All right, so just to wrap up, this has got a great license where again with the Open MDW license allows for commercial use, distillation of outputs, no attribution is required, etc. Basically, you own the model and you can do what you like with that model, which is pretty awesome and also is sort of going in the spirit that Jensen actually released on Twitter as his first tweet a week or two ago now. As always, one of the things I love about the releases is that on top of the weights, they're They're releasing post-training data sets, plus training recipes. You can actually go through and see how this was actually put together. And if you're looking to learn anything about that, that's really just gold, right? Being able to look at like how are they actually doing these things? And then if you want to sort of try doing experiments, etc., you can try your own strategies around curriculum learning, around sort of on-policy distillation, RL, all those kinds of things. So, unfortunately, this is just a text-in, text-out model. We don't have any sort of vision here. It seems to be that they're leaving that to the realm of the Nano Omni model, which I made a video about before. And like I said, this is not the sort of high level of intelligence kind of model. This is something that's going to replace your high-cube model, perhaps replace some of the proprietary flash models and stuff like that out there. If you're just purely looking to run a model as a coding agent, for example, this is not the model for you to actually check out. Just finishing up, another key thing is this model was really sort of made and actually came out with the Switch Yard release. So, if you haven't seen my video on that, check out the whole Switch Yard routing system that Invidia's released. Very cool to see what they're doing there. And I think we're going to see a lot more things like that going forward from Invidia and others. So, let me know in the comments if you are fine-tuning these kind of models, what tasks you're actually sort of focused on there. And have you actually looked into using a full sort of multi- team model strategy, not multi-modal, multi-model strategy. That's certainly something that that's taking off. Anyway, as always, if you like the video, please click like and subscribe, and I will talk to you in the next video. Bye for now.