NVIDIA’s Nemotron 3 Is... Awesome?

Over the past 6 months, Nvidia has been releasing Neotron 3 in three variants. But why exactly three? What criteria did Nvidia use for sizing these models? We try and size them based on the most common compute footprint available from Nvidia's install base. And we do that so there's a variety of both accuracy and cost latency throughput trade-off. Essentially, this by the way is Joey from Nvidia during our interview. So the size of nano and super variants target the chip layer and the ultra targets the infrastructure. For example, Neotron 3 Nano is a 30 billion parameter model meant to run on consumer hardware that we could get at home. The Nano from December is a 30 billion total parameter active 3 billion. The MVFP4 allows us to run that model at essentially half FP8 memory capacity. So instead of 30 billion total say gigabytes, we can now run it around 15. Now the super variant is 120 billion in size similar to open adoss 12b. Now, you could run this on Mac Studio or DJX Spark, but the compute profile mirrors closer to server grade GPUs like the H100 and A100. In super, it's 120 billion, active 10 billion. And inside of there, say like on one H1 80 gig, we can run an FP8. The MVFP4 runs great on the Blackwell architecture. And so, say like B200, B300, there's not just a memory reduction, but a compute acceleration there. That's also what is in DGX Spark and allows the super model to run on on a smaller footprint in DGX Spark and we'll see more of that going forward with things like DJX Station as well. Finally, we have the Ultra, which is massive 550 billion parameter model with 50B active, similar sizing to models like Deepseek V3. At this scale, you're going to need more than one GPU to run models like this, which is why the Ultra is fitted for AI factory level at the infrastructure layer. On the ultra side, we're looking at around 550 billion total. And as we went through and looked at some of the base model accuracy valuations and so these are essentially kind of next token prediction. So it's not it's not reasoning and it's not instruction following. But even looking at previous base models out in the community, we were able to see the best accuracy scores we've ever seen of any base model published. As you can see, Nvidia is slowly moving up the AI5 layer cake and pushing out models like Neotron 3. And because Nvidia practically dominates in the lower layers, they have an advantage in fitting their model architecture to be fully optimized at the model layer. In other words, Nvidia can leverage their hardware to make the best fit for the model given various hardware profiles. So the natural question here is how exactly does Neotron 3 stand out against other models in the model layer? What architectural decision did Nvidia make at the model layer to fully optimize the hardware underneath? When it comes to the model architecture, there are three things worth noting about Neotron 3. Hybrid Mamba transformer, latente, and MTP. Now, most of you might have never heard of these terms before. So, at first glance, it can be really difficult to contextualize the significance of these, but I'll walk through them one by one. Let's start with the hybrid Mamba transformer architecture, which directly targets the biggest challenge when it comes to scaling large language models, which is attention. Yep, we've all heard it before. attention is expensive and this and that. And we also hear things like attention makes compute scale quadratically as the context window grows. All of these are definitely true and we certainly are in need of a good solution for it. Especially if you're going to offer a context window in the size of 1 million. Nobody in their right mind would release a model with 1 million context window without doing something about this quadratic scaling in attention. And for Nvidia's case, they decided to take a shot at this by changing the model's architecture itself to use what's called Mamba 2 and then interle them with full attention layer. Some of the more traditional attention mechanisms can be a bit more quadratic and they grow significantly the longer the input. Our concern with that is that often in an enterprise setting, you'll have existing knowledge bases or experts and there's a lot of context to feed a model and it can give a much more accurate answer if it has a thorough understanding of what the question's about. Even something simple like, you know, what's the tire pressure, you know, in a car? Being able to go through the owner's manual and pull out the sections relevant to how to measure the tire pressure and what the recommended number is, being able to feed those into context, our concern is that it would grow the memory significantly, which then reduces the memory we have for concurrent queries. And so in thinking about things as as an AI factory where a model should be able to handle many queries concurrently, we were getting frustrated by the amount of memory used. So given this scaling problem when it comes to attention, commonly labs would target this problem by swapping out the attention head from multi head attention to different ways like GQA and MLA to essentially minimize the KV cache. There are of course other ways like turbo quant where you quantize a TV cache itself to lower precision. But Mamba 2 works completely different than any of the ways above. >> And so one of the exciting things about Mamba and kind of the state space mechanism there is that it's more linear than it is quadratic. And in that sense we can be able to take these longer input sequences and use less memory. The actual mechanics of Mamba 2 is quite technical. But unlike traditional transformer architecture, Mamba or state space models don't actually use KV cache at all. Instead, memory requirement is constant since the representations are stored in a fixed size matrix that updates as tokens are processed. And this might sound really familiar because it sounds a lot like how recurrent neural networks actually work given their hidden state that updates as tokens are processed. And Mamba 2 essentially compresses the path into a state then uses a hardware friendly algorithm. So those state updates can be done as a big matrix multiplication and paralyze efficiently. And we couldn't really do this with RNN's. And since we risk losing broader dependencies when using mechanism like this, interle mamba 2 layers with full attention is how Neotron was able to keep their inference so tight and offer up to 1 million context window without too much sweat. Now, let's go to the second point in the architecture, latente. And this is a really interesting concept, and here's why. Quick shout out to Code Rabbit sponsoring this video. Ever since I started relying on codeex and cloud code to write code for my projects, not only did I start making more pull requests, I find that my focus tends to be more on the flow state to develop things than actually reading the code that was generated with coding agents. Code Revit works besides me that reviews my pull requests automatically with the entire repository in mind. and part of its review includes fixing changes, security and quality checks and most importantly gives actionable changes that's ready to be merged right away. You can also use Code Rabbit agent in Slack so you and your team don't have to leave your usual environment to talk with your agent and bring the context of your repository directly into your workflow. My favorite feature is that Code Rabbit will remember you and your team's preferences. So as you continue to work with Code Rabbit, you will get better over time. Check out Code Rabbit using the link in the description below. Now, we all know how mixture of experts ore works where instead of activating the entire model's weights, you sparsely activate the model. Let's say 10% of the model's weights are activated instead of 100%. Now, on the hardware side, you see a huge improvement here since only 10% of the model's weights needs to travel from HBM to SRM for compute to happen. Now, imagine we disperse these experts across a node of GPUs through what's called expert parallelism. Spreading the experts like this actually speeds things up because you now have multiple funnels that carry the model's weights from HPM to SRAMM. And since the funnel, which is the bandwidth that typically gives you 3 to 8 terabytes per second, you can have models like Neotron 3 Ultra, which is a 500 billion model, which is roughly 275 GB in VRM in 4bit precision, takes anywhere between 91 milliseconds to 34 milliseconds using just one funnel. But having eight funnels that each have 3 to 8 tabytes per second essentially speeds up the read rates theoretically since each GPU should only have roughly 35 GB to stream from HPM to SRM. And because we're only activating 10% of the model on top through mixture of experts, we can go ahead and slash that number down even that much more. Now, of course, this is only looking at the hardware side, which means the model architecture also needs to make sure that the quality and accuracy isn't dropped from this kind of setup because clearly it's a benefit on the hardware side. The company that really took Mixer of Experts to another level was none other than Deepseek. Deepseek essentially proved to the world that we can add load balancing to make sure that you don't underutilize or overutilize experts. And they also find a way to really specialize each expert to make this kind of parallelism work in quality while having one shared expert for general knowledge. So what about Neotron? What does this really look like in Neimotron? Turns out beyond Deep Seek's breakthrough when it comes to making mixture of experts actually work, there's still a lot of room left on the table when it comes to hardware optimization. And Nvidia introduced what's called latente targeting this very gap. And the core idea behind it is this. What if we have a down projection on the dimension so that each token being processed is reduced in footprint? This way we can get away with activating our experts on a much smaller latent representation which cuts down memory bandwidth and also cuts down on compute that's typically needed to route and process each token as it comes. For specifically for the late MOE, every query goes through the router and we wanted to think how do we make that router more efficient? How do we think about getting more experts exposed to pick the right one? And so, you know, in that scenario, what we thought about was for the late demo, we wanted to be able to essentially kind of reduce the footprint, the compute, and the memory it takes and be able to get more experts to weigh in on a query to make sure we're always picking the best one. So, what you find is that this kind of architecture actually creates more room since you're doing math on latent representation. And Nvidia made a design choice here to use the surplus that we have gained to pack in more experts which gives each token ability to see more experts than it would have otherwise. Pretty clever, right? Now we finally get to MTP or multi-token prediction. And this has to do with how output tokens are generated by the model. And these are the tokens that would typically see on screen as models generates tokens. Traditionally, tokens are generated auto reggressively, which means the model will generate the next token and takes that into account to generate the next one one at a time. And MTP is meant to address this very specific region of the model by changing up what the model can see during auto reggression. Now, I wish I could cut here to Joey talking about MTP, but I totally forgot to ask Joey about Neotron's incorporation of NTP, so I guess you're stuck with me trying to explain it, but I'll do my best. Multi-token prediction lets the model predict not just the next token but the tokens after that. We can have the model produce let's say next five tokens which theoretically gives the model deeper intuition on how the next five tokens would look like if the model could generate them. Now MTP is typically used for training to make the model more expressive and theoretically have a better forwardinking ability as it generates token after token. But it can also be used during inference as well in the form of speculative decoding where the model drafts few tokens ahead then checks them in one pass and keeps ones that match what it would have generated anyway. And Neimotron 3 allows speculative decoding to be done using MTP which theoretically should speed up your token production side since it could skip over multiple tokens as it predicts and accepts tokens beyond the next coming token. Now that we covered with a broad stroke what Neotron architecture looks like, let's briefly talk about licensing. We all know that labs tend to flip-flop between how open they really are when it comes to their model releases. And what's even more confusing is licenses like Apache 2, which can be confusing when it comes to scope. And that's largely because Apache 2.0 is meant to be used for software, not for AI models. For example, we know that many open labs release the models weights, but they also release code that's used to run them for inference as well as few other artifacts like documents, RL environment, and training recipe. So, the question really becomes when a lab says their model is open, what exactly does that mean? Recently, Linux Foundation revised their open MDW license from 1 to 1.1 to help reform the language around AI model licensing. And Nvidia adopted this very license to clarify their stance on Neotron model along with some of the other things they're working on like Cosmos, Isaac Groots and