NVIDIA Doubles Down on Local AI With PAIR

0:00 Okay, so this past week has been, to steal a YouTube cliche, a huge week in AI. And in many ways, this week we've seen a lot of things that were bubbling away below the surface for perhaps even

0:13 6 months for some of these things rise up to the top and actually get released. So, of course, I'm talking about this week, we had the launch of Fable 5.1 and Mythos from Anthropic. probably a little

0:26 bit more interesting, we had the whole launch of GPT6 Astra. And I feel that those models really show that on the continuum, and I really stress the word continuum there, we're moving speedily

0:39 along towards AGI and perhaps even ASI. But while there are sort of 10,000 videos out there covering these models and showing you all the cool games that you can build and all those things are

0:51 true and very cool and I've definitely enjoyed this weekend playing with some of these models. In this video I want to focus on Nvidia. Now later on in the video I'm going to talk about a new

1:01 release that they had that still is very early days but kind of signals where they're going for a lot of this stuff. But first off let me show you a few interesting things that happened with

1:11 Nvidia this week. So, if you've been watching my videos for a while, you won't be surprised that one of the things I've consistently complimented Nvidia on when they release their

1:21 openweight models is that they're not just actually releasing weights, they're releasing papers, they're releasing recipes for training. And often, most importantly, they're releasing these

1:31 data sets which allow people to sort of see how these models are being made and then to work out improvements that they can make for fine-tuning their own versions. Now because of that not

1:43 surprisingly this year Nvidia has overtaken all the other enterprise players to be the number one contributor of repos on hugging face. Now the key thing is that Nvidia is making a huge

1:56 bet on open weights and to use a term that perhaps was corrupted by another company on open AI and by open AI here I am talking about the world of open weights open models open data sets etc.

2:10 So back at the end of July, the CEO of Nvidia joined Twitter or joined X.com for the first time. And his first post was literally this letter talking about open weights and what Nvidia was doing

2:24 to be committed to this and encouraged other companies to sign it to get on board with supporting the concepts around open weights and the open use of AI and not having it dominated by any

2:36 one particular company. And I think in the end pretty much all the AI companies with the exception of Anthropic ended up signing it. Jump forward back to this week. Nvidia is doubling down even more

2:48 on the concept of openw weight models and making this stuff even more available to people by buying Hugging Face and they certainly paid a large amount for it. This is just under $13

2:59 billion. So congratulations to all my friends who work at Hugging Face. I hope you waited your compensation in shares more than salary because you've probably made a very nice payday here. And one of

3:09 the funny things with that acquisition, the actual amount turns out to be the decimal value for what now is called the hugging face emoji here. So contrary to what a lot of people are saying, I don't

3:20 see that Nvidia buying hugging face is actually going to be really bad for the open community. I think in some ways this may have actually defended off a number of other companies that were

3:30 trying to buy it that perhaps would have made changes to it. It's in Nvidia's interest to keep this open because they are benefiting so much from this. In many ways, you could think of Hugging

3:41 Face at the moment being the hub of local AI and this is what Nvidia really wants to own a large chunk of. They can see clearly the changes that are starting to take effect. And the changes

3:53 that I'm talking about is the whole play of AI hitting mass adoption. So if you're listening to this video, you're almost certainly not one of the masses. If you've been listening to my videos

4:04 for a year or two, you're one of the super early adopters here. And that may have been because of your job, that may have been because your interests, etc. But don't be fooled. While people like

4:14 you and I have been watching all these models come out and getting amazed over the last x number of years, we are not the masses. The masses are coming along and starting to realize that one AI can

4:28 be useful for a lot of tasks and two that probably they don't want to give away all their data after they've seen what's happened with a lot of the social media companies over the years. that's

4:39 creating this huge market in local AI and rightly so Nvidia wants to capitalize on that and capture as much of that as possible. Now these are some of the interesting stats that if you

4:50 look at the world of openweight models and local AI you can see that year to date already on hugging face open models are well over four times what they were last year and we've still got a long way

5:03 to go in the year. We can see that the whole sort of generative AI models just in general are going up massively as well with things like Comfy UI. And as much as I love Comy UI, that's probably

5:15 not a mass product yet. Already it's well over doubled this year compared to last year as well. And the reasons driving this is that we're just seeing the whole openweight system catch up

5:27 faster and faster to the proprietary models. Now, I'm not saying that the openweight models are better than the proprietary models. I think this week's releases show that yes, the proprietary

5:37 models still ahead in many ways, but look at things like this. It's only a few months ago that we had something like Claude Opus Max being released in April 2026 that was reaching a score of

5:49 55 on the artificial analysis intelligence index. Quen 3.8 flash next has already surpassed that and it came out in August this year. If we look at video models, Gemini Omni Flash, which

6:03 came out just in May around at the time of Google IO, again looking at artificial intelligence scores for video models, the original version of it has been surpassed by Miniax H3 in August

6:17 this year. Now, yes, these are not models that are going to fit on your phone or anything. They require a serious amount to compute to run them locally, but they are able to be run

6:26 locally. So that brings us to what Nvidia launched this week. Now they've launched this new kind of router. So this is called Nvidia pair and pair stands for personal AI router here. And

6:40 the whole idea here is that Nvidia is kind of betting that in the future people are going to have multiple machines in their houses that are GPU enabled or that are going to be

6:51 accelerators. And you've got to remember that this is coming just before next month's launch of the RTX Spark laptops. And there already a number of hardware makers that have announced that they're

7:02 going to be bringing out their own RTX Spark. Now, those are Windows systems with RTX built into them. And you could imagine in the future as people start to upgrade their laptops, their computers

7:14 over the next 5 years, many of these machines are going to have some form of accelerator in there. Whether it's a GPU, whether it's an NPU, whatever people are going to call it, it's going

7:26 to be a chip or part of a chipset that is built to process AI models locally. And what video pair here is trying to get ahead of is this whole idea of if you've got multiple machines in your

7:39 house, how do you actually get the best use out of them for agents like Hermes, like Open Claw? So this is fundamentally a router that's quite different than something like Switchyard which we

7:51 looked at last month. Switchyard was all about picking the right model across an API whether that was cloud hosted or self-hosted and it would pick that model based on a variety of different

8:04 conditions or models to actually go through this. So you could sort of think of switchyard as being the quality and cost decisions for things that may leave your network. Pair on the other hand is

8:16 for hardware that you control. And the problem that this is going to solve here is that as we've got a lot more agentic software, things like Hermes and Open Claw, we know that one of the key things

8:27 that makes them so good is that they have a lead agent that can delegate whether that's sort of research jobs or coding jobs or other sort of tasks to parallel sub agents. and those sub

8:40 agents can then go off and do their bit and come back and deliver their results to the orchestrator. Now, if you're trying to run that fully locally, the challenge is often going to be is those

8:51 sub agents are not actually working in parallel at all. They get into a queue waiting for the GPU to be available. So where pair comes in is it allows you to basically serve other models on multiple

9:04 machines on your network and have those sub agents be spread out across your entire network and not just queued up to work on the one machine. So in some ways you can think of pair as like your sort

9:16 of virtual inference traffic cop. It sees that okay machine number one is busy so it then delegates these calls over to machine number two. And the cool thing here is that your agent never

9:27 really needs to know this. So your agent thinks that it's just calling Alama or just calling LM Studio and it makes the calls just like they would before. But pair then sits in the middle directing

9:39 the traffic over to your machine number one, machine number two, etc. as you go through. So, as people start to have multiple machines that can do this, and Nvidia points out that a large

9:50 percentage of households in the US already have two or more machines in their house, probably not with state-of-the-art GPUs. But the idea here is that you've got something which

10:02 allows you to take advantage of your gaming machine with perhaps an Nvidia GPU in there, of a laptop with an RTX Spark in there, perhaps a bigger workstation that you've got in an

10:12 office, that kind of thing. And of course, this clearly is early days. It's not something that everyone is going to have lots of these machines lying around. But it does show that Nvidia is

10:21 thinking about where local AI is actually going to go. Now, one of the things that's really important to stress here is that this is not VRAM pooling, right? It's not building a cluster out

10:31 of your multiple machines to make one bigger, stronger machine with more VRAM. Here, each request is running on a single machine and just that one machine. More nodes on your network are

10:44 going to give you more parallel requests, not faster performance for a single prompt, etc. And it's not going to allow you to then load huge models or something. Now, for me, one of the

10:54 biggest things that makes all of this interesting is that this is not just software that Nvidia has launched. This is an opensource Apache 2 license project. Anyone can fork this and can

11:08 change it and can customize it in any way they want. Currently, this is built just for your own personal network. But you could imagine if you wanted to give this to one of the latest coding models,

11:20 you could probably get it to work with networks in the cloud. And I do wonder eventually, do we see something like this and Switchyard merge together rather than be separate pieces of

11:30 software. So it's all up on GitHub. It is still literally at a 0.1 release here. This is very early days for where this is going. But you can see already out of the box they're supporting

11:42 Windows, Linux, and Mac. And yes, if you're running an Olama on a Mac machine, you can actually add that to pair. It's not just for Nvidia GPUs here. And currently, it's only

11:53 supporting Olama and LM Studio. But I've been on calls where Nvidia employees who've been working on this have said openly that they're open to pull requests. And sure enough, you can see

12:04 that there are already pull requests in there to support things like the anthropic style of API rather than just the OpenAI endpoint. people are already submitting pull requests for supporting

12:15 llama CPP and it's going to be interesting to see what does Nvidia actually accept, which pull requests do they actually accept and which do they not like, etc. But the cool thing is

12:26 because it's Apache 2.0, people are going to be able to fork this. If Nvidia ends up blocking these kinds of things too much, the community is just going to fork it and create their own version

12:36 going forward. And I think Nvidia is very conscious of that just by the fact that they've made it Apache 2 licensed in the first place. Okay, so how does all of this work? Well, pair actually

12:47 just proxy sort of takes over the ports for Olama and LM Studio. It just changes the ports for those inference engines so that it can sit on those ports and basically direct the traffic. It is then

13:01 connected over your network directing where the requests go to which machine running which engine etc. So the cool thing here is all your existing apps keep the same ports and you don't need

13:12 to make any changes in those whether that's Hermes or another coding agent etc. But pair will then redirect that traffic to the right place. So the setup for this is pretty simple. You can

13:22 basically just install it here. You can see I've installed the version for the Mac and straight away it's picking up that I've got Llama installed. I actually have LM Studio installed as

13:32 well. Not sure why it's not picking that one up in there. So I think technically they only support Apple M4 and above. So I'm using an Apple M2 Pro, an older Mac Mini in here. But you can see that okay

13:45 with Alama I can see like the VRAM that I've got, how much memory I've got available, etc. And then I can actually come in here and look it at OAMA. And I don't actually use OAMA that much

13:57 nowadays. So it's got some old models in there still. Yeah. Some of the embedding models, some of the other ones that I was using for research and stuff like that. So just looking at their video

14:06 demo here, you can sort of see that okay, once this is set up, if something pings it, it can allocate different models to different machines. So you can see in the demo here that they've got

14:19 basically a Quen model running on an RTX 5090 on the DGX Spark there. It looks like they've got another version of the Quen 3.6 Mixture of Experts model going on there. And this is able to basically

14:34 ping these and obviously you need to have them loaded to and ready to go in here. It's pretty cool to see that like they're showing that this is not only working with the RTX5090 in here, but

14:46 also with a Radian graphics and stuff like that in there as well. That said, for all of this, it's still early days. I think we're going to see, you know, a lot of changes with this. I I think

14:57 Nvidia is also looking to the community to see what changes people want to make and what changes people are interested to actually have here. Right. So to finish up, this is definitely worth

15:09 checking out if you've got multiple machines with GPUs in them. It could be useful for a lot of different things. In the past, I've tended to use tail scale for this. And I've had special models

15:23 for the sub agents on particular nodes that were running. So for example, on the DGX Spark, I'd have multiple GGX Sparks running smaller models. and then something like the RTX Pro 6000 running

15:36 the Quen 27B model or something like that. I don't think the 0.1 version of this is necessarily going to change hugely how I do things, but I can see that this is a project that's going to

15:50 be really interesting because as people start to integrate things perhaps like Switchyard to it or other routers in here, then for sure the whole thing gets a lot more interesting to be able to run

16:01 with your agents. The biggest downside I see of this currently is just the speed. So while I'm very lucky to have a lot of accelerators that I can use etc. the challenge always becomes about speed of

16:15 like how do you get the tokens to be faster? How do you sort of improve that? Now since I made that last video about the Quen 3.827B model, I've got that up to around about

16:26 300 tokens per second. Now there's versions out there that are doing faster than that. If I'm doing things in batches, I can average around about 380 tokens per second out of that. And

16:38 that's certainly been a workhorse there. And I do wonder as I delegate this using pair to other machines for doing inference, is the speed hit just going to slow it down too much at the moment?

16:49 But anyway, go over to the repo, give it a star. It's definitely worth checking out and following this going forward. And feel free to chime in on the issues and pull requests for this. and let's

17:00 see where this actually goes in the future. I'm also interested in testing out other routers. If you're using anything else that you think is really good, please let me know in the comments

17:09 cuz I'm definitely looking for what is the best out there with this kind of thing. Anyway, if you found the video useful, please click like and subscribe and I will talk to you in the next

17:18 video. Bye for now.