Microsoft Joins the Local AI Push

0:00 Okay, so yesterday Microsoft held their Windows and Surface event in San Francisco and this video I want to talk about some of the interesting things that they showed there. So they had an

0:09 interesting section where they showed 1.6 million tokens. That was how much one coding agent session sent into a model during one of the demos and they made it clear that the bill for that was

0:20 zero. Not because Microsoft was picking up the tab or anything like that, but because the model was running on the laptop itself. And I got to say that it didn't look like a toy demo either. The

0:30 agent was investigating some GitHub issues on the Windows terminal code base. And maybe even the most interesting part here was that they claimed that it wasn't actually even the

0:39 developer who decided it should run locally, it was the tool. And while most of the coverage that I've seen about this event really focused on the Surface Laptop Ultra and the Nvidia RTX Spark

0:50 chip here, both of those looked really cool, although definitely a steep price tag there. A lot of that was actually announced back in June. Really the big thing here I guess is the shipping

1:00 dates, the pricing, that kind of thing. But for me, the interesting thing here was what is Windows actually going to do with this powerful processor? And it seems one of the key things that it's

1:11 going to do is decide which of your tokens run on your machine and which of them go to the cloud. So Microsoft is calling this hybrid intelligence. They had the person who leads the Windows

1:22 team actually described it as intelligent routing. basically the models, a runtime, and the chipset all working together. So, let's start with the router because that's the piece that

1:32 actually made the demo work. So, in the GitHub Copilot demo, the presenter had the model picker set to auto. And the new version of auto doesn't just choose between cloud models anymore. It can

1:44 also choose a local model. So when she gave it a pretty basic sort of cleanup task on the Windows terminal code base, it actually sent that to the Microsoft AI code flash model that was running on

1:56 the laptop and you could actually see the GPU ramping up on the screen. She even pointed out that you could put the whole machine into airplane mode and it would still keep going. After that

2:07 though, she then gave it something harder to go through all the issues that were tagged needs repo. When she did that, what happened there was a whole different thing. So basically a cloud

2:17 model. It was one of the open AI models GPG6 or something did the planning then three sub aents ran locally on that Microsoft AI code model. So this is a very standard pattern we've seen right

2:29 the expensive model does the thinking the local cheap models do the bulk reading and that's where we get to the actual 1.6 6 million tokens coming in here, right? The majority of those

2:40 tokens, in fact, I think all but maybe like 10,000 or so were running locally. And that's where the 1.6 million tokens come from. 1.6 million in and a bit over 10,000 out. That's roughly 150 to one.

2:55 And more and more this is the kind of pattern that you see if you're sort of building things with multiple agents where the context window keeps getting longer and you keep sending it through

3:04 to generate something small out. And I thought this this was really interesting to see basically how they handled this routing between the local model and the cloud model because I do think that this

3:15 is something we're going to see happen more and more going forward. While it's not in the interest of companies like OpenAI or Anthropic to really talk about that, it is in the interest of any of

3:26 the hardware makers and even companies like Microsoft who are making some of the software and the hardware, not just the actual models. So, this is going to ship in the GitHub Copilot app on

3:36 October 15th. And it seems that that app is getting both local context, local actions, local models over the next few months. And it's really going to be something that's very interesting to

3:47 sort of look out for because we know that every router lives or dies by what happens when it gets it wrong. If the local model can't handle the task it was sent, does it then know that and send it

3:57 back to the cloud or do you just get a worse answer and never actually know what failed and what didn't work out the way you wanted? Unfortunately, that wasn't something that they showed here,

4:07 but it's certainly going to be interesting to see when we've seen things like the whole model router idea with things like pair from Nvidia through to other projects and ideas that

4:17 people have put out there of how this is actually going to work. All right, so what did they actually say runs locally? So, first off, this was the Microsoft AI code 1.1 flash model. This is the coding

4:29 model that Microsoft introduced at the build conference. And the thing that kind of amazes me here is that they've taken it down to three bits, which has cut the size by close to 80%. And

4:42 they've got it running on these devices with a 256k context window fully local in there. Now, I got to say that I'm on the fence a lot about a 3bit quantized version of a model, especially when

4:55 you're trying to use it for something like coding. I think time's going to tell. either they've worked out something really good that they're actually doing there or this is going to

5:03 be calling the cloud for any sort of coding task that's remotely difficult. And that same idea carries into the second model. Right? So they talked about a new coming Neotron model that

5:14 runs at two bits in about 20 gigs of memory. And interestingly, they're pitching that as a good fit for the RTX Spark. So don't forget the RTX Spark now is not just going to come in 128 gig

5:27 version. We've already seen the DGX Sparks include a 64 gig version. And it seems that the RTX Spark might even have variations that go down to a 24 gig version. Now, the third one that they

5:39 showed, and I got to say I'm a bit suspicious about this one. I really hope I'm proven wrong, but this was them running DeepS 4 locally at 1.6 bits in about 60 gigs of memory. The line that

5:52 they used on stage was something along the lines of that a year ago this capability was only available from the Frontier Cloud models. And for most people that is kind of true. Now you can

6:02 run DeepSync 4.1 on multiple DGXs. I actually do that. It's one of my favorite models that I really like. But it's going to be really interesting to see what a 1.6 bit version of that model

6:14 actually is like. Now to be fair, they're actually saying that the 1.6 bits is an average. So in practice that means some layers are going to be kept at higher precision and a lot of them

6:25 are going to be pushed very low. So you try and keep the layers that are the most important at higher precision levels and then layers that don't get activated as much you then go for lower

6:34 precision on those. The real question is though how much of that quality actually survives. So the claim on stage was that it outperforms on coding and reasoning benchmarks but they didn't really

6:45 clarify what that meant. And I would say that there are really sort of three key things here. One, does the coding pass rate hold up against a higher precision version of the same model? I'm very

6:56 suspicious of whether that actually does. Secondly, does it fall apart at long context? And then thirdly, if you've got sort of complicated use of tools, does it actually still produce

7:08 clean tool calls? Because structured output is often the first thing to break when you push quantization too far. And for any sort of agentic task, a broken tool call is generally going to be a

7:19 failed task. The other thing you got to remember too is that if you're really going for a sort of long context like 256k, that's not going to be free, right? At

7:29 that length, your KV cache takes up a lot of memory on top of the actual weight. So there's a lot of nice tricks of where people quantize that KV cache, whether it's down to 8 bit, 4bit,

7:40 perhaps even lower than that now. But my guess is you're probably going to need one of the bigger versions of this machine with a large amount of memory to be able to do that. Now, another

7:50 interesting point just to raise here that I haven't really talked about on the channel before is that it's a very interesting debate of do you go for a bigger model with a lower quantization

8:01 rate or do you go for a higher quantization rate with a smaller model. Now, personally, I've been seeing as long as your quantizations don't go below 4bit, actually going for the

8:12 bigger model usually gives you a better win, but that's mostly me saying that anecdotally. I can't recall off the top of my head any sort of papers or anything that have been talking about

8:21 that. All right, so to be able to run these models on the machine, you're going to need a runtime. And one of the announcements that they had, I think is just a clear win here, was the whole

8:31 idea of Llama CPP inside of Windows ML. So, Windows ML is Microsoft's runtime for deploying models across GPUs, NPUs, CPUs. And the cool thing here is that they're bringing Llama CPP into it. So,

8:45 why does that matter? Basically, it's going to allow for a lot of compatibility. It means open models on day one. So, when a new model drops and there's a GGUF quant that shows up a few

8:55 hours later, you're not having to wait for Microsoft to package it. And that's especially true if you're not just using the normal versions. Perhaps you're using an uncensored version, a

9:06 specifically fine-tuned version, which Microsoft is never going to convert for their sort of system. So having Llama CPP in there now gives you that ability to just pick the models that you want to

9:17 run and actually run them. The other cool thing here is that if you are building a Windows app that ships with a local model, you've now got this one runtime that handles whatever the

9:28 hardware your user happens to have. I'm guessing as long as that hardware probably is reasonably recent. The thing I don't know yet and I'll be looking into is how you can actually plug your

9:38 own local model into the co-pilot router here or whether it's only the Microsoft pick. I got to say if you could plug in any model there, this could be a huge win for Microsoft in that you could just

9:51 basically decide, oh, I want Quen 4 35B mixture of expert model plugged into my OS and it just gets automatically used by a whole bunch of different apps or something. All right, the next thing up

10:04 was once you've got capable local models here is obviously how do you stop them from doing things that you don't want them to do? And in many ways what Microsoft announced here was their

10:15 version of Nvidia's OpenShell or their version of Docker sandboxes etc. And this is called the Microsoft execution containers MXC. It's basically a runtime sandbox. Organizations can set policies

10:29 for what an agent can access and then Windows itself enforces them while the agent runs. The bit that I think here is really quite interesting is the whole thing around agent identity. every

10:40 action the agent takes is attributed to the agent, not to you. And so you can imagine that this helps both in sort of tracing things, but also when you're actually working on something, if the

10:52 agent actually goes to do something, you can actually block it and then it loses access to take that action, but you as the logged in user still can have access. All right, lastly up, I guess,

11:03 was the hardware itself. And I got to say Microsoft's done a really nice job at design here. These look really impressive. And don't forget, these are just the first of the RTX Sparks that

11:13 we're seeing coming out. These machines are up to 128 gigs of unified memory. They have full CUDA support, and it's all running on a Windows laptop. You don't have to go into Linux to actually

11:25 do this. So, reading some of the coverage, I think in gadget puts this GPU at around about an RTX 5070 level with 20 CPU cores, and it won't draw more than 80 W. Now, the prices for this

11:38 started, I think, around about $2,600, but obviously the more memory that you put into this, the more that price goes up. And my guess is there's really nothing that Microsoft or Nvidia can

11:48 really do to prevent that. The whole memory and RAM issues that are going on at the moment is just making all of this hardware get more and more expensive. Along with the laptop version, they've

11:59 also got a desktop tier in there. And interestingly, they even teased a Windows version of the DGX station coming later on this year. So, that's going to be interesting to see. I think

12:09 more for unlocking the ability for other people to be able to actually use these. So, hopefully at some point I will get my hands on an RTX Spark and be able to run some tests on it myself and see just

12:20 how these things perform and work out for which group of people is this going to be a viable alternative to a GGX Spark, etc. Anyway, so to pull it all together, I thought this was actually a

12:30 more interesting than usual event from Microsoft because it really planted a flag of them saying that you're going to need to be able to run models locally and not just use the cloud. And I think

12:41 by them getting behind it really is an important step in the direction that local AI is going. So no longer do we just have sort of enthusiasts or ML people or even just startups sort of

12:53 pushing for local AI now. And while we've had Nvidia clearly pushing for local AI this year, having Microsoft join that bandwagon is definitely a step forward in making all of this the norm

13:04 rather than the exception. So anyway, let me know in the comments what you think. What's your take on the RTX Spark? How do you feel about Windows adopting Llama CPP rather than just

13:14 making their own protocol or their own way to do it? I'm really interested to hear from people about this one. I do think Microsoft's gone in an interesting direction here. Anyway, as always, if

13:24 you enjoyed the video, please click like and subscribe, and I will talk to you in the next video.