Switchyard NVIDIA's Local Agent Router

Okay, so there are so many models out there nowadays. And if you're trying to build an agent, you're constantly trying to decide which model do I use for which parts of the agent. So, it's become quite a common thing now where people will have sub-agents that use a quicker, faster model than the orchestrator agent which uses a bigger model. But, taking that to the whole next step is having a router that actually sits there and decides on each call which model to use and takes care of making sure that the calls are in the right format, that they're using the right kind of API, that you're using the right kind of tool request, etc. So, this brings me to the topic of today's video. Today, Invideo is releasing an open-source library that decides per step which model handles a task. So, along with them releasing a model today, which I'll cover in another video, in this video, I want to talk about that router. I want to talk about what the problem is and I want to talk about one of the things that I've been seeing that a lot of the top companies have been doing over the past 6 months, but it's been much harder for many people to run locally or even for the cloud environments, etc. All right, so the problem here is all about static model choice, right? You've got long-running agents. They're often going to take hundreds of steps to basically get from the sort of initial task through planning, through to the final results. And those steps are going to vary widely in difficulty, right? You've got things like retrieval in there. You've got summarization. You've got things like simple entity extraction in there. You've got classification. And those things don't need a frontier model. You're kind of crazy if you're wasting tokens on something like Fable or even GPT 5.6 soul with a model that's costing you a lot when you just want it to do a very simple task. And on the other hand, you've got things that are complex. You've got things like planning, orchestration, which take a lot to actually do. So, as the agent state keeps changing, you've got tool results coming back, errors appear, an orchestrator suddenly spawns a whole bunch of subtasks that need to be done. Really, what you want is a model or a choice of models that actually adapt to that. So, the whole challenge here is about routing each step in an agent trajectory to the right model at the right time. And what In video is actually releasing here, they're claiming can get you 50% faster responses and 25% better token efficiency. So, when they reached out to sponsor the compute for this, I was totally open to see what they're actually doing. All right, so that brings us to what they're actually releasing. And this is Switchboard. And they're claiming that this basically, with combining open and proprietary models, can get you 50% faster responses and 25% better token efficiency. So, this is an open-source model routing library here. And what it actually lets you do is sit between your agent and the models, and then it decides which model gets used for which call from your agent. So, it's important to understand that Switchboard itself is not a model. This is a library, right? This is not In video trying to force you to use a particular model for this. This is just a regular library that you can download from GitHub and you can use yourself. Now, at its core, it's got an SDK inside there that allows you to basically swap cloud providers very quickly. One of the cool things that I like about this a lot is that we know now there are lots of different formats for sending data to the various providers and getting data back from the various providers. You've got things like the traditional OpenAI endpoint, which pretty much everyone supports, you know, on things like open router and stuff like that. But you've also got the Anthropic style endpoints, which a number of companies like Kimmy and I think some others started to support as well. And then you've got things like the newer Open AI responses API endpoints. And this can automatically out of the box translate what's going into an endpoint like that and what's coming back from an endpoint like that. So if you wanted to use the response API format you're formatting all your code for that perhaps because you've been using Open AI, but now you want to try using Open Router. This will actually be able to convert the responses API into a format that Open Router can then use for all the open models that it it's got access to. So that alone is a huge win here. On top of this, obviously, you can use the models that they've got or you can use your own models and rules to decide how that routing is going to be done on a per request basis. Another cool thing that this has got is built-in observability. So you can get your logs from a selected model, you can get the decision rationale of why it decided to route to a particular model, you can get token usage, latency, outcomes, all from being able to inspect what's actually going through this router here. All right, so let's look at the routing algorithms themselves. There are sort of two families here, the tuning free and the tunable ones. So the tuning free ones are things like LM classifier, LM as a judge. It basically picks the model and then you stay with that model for the rest of that session. So in a case like this, you're not sort of reclassifying for every single turn. And that allows you to take advantage of things like caching and other things that get you perhaps price discounts, etc. So this is really good for domain routing. If you know, for example, like, "Okay, I've got my homies agent or got my particular agent and some of the calls that are going to make are actually coding related things. Some of them are going to be looking up news kind of things or general writing kind of things. They're very different tasks and it would actually make sense for having them to be different models etc. The second tuning freeway is a stage router. So this is built more for sort of coding agents and this estimates sort of what stage the run is from the tool results and stuff like that. So if you've got errors, if you've got things going into loops or something, it can then start to spot some of these things and escalate them to basically pushing it to perhaps a better model for doing this kind of thing. And if it sees that if you're grinding on a particular kind of call, it can then push you towards a more efficient model as well. The third tuning freeway is the escalation router itself. So this basically starts every conversation on a cheap model and then an LLM judge watches how the run is going and once it sees problems, it then basically connects to a much stronger model. So one of the key distinctions is that the LLM classifier that I talked about before predicts the difficulty before running the actual prompt. The escalation is used later on to judge whether the run is actually going well. All right, the second family of routing algorithms are the tunable ones. So an example of this is the prefill router. So this is a learned model that reads the prefill stage signals and then its output is not actually answering any of those things. It's output is just predicting the model success. So it's not actually generating answers. It's basically got a policy that blends that predicted accuracy with things like cost and latency. Now videos actually got a whole paper around this particular approach. All right, so at this point probably the main question you're going to have is hey, is this actually going to save me money? Is it actually going to give me better outputs? And I should point out here that this is actually an open version of what we've seen other companies do already. So you've got things like Model Fusion from Open Router where it basically has its own way to decide what models. We've had things like Fugu from Sakana, which is really combining a bunch of different LLMs in a pool and then it's basically just being a routing system that selects that. So we do know these things kind of work. It can just often be a pain to actually set them up if you want to use it locally and if you want to pick what models it's basically going to have access to and what models it's not going to have access to. All right, so some of the key takeaways here. I think this is really sort of showing that the system of models pattern is where production agents are heading here, right? This is sort of core infra for actually doing this. And NVIDIA's framing itself is that, you know, orchestration is here to stay in here. So the repo itself has some introductory routing examples at launch. You can create, test, and contribute your own algorithms in there. It would be great to sort of see if people start sharing what are their best ways to actually do a lot of these things. For things like the harnesses and the agent frameworks that NVIDIA's actually partnering with, it seems like for most of those this is all just going to run on the CPU alongside your harness. You're not even going to need any GPU RAM for the library itself. So I've got to say that this is a really cool step from NVIDIA. Routing here can really be a hard engineering problem. You've got lots of decisions to make, you've got state management, you've got the whole thing about when do you use it, when do you not use it. This library basically gives you the infrastructure to get started on doing this. The actual policies for what's going to be routed to what is still going to be up to you. And I do think over time, as some of these patterns become clear, this is something that I may even look at doing more videos about. All right, so if you want to get it, just go and check out GitHub now. There are some links in the description of where you can find out more, their blog posts, etc. I think they've got integrations are rolling out through their partners. And like I said, perhaps in the future we could look at doing a follow-up video on building a custom routing policy hands-on and using the different staged routers in there as well. So, let me know in the comments if you've tried something like this before, and if you're excited about it. This is definitely something I think is a step forward and something that this is going to become standard 6 months a year from now. As we're seeing so many good open models come out, it just doesn't make sense to pay top dollar for every token that you're actually generating to build something. Anyway, as always, if you found the video useful, please click like and subscribe, and I will talk to you in the next video. Bye for now.