NVIDIA Releases NeMo Switchyard to Route AI Agent Tasks

NVIDIA has launched NeMo Switchyard, an open-source, provider-agnostic SDK designed to optimize AI agent workflows by dynamically routing tasks to the most appropriate models. Rather than relying on a single expensive frontier model for every query, the framework evaluates incoming requests based on model capabilities, cost profiles, and real-time infrastructure signals. This allows developers to build systems that balance performance and budget by shifting workloads between specialized and frontier models.

The toolkit, powered by the switchyard-libsy library, supports both tuning-free and tunable routing algorithms. Tuning-free options include an LLM classifier that acts as a judge, a stage router that tracks tool activity to gauge required capability, and an escalation router that starts with cheaper models and scales up when errors occur. For more advanced setups, tunable routers analyze the residual stream of a model's prefill state to predict the likelihood of a model successfully completing a task, allowing practitioners to blend accuracy predictions with latency and cost constraints.

Early implementations demonstrate significant efficiency gains. In a benchmark conducted by LangChain using its deep agents evaluation suite of 145 multi-turn tasks, routing between NVIDIA Nemotron 3.5 Lightning and Claude Opus 4.8 with the escalation router achieved a 74 percent cost reduction over five runs. It sent only 7 percent of calls to the frontier model while accepting a minor six-point accuracy tradeoff. Similarly, Cognition integrated the technology into Devin Desktop, routing between Opus 5 and Kimi K2.7 on the FrontierCode Main benchmark. This setup achieved 50.6 percent accuracy at a mean cost of 3.11 dollars, coming within 2.8 percentage points of pure Opus 5 accuracy while cutting costs by approximately 28 percent.

For AI practitioners, NeMo Switchyard decouples routing logic from specific model providers, meaning teams can swap endpoints or update models without rewriting application code. NVIDIA is collaborating with partners like Nous Research for Hermes Agent routing, Ramp for financial engineering, and integrations with LiteLLM, Kong, Classmethod, Boomi, Cadence, and Siemens. The system also supports benchmarks like Terminal-Bench Hard with the Terminus agent—where models like DeepSeek V4, Kimi K2.6, and Qwen3.5 397B A17B excel at different sub-tasks—as well as evaluations on Pinchbench and ClawdQA.