NVIDIA Switchyard cuts agent costs by 74 percent
A LangChain evaluation of NVIDIA's open-source Switchyard router reveals that smart model routing can slash agent costs by 74 percent while retaining 93 percent of frontier model accuracy.
LangChain recently benchmarked NVIDIA's open-source model routing library, NeMo Switchyard, to determine if AI agents can offload tasks to cheaper models without sacrificing performance. Using their Deep Agents evaluation suite—which consists of 145 multi-step tasks averaging 6.3 calls each—researchers routed queries between the 30-billion-parameter NVIDIA Nemotron 3.5 Lightning and Anthropic's Claude Opus 4.8. The system used an LLM classifier in escalation mode, where a small judge model evaluates turns and escalates a task to the frontier model only after two consecutive negative verdicts.
The evaluation proved that the vast majority of agent queries do not require expensive frontier models. Nemotron 3.5 Lightning successfully handled 93 percent of the model calls, while Claude Opus 4.8 was called for just 7 percent of the turns. This routing strategy reduced overall costs by 74 percent compared to running Opus alone, while retaining 93 percent of the frontier model's accuracy. Specifically, Nemotron alone scored 77.7 percent accuracy for $0.72 per run, Opus alone scored 86.0 percent for $11.45, and the routed setup achieved 80.0 percent accuracy for $3.00.
While the routed approach saved money, it introduced unique cost dynamics. Nemotron accounted for 10.4 percent of the spend, while Opus consumed 68.4 percent despite its low call volume. The judge model, which acts as a constant overhead cost on every unescalated turn, accounted for the remaining 21.2 percent of the bill. This is because an individual Opus call costs $0.0324 compared to Nemotron's $0.00037, representing an 87-fold price difference.
For practitioners, these findings show that routing is highly effective but depends heavily on the price gap between models. LangChain outlines a formula where the minimum offload required to break even equals the judge's cost divided by the price difference between the expensive and cheap models. If the models are priced too closely, routing will not be cost-effective unless developers host the cheaper model locally. Ultimately, routing is best suited for teams that require frontier-level capabilities for unpredictable, complex tasks but want to avoid paying premium rates for trivial steps.
This is our own summary of reporting by LangChain Blog
