NVIDIA Nemotron 3.5 Lightning Speeds Up Agent Tasks
NVIDIA has outlined a comprehensive framework for evaluating AI agents based on end-to-end task completion, showcasing its own Nemotron 3.5 Lightning model as a prime example.
Evaluating artificial intelligence has shifted from scoring isolated function calls to measuring complete task execution within stateful environments. According to NVIDIA developers, traditional benchmarks like the Berkeley Function-Calling Leaderboard (BFCL) and HumanEval are no longer sufficient for multi-step agent workflows. While suites like Terminal-Bench 2.0 measure steps-per-turn, comprehensive evaluation requires tracking step-level process scoring alongside end-to-end outcome scoring. This methodology monitors the entire execution trace across a hierarchy of benchmarks, trials, tasks, turns, and steps to determine if an agent actually finishes its assigned work.
NVIDIA demonstrated this rigorous testing framework using its Nemotron 3.5 Lightning model. On the PinchBench benchmark, Nemotron 3.5 Lightning achieved an impressive 86% accuracy rate. Crucially, the model completed 10,000 tasks up to 30% faster than the Qwen3.6 35B model at a comparable accuracy level. This efficiency is vital for enterprise deployments where steps per success and cost per success directly dictate operational viability. The model was also evaluated on GDPval-AA v2, which uses a panel of LLM judges with Elo ratings anchored to a 1,000 human-expert baseline.
To illustrate how these evaluations function in practice, NVIDIA analyzed a public trace from the SWE-bench Verified dataset, specifically task ID pytest-dev__pytest-5262 using trial .2 and turns 0-4. Operating within the OpenHands agent harness, the model navigated a real repository to resolve a bug. While the agent successfully passed the end-to-end check with a score of 1, its step-level score and tool-call precision were both marked at 3/4 due to a single redundant file-viewing step. This granular tracing helps developers pinpoint exactly where a multi-step chain falters, distinguishing minor inefficiencies from total task failures.
For production environments, NVIDIA recommends that enterprises move away from static tests and build domain-specific evaluations using real tickets and APIs, recording success rates across 3-5 trials. Rather than relying on subjective LLM-as-a-judge metrics, organizations should prioritize executable verification that checks actual database updates or system states. Developers can explore these evaluation configurations through NVIDIA's published reproducibility documents, test the model on build.nvidia.com, or deploy it using the NIM guide.
This is our own summary of reporting by NVIDIA Developer Blog
