NVIDIA Jetson Runs Advanced Reasoning Models at the Edge

NVIDIA announced that its Jetson edge computing platforms, including Jetson AGX Orin and Jetson AGX Thor, can now run advanced reasoning and agentic AI models locally. This shift allows developers to deploy complex multi-step reasoning systems without relying on data centers, reducing latency and securing data. The company highlighted two primary models for these workloads: Nemotron 3.5 Lightning and Qwen3.8-27B.

These models use different architectures to balance speed and capability. Nemotron 3.5 Lightning is a mixture-of-experts model with 30 billion total parameters that activates only 3 billion parameters per token, making it highly efficient for response-heavy workflows. In contrast, Qwen3.8-27B is a dense model that activates all 27 billion of its parameters for every token, which is better suited for complex decision-making tasks. For smaller hardware like the Jetson Orin Nano, NVIDIA recommends the Gemma 4 E4B model.

To maximize performance on Jetson, NVIDIA combined NVFP4 quantization with speculative decoding, achieving up to a 6.28x decode throughput speedup compared to standard BF16 precision. NVFP4 quantization reduces memory usage and processing work, while speculative decoding uses a smaller draft model to propose tokens that the main model verifies. The optimal speculative decoding setup varies by model. Nemotron 3.5 Lightning performs best using the DSpark method, while Qwen3.8-27B achieves its highest speeds with DFlash2.

Performance benchmarks show that throughput varies significantly by workload category, such as writing, reasoning, summarization, and retrieval-augmented generation. When optimized, Nemotron 3.5 Lightning with DSpark reached throughputs between 123.01 and 138.02 output tokens per second. Meanwhile, Qwen3.8-27B running with DFlash2 achieved speeds ranging from 27.69 to 34.44 output tokens per second. NVIDIA advises developers to validate these configurations using JetPack 7.2 and vLLM with representative prompts to ensure accuracy is maintained.