NVIDIA Dynamo-Triton Simplifies Multi-GPU Serving
NVIDIA has updated its Dynamo-Triton framework to support multi-device TensorRT inference, allowing developers to scale demanding generative AI models across multiple GPUs via a single endpoint.
NVIDIA released Dynamo-Triton version 26.07, integrating the multi-device inference capabilities of TensorRT 11.0. This update allows a single model instance to span multiple GPUs using NCCL-backed distributed collectives. Instead of requiring developers to manually coordinate individual GPU ranks, the system exposes a single gRPC endpoint. This abstracts away the underlying multi-GPU infrastructure, making it easier to deploy large models that exceed the memory of a single chip.
To demonstrate the integration, NVIDIA tested the 36-layer denoising transformer of the Cosmos 3 Nano video generation model. Using Ulysses context parallelism, the system distributed 44,160 video tokens across up to eight GPUs. At a context-parallel size of eight, each rank processed 5,520 video tokens outside of attention, while the 2,992-token text path remained replicated. The compiled plan utilized two initial reduce-scatters, 108 all-to-alls, and one final all-gather.
The benchmarks, run on an eight-GPU system producing 1280x720 video at 24 frames per second over 189 frames and 35 denoising steps, showed massive latency improvements. End-to-end generation time fell from 156.595 seconds on a single GPU to 87.999 seconds on two GPUs, 53.093 seconds on four GPUs, and 34.183 seconds on eight GPUs. The mean transformer RPC latency dropped from 146.192 seconds to 23.993 seconds on eight GPUs, representing a 6.09x speedup. The RPC share of total generation time decreased from 93.4 percent to 70.2 percent.
Visual validation confirmed that the distributed outputs met quality thresholds of a mean absolute error under 25 and a peak signal-to-noise ratio above 18 decibels. The two-GPU and four-GPU setups achieved a mean absolute error of 12.759 and a peak signal-to-noise ratio of 21.111 decibels, while the eight-GPU setup recorded a mean absolute error of 16.316 and a peak signal-to-noise ratio of 19.400 decibels. For practitioners, this development simplifies the trade-off between hardware resources and latency. Teams can now accelerate demanding generative workflows without changing their client-side application interfaces or writing custom rank-coordination code.
This is our own summary of reporting by NVIDIA Developer Blog
