NVIDIA Boosts DeepSeek-V3 Training Speed by 10x
NVIDIA integrated its Transformer Engine with JAX to achieve a 10.4x throughput boost training DeepSeek-V3, overcoming a major communication bottleneck in dropless MoE models.
NVIDIA has announced a major optimization suite for training Mixture of Experts (MoE) models using its Transformer Engine library alongside the JAX framework. By targeting the inherent inefficiencies of "dropless" MoE architectures, the company achieved a 10.4x throughput improvement when training the 671-billion-parameter DeepSeek-V3 model on NVIDIA GB200 hardware. The optimized stack raised performance from an unoptimized baseline of just 103 TFLOPS per GPU to 1,068 TFLOPS per GPU. Furthermore, the system demonstrated a 97 percent scaling efficiency across 1,024 GPUs on NVIDIA GB300 NVL72 hardware.
In dropless MoE models, every token is processed by its selected expert to preserve model quality, avoiding the data loss or wasted compute associated with capacity-based MoE padding. However, this creates highly irregular "ragged" tensors because different experts receive varying numbers of tokens. An unoptimized setup suffers from severe communication bottlenecks, with inter-GPU communication consuming up to 84 percent of accumulated kernel time. NVIDIA resolved this by utilizing grouped GEMM kernels backed by cuBLAS and cuBLASLt, which process variable-length token groups in a single call. This approach ensures high Tensor Core utilization without requiring device-to-host copies that disrupt CUDA graphs.
The training stack also integrates expert parallelism operations through NCCL EP, a communication backend that fuses the dispatch and combine stages while deduplicating tokens to minimize network traffic. To further reduce memory bottlenecks, the system leverages JAX host offloading to shift query and value projection activations to host memory. Additionally, XLA multistreaming collectives allow the compiler to run independent communication tasks concurrently across separate CUDA streams, overlapping InfiniBand and NVLink transfers.
For AI practitioners, these advancements make training massive, state-of-the-art MoE models significantly more viable and cost-effective at scale. Developers can access these optimizations directly through the NVIDIA NGC MaxText container with the Transformer Engine enabled. Looking forward, NVIDIA plans to introduce further enhancements, including NVFP4 support, quantization fused with GEMM, and all-to-all communication overlap.
This is our own summary of reporting by NVIDIA Developer Blog
