AMD Boosts PyTorch FP8 Training on Instinct GPUs

AMD has integrated its hardware-specific FP8 training optimizations directly into the upstream PyTorch repositories for TorchAO and TorchTitan. This collaboration with Meta allows developers to achieve high-performance FP8 training on AMD Instinct GPUs without installing proprietary libraries. On dense models like Llama3-8B, running on an eight-GPU AMD Instinct MI300X cluster with a batch size of 1 and sequence length of 8192 over 100 steps, FP8 training achieved a 13.4% throughput improvement over BF16. This speedup maintained a nearly identical peak memory usage of approximately 39 GB, leveraging faster matrix cores rather than memory savings.

The integration also addresses Mixture-of-Experts (MoE) architectures, which typically suffer from high quantization overhead. By implementing fused Triton quantization kernels, the team recovered 89% of the FP8 quantization overhead on DeepSeek-V3 671B MoE shapes. On an eight-GPU MI325X system, this forward-pass kernel fusion boosted end-to-end throughput by 17%, rising from 5,996 to 7,027 tokens per second, close to the 7,156 tokens per second BF16 baseline. Additionally, backward-pass fusions on an eight-GPU MI300X setup running DeepSeek-MoE-16B delivered a 4.2-fold throughput improvement by eliminating redundant transpose copies and fusing multi-step scale-and-cast chains.

Individual kernel optimizations yielded even larger gains. For instance, transposing the output tile through Local Data Share to fix non-coalesced memory writes accelerated the colwise scales kernel by 6.2 times on MI300X GPUs, dropping execution time from 7,290 to 1,170 microseconds per MoE layer. To ensure numerical correctness, the update introduces automatic detection for AMD's unique FNUZ (Finite, No NaN, Unsigned Zero) FP8 format, preventing the activation clipping and gradient corruption that occurred when using standard NVIDIA formats. The software stack also enables grouped GEMM on ROCm via the Composable Kernel backend and switches Triton's atomic operations to relaxed memory ordering to strip unnecessary hardware-level synchronization.

For practitioners, these updates mean that standard PyTorch installations now support optimized FP8 training on AMD hardware out of the box. Developers simply need to upgrade TorchAO and TorchTitan to access these features. Looking ahead, the engineering teams are already developing MXFP8 grouped GEMM and quantization kernels for upcoming AMD Instinct MI355X GPUs.