AMD and Meta Boost PyTorch Speed on MI350 GPUs With FlyDSL
AMD and Meta have integrated FlyDSL into PyTorch for AMD MI350 GPUs, achieving up to 3.01x serving speedups by providing low-level hardware control over core transformer operations.
PyTorch users running workloads on AMD Instinct MI355X GPUs with the gfx950 architecture can now leverage FlyDSL through standard operator APIs. The domain-specific language serves as an optional execution backend for dense matrix multiplication, grouped matrix multiplication, scaled low-precision matrix operations in MXFP8 and MXFP4 formats, RMSNorm, and TopK selection. Integrated through TorchInductor for compiled operations, eager mode dispatch, and AOTriton 0.14.2b for scaled dot-product attention, FlyDSL provides explicit layout algebra and hardware thread mapping while preserving standard PyTorch interfaces.
In kernel-level benchmarks, FlyDSL demonstrated strong acceleration over existing ATen and Triton baselines. For 15 BFloat16 dense matrix shapes, it posted a 1.10x geometric-mean speedup, reaching 1.33x for a 64 by 4096 by 4096 matrix. In 17-shape scaled tests, MXFP8 achieved a 1.58x geometric-mean speedup over ATen (peaking at 2.23x at 32 by 4096 by 4096), while MXFP4 hit a 1.68x mean boost (peaking at 2.96x for a 32 by 14336 by 4096 layout). Across 24 grouped matrix configurations, FlyDSL surpassed Triton by a 1.20x geometric mean and ATen by 1.95x. RMSNorm speedups ranged from 1.17x to 3.66x across 22 cases, and TopK selections with small K values yielded a 4.82x geometric-mean gain over ATen with determinism disabled. Updated attention throughput in AOTriton 0.14.2b spanned 0.92x to 2.34x for forward passes and 0.93x to 2.47x for backward passes compared to version 0.13b.
End-to-end inference benchmarking using vLLM on a single MI355X accelerator showed whole-request performance gains across Llama-3.1-8B-Instruct, Qwen3-32B, and Llama-3.3-70B-Instruct models processing 256 input and 512 output tokens. Speedups reached 1.13x for BFloat16, 2.04x for MXFP8, and 3.01x for MXFP4, with Qwen3-32B achieving 1.53x to 3.01x gains in MXFP4 mode. Developers can access these optimizations on gfx950 hardware by installing version 0.3.0 or higher of the flydsl Python package on ROCm builds and enabling FLYDSL in the TorchInductor autotuning configuration.
This is our own summary of reporting by PyTorch Blog
