PyTorch 2.14 Speeds Up Apple Silicon and GPU Workloads

PyTorch 2.14 has officially launched, integrating 2,995 commits from 487 contributors to upgrade from version 2.13. This release brings substantial performance improvements, particularly for developers working on Apple silicon. The Metal Performance Shaders (MPS) backend now features native kernels that replace older CPU fallbacks. For instance, lu_factor and lu_solve yield performance gains of more than 100x when handling small batched matrices, while Cholesky factorization is accelerated by 1.2x to 2.8x. Additionally, a fresh prefill-attention kernel on macOS 26.2 delivers a 2x to 4x speedup for fp16 and bf16 dtypes on M5 and earlier chips, targeting head dimensions of 64, 96, 128, or 256 and query lengths exceeding 8. The release also resolves an 8.5x slowdown previously affecting single-token F.linear decode operations in fp16 and bf16 formats.

For GPU-heavy workloads, PyTorch 2.14 transitions NVGEMM into an Inductor backend. NVGEMM leverages CuTeDSL-generated CUTLASS kernels, requiring nvidia-cutlass-dsl 4.6.0, and supports fused low-precision NVFP4 paths on NVIDIA Blackwell hardware. It also targets Rubin's sm_107 architecture. Meanwhile, cuBLASLt serves as the default grouped-GEMM backend for fp16 on Blackwell with CUDA 13.2 or later, and Hopper with CUDA 13.3 or later. On AMD RDNA3 GPUs, FlexAttention delivers 2x to 8x latency reductions for sequence lengths in the low hundreds. CUDA TunableOp testing on H100 GPUs showed individual shape performance ranging from 0.66x to 1.57x.

Distributed training receives a major upgrade with the nccl2 backend for the c10d layer, replacing the older torchcomms. This backend introduces fault-tolerant process-group reconfiguration, allowing clusters to rebuild in place after a rank failure rather than restarting the entire job. For compilation, the new ShapesSpec API, accessed via @dynamic_spec, unifies dynamic-shape declarations across torch.compile, torch.export, and make_fx. Additionally, torch.while_loop is now capturable in CUDA Graphs, and torch.switch (available at torch._higher_order_ops.switch) introduces multi-way branching.

Practitioners should note that while eager-mode wheels for Python 3.15 and free-threaded 3.15t are now available, running torch.compile on Python 3.15 will trigger a RuntimeError. On the ecosystem front, torchvision 0.29 is ABI-stable with PyTorch 2.14, guaranteeing compatibility through versions 2.15 and 2.16.