Google Open-Sources TorchTPU for Cloud TPUs

Google, alongside Meta's PyTorch team, has open-sourced TorchTPU version 0.2.0 under an Apache 2.0 license. The software stack integrates directly into PyTorch via its PrivateUse1 interface, replacing the older PyTorch/XLA library and allowing developers to target hardware using a standard torch.device("tpu") call. Unlike its predecessor, TorchTPU treats tensors as ordinary PyTorch objects without wrappers, supporting eager mode execution, preserved tensor views, in-place mutations, and divergent MPMD execution alongside pure SPMD workflows.

The framework operates in three execution modes: Strict, Debug, and DeferAndFuse, the latter of which fuses eager operators to achieve up to a 50% speedup over Strict mode. For peak throughput, torch.compile(backend="tpu") routes graph generation through TorchDynamo, StableHLO, and OpenXLA. In benchmarking, fine-tuning step times dropped from 11.49 seconds in eager mode down to 2.78 seconds when fully tuned—representing a 3.80x speedup. In large-scale training, TorchTitan hit a self-reported 52% Model Flops Utilization (MFU) on Qwen 3 8B, demonstrating near-linear scaling up to 8,192 Ironwood (TPU7x) chips.

For AI practitioners, TorchTPU bridges the gap between PyTorch usability and TPU hardware efficiency. It natively supports ecosystem tools like TorchTitan—with recipes for models including Llama 3 70B/405B, Qwen 3 32B/235B MoE, and FLUX.1 12B—as well as vLLM-TorchTPU and SGLang-TorchTPU. The stack handles FP8, MXFP4, and NVFP4 checkpoints for inference, as well as speculative decoding methods like MTP and EAGLE-3. Hugging Face Transformers v5 and Diffusers can deploy directly via device_map="tpu", while custom acceleration remains accessible through Pallas, Tokamax, and Helion kernels.