Meta Cuts GPU Interruption Rates 99% on Prometheus
Meta released technical details for Prometheus, its Ohio supercluster scaling toward one million GPUs, showing a 99% drop in daily training disruptions for foundation models like Muse Wonder.
Meta published an architectural account of Prometheus, a gigawatt-scale infrastructure in New Albany, Ohio, expanding toward roughly one million distributed GPUs. Phase one connects hundreds of thousands of GPUs across multiple facilities over 100 km using a multi-petabit Ethernet Backend Aggregation (BAG) network. While training its Muse foundation models, including Muse Wonder, Meta hit over 85% goodput during pretraining and above 90% during reinforcement learning. Mean time between interruptions hit 47,000 GPU-hours, later rising to 54,000 after firmware updates.
Daily disruptions dropped 99% from almost 1,000 per 8,000 GPUs in mid-2025 toward a target of two daily interruptions (100,000 GPU-hours). To prevent domain-level rack failures on NVIDIA NVL72 platforms from halting workloads, Meta allocates 64 active GPUs and eight local spares per rack, alongside a 2% whole-rack buffer. A 32,768-GPU workload provisions 523 racks containing 37,656 GPUs, representing a 13% capacity overhead. Hardware deployments include Catalina (GB200 NVL72) with 800 Gb/s per GPU networking, Clemente (GB300 NVL72) with ConnectX-8 SuperNICs offering a 5% to 10% performance gain over GB200, and San Miguel (Vera Rubin) promising a 10% to 20% performance-per-watt boost over GB300.
Prometheus manages inter-building fabrics using 51.2 Tb/s shallow-buffer Ethernet switches running FBOSS software and Arista 7800 chassis with Jericho3 silicon, deploying 800G-ZR optics beyond 10 km and 2x400G FR4 LITE modules for shorter runs. Individual building fabrics support up to 20,736 accelerators. Operationally, Meta distributes firmware to over 100,000 GPUs in hours and reduced 90th-percentile pod upgrade times from over seven hours to two. To counter 30% to 40% electrical load fluctuations, software called Dynamo Dimmer throttles GPU power limits alongside facility batteries.
For AI engineering teams, Meta's blueprint shows that ultra-large training depends heavily on architectural fault isolation, localized spare pooling, and power management rather than raw compute counts. Meta plans to mirror this operational framework at Hyperion, its next supercluster in Richland Parish, Louisiana.
This is our own summary of reporting by AlphaSignal
