Skip to content
AI.info

The Pulse

Meta Details Prometheus, Its Gigawatt-Scale AI Cluster

The first phase of Meta’s Prometheus AI supercluster in New Albany, Ohio, is complete and has trained the company’s Muse models. The company outlines plans to scale the cluster toward one million GPUs and describes its networking, reliability and cooling systems.

Meta Details Prometheus, Its Gigawatt-Scale AI Cluster

The first phase of Prometheus, Meta’s gigawatt-scale AI supercluster in New Albany, Ohio, is complete and has trained the company’s Muse models. The company’s October 19 engineering post describes a facility already running hundreds of thousands of GPUs across connected buildings, with plans to scale toward roughly one million.

Source engineering.fb.com

The site, which Meta says was a conventional data center handling web traffic two years ago, now combines traditional and AI-optimized buildings with rapid-deployment structures. Meta presents Prometheus as a single machine spread across those facilities, built to train larger models and support longer context windows. The account is the company’s own; its performance figures and descriptions are not independently audited in the post.

Muse training put reliability to the test

Muse Wonder pretraining runs achieved more than 85% goodput and reinforcement learning more than 90%, Meta reports. Goodput measures productive training time rather than simply whether equipment is powered on. During those runs, the company says, mean time between interruptions was about 47,000 GPU hours; subsequent firmware and fleet changes raised that figure to more than 54,000 GPU hours.

The post describes interruptions as faults that force a training job to pause and recover, including problems involving GPUs, networks, storage and temperature. Prometheus experienced nearly 1,000 interruption events per day per 8,000 GPUs during its initial phase in mid-2025, Meta says. After a year of engineering work, the company reports a 99% reduction, while setting a goal of two interruptions per 8,000 GPUs per day.

GB200 and GB300 racks share the workload

Prometheus currently uses two NVIDIA platforms, both built around 72-GPU NVL72 systems. The first, Catalina, uses GB200 GPUs in a two-rack configuration. Catalina provides 720 peak petaflops of FP8 training compute and draws about 120 kilowatts across the rack pair, Meta says.

The second platform, Clemente, uses GB300 GPUs and fits the 72-GPU domain into one rack. Meta lists 288 gigabytes of HBM3e memory per GPU, 50% more than the GB200 configuration, and more than one exaflop of peak FP4 compute. Clemente improves performance by 5% to 10% over GB200 NVL72, the company says, while also offering air-assisted or facility-based liquid cooling.

NVIDIA’s Vera Rubin NVL72 systems will arrive in a later phase under the San Miguel platform. Their specifications in the post describe a planned system, not hardware already operating at Prometheus: each rack is listed at more than 170 kilowatts, with fully liquid-cooled compute trays.

Fiber links buildings as far as 100 kilometers apart

Prometheus relies on a dedicated Ethernet back-end network for GPU-to-GPU traffic, separate from the network Meta uses for ordinary data-center services. Building-level fabrics connect as many as 20,736 GPUs apiece. A backend aggregation network links those fabrics across distances from two kilometers to more than 100 kilometers, with multiple petabits per second of bandwidth per regional pair, according to Meta.

The network includes millions of miles of fiber strand, tens of thousands of switches and millions of optical components. Its design mixes two kinds of building-level fabric: an earlier scheduled system and a newer non-scheduled system using 51.2-terabit Ethernet switches. The longer links make training harder, Meta says, because signals cannot travel faster than light; engineering teams adjusted training and communications to account for the delay.

Power swings and mixed cooling systems

Meta describes training workloads that can change the cluster’s power demand by 30% to 40% as GPUs switch between computation and communication. Starting or pausing a major job can shift loads by tens of megawatts within seconds. Meta’s Dynamo Dimmer framework adjusts GPU power limits as the cluster nears its electrical ceiling, supported by batteries at rack and facility levels.

The distributed site uses more than one cooling approach. Existing buildings and rapid-deployment structures use modular air-assisted liquid cooling, while a purpose-built AI data center uses facility liquid cooling. The company says its New Albany arrangements cover the full energy cost of the site and match energy use with new clean and renewable energy.

Meta’s post makes the engineering case for Prometheus through utilization, network reach and recovery time as well as its headline capacity. The first phase has trained Muse models; the next steps are additional GB300 deployments and the planned move to Vera Rubin systems.

Sources

  1. Original source · Blog post Engineering at Meta engineering.fb.com

Companies in this story

Related stories

Explore

More articles