Halley: A Predictive Control Plane for Optical AI Data Center Networks

AI summary

Overview

• Research area: Optical data center networking for AI training (cs.NI) — co-packaged optics (CPO), optical circuit switching, laser pool provisioning, microring thermal control, and network control planes, with a formal reliability analysis component.
• Technical level: Advanced. The paper combines stochastic reliability theory (de Finetti exchangeability, beta-binomial models), scheduling theory, simulation, and FPGA/ASIC circuit design, but each section is written with plain-language analogies.
• Scope (1 sentence): The paper proposes and evaluates Halley, a four-component predictive control plane that forecasts the next state of each optical layer in an AI data center fabric and acts before failures or drift occur, and quantifies the availability ceiling that correlated laser failures impose on pooled spares.

What This Paper Is About

Frontier AI training runs span tens of thousands of accelerators joined by an optical fabric, and every training step ends in a collective that acts as a barrier, so one failed laser, drifted microring, or late switch reconfiguration idles the whole cluster. Today each of the fabric's four optical layers runs a fixed or after-the-fact policy. The paper's goal is to replace those reactive policies with per-layer predictors that carry formal guarantees and that fall back to the deployed policy when unsure, while proving that correlated (common-cause) laser failures cap pool availability no matter how large the pool grows.

Key Contributions

1. Correlation ceiling for pooled lasers. The authors carry the classical exchangeable-failure limit into pooled laser supplies and prove that in a pool of many modules it is the correlation between modules, more than within them, that sets the availability cap; under a stated cost model they find that decorrelating the pool is cheaper than buying spares across the range of correlations considered plausible.
2. The Halley control plane. Four components, three with formal guarantees: LightReserve (sizes the laser pool to a coverage bound), FoldFit (schedules the OCS within 11/9 of the optimal total dwell time, a bound that excludes switching delay), ThermaLead (one-step-ahead microring heater control), and TriageGuard (steers a limited budget of hot spares to the highest-risk modules).
3. Composition into cluster goodput. The components are composed into effective cluster goodput under two readings of a spare shortfall, with laser and thermal losses coupled through a shared common cause, tail as well as mean reported, and a closed form proved for the correlation-driven part of the tail.
4. Circuit-level realization. Fallback circuits are synthesized, placed, and routed on a commercial FPGA, carried through SkyWater 130 nm signoff, and formally verified at the register-transfer level — the part of Halley buildable today. Software and hardware are released (reference [13] in the paper).

Main Findings

• Availability ceiling binds at the budget cap: At the assumed stress point (per-laser per-window failure probability λ = 0.05, correlation ρ = 0.15) and the 2× pool budget cap used throughout, availability cannot pass 99.670% however large the pool grows.
• The four-laser reserve is far from enough: A reserve of four spares for 32 channels — the proportion an analyst reports for a related shipping part — reaches only 86.3%; a large pool at that 1/9 spare ratio approaches 85.1%.
• 99.999% is unreachable within the cap above a threshold: For a 32-channel pool at ρ = 0.15, 99.999% stays out of reach within the 2× cap once a laser's per-window failure probability exceeds about 2.4 × 10⁻⁴, which at 50 FIT (the OIF target applied per laser) occurs once the restock window exceeds about 200 days (about 40 days at 250 FIT).
• Decorrelation is the cheaper route: Holding spares at the 23 lasers the sizing rule returns for a 99% target, cutting ρ from 0.15 to 0.03 moves availability from about 99% to just past 99.999%; reaching 99.999% by separating the pool first and then sizing spares costs $44,100 in total versus $234,000 for spares alone. A 2× pool's ceiling rises above 99.999% once ρ falls below about 0.056.
• Goodput gain, coverage reading: Composing the simulated components lifts effective cluster goodput from 65.2% to 76.5% (a 17.3% relative gain) when any window that exhausts its spares is counted as lost. Holding the larger pool to the 2× cap costs only 0.2 points.
• Goodput gain, lit-share reading: If only the dark channels' bandwidth is lost, goodput rises from the same 65.2% baseline to just 67.4% — so most of the gain comes from the provisioning decision (the larger pool LightReserve sizes for correlation a static reserve ignores).
• ThermaLead's value is in gating: Its simple forecast does slightly worse than the deployed dither lock on the stress trace; gated to fall back to that lock whenever its recent error is the larger of the two, it has 2.1% fewer lock losses.
• Gain shrinks with better lasers: At 50 FIT and a 90-day restock the fixed reserve already covers nearly every window, so the composed relative gain falls to about 1.3%; with yearly restocks 99.999% still exceeds the 2× cap at 50 FIT.
• Capacity and power impact: At the stress bound, the recovered goodput means up to 14.7% fewer new megawatts to site and connect for a given training target, or up to 1.17 times as much training from power already in place; on the lit-share reading the figures are 3.3% and 1.03 times. Energy per unit of training falls by less, since a waiting accelerator draws less than full power.
• Variance floor: Under correlation the variance of the failure fraction flattens at λ(1−λ)ρ = 0.0071 instead of shrinking as 1/N, so the failure fraction keeps a standard deviation of 0.084 however large the pool grows.
• Hardware feasibility: The fallback circuits achieve close timing on a commercial FPGA and pass a 130 nm signoff, with formally verified RTL.

Methodology in Plain English

The authors first model laser failures as "exchangeable" — no laser is special — with a shared, window-varying severity level drawn from a beta distribution, so that a single number, ρ, captures how much failures cluster. They then prove what happens to availability as the pool grows, and instantiate the model at an explicitly stated stress operating point, sweeping ρ because no field data yet pins it down. Each control-plane component is designed around a known physics of its layer: a coverage (chance the spares absorb every failure in a window) target for laser spares, a worst-case ratio for switch dwell time, one-step-ahead heater control driven by temperature telemetry for ring drift, and risk-ranked hot spares for the modules most likely to fail. They then compose simulated component outcomes into a cluster-level figure of merit (effective goodput) under two readings of what a spare shortfall costs, report tail as well as mean, and couple laser and thermal losses through a shared common cause. Finally, they build the fallback logic in hardware — synthesis, place-and-route on an FPGA, SkyWater 130 nm signoff, and formal RTL verification. The key intellectual move is using the physical grouping of lasers (they share an enclosure, cooling, and often one power supply) to show that redundancy cannot diversify a shared-cause risk, then pricing the alternative.

Why This Matters

Impact on research: The paper turns a classical result about exchangeable failures into an actionable design constraint for pooled laser supplies, and it argues that existing optical control planes (Google's Jupiter, TPUv4, Ironwood's 9,216-chip superpods, MixNet, Opus) forecast traffic and parallelism phases but not the co-packaged hardware itself. It gives the reliability and networking communities a stated ceiling, a threshold rule for when that ceiling binds, and a cost comparison between spares and decorrelation.

Real-world applications:

• Sizing and restocking laser pools in co-packaged-optics switches, where the paper's rule replaces a fixed four-spare reserve with a coverage-bound calculation and a restock-window threshold.
• Power and siting decisions for new AI data centers, since recovered goodput translates into fewer new megawatts to site and connect — relevant given that a new grid connection can take five to ten years and around 45 US data center projects worth $68 billion were blocked or delayed between April and June 2026.
• Optical circuit switch scheduling for mixture-of-experts all-to-all traffic, where FoldFit's dwell-time bound applies and where prior heuristics state no worst-case bound.
• Microring thermal control in packages where MoE compute bursts heat the interconnect by several kelvin within milliseconds and stall rings unless the heater is pre-driven.
• Built-today logic: the FPGA-verified, 130 nm-signed-off fallback circuits give a deployable path even before the predictive parts are trusted.

Industry relevance: The analysis touches concrete shipping and announced parts — NVIDIA's Quantum-X and Spectrum-X packages, Broadcom's Tomahawk 6-Davisson (102.4 Tb/s of CPO), Lightmatter's Guide light engine, and Lumentum's eight-wavelength DWDM ELSFP with initial availability in the first half of 2027 — and the paper notes that no published vendor pool-sizing rule with a coverage guarantee was found.

Future Directions

• Measure ρ in the field. The correlation value of 0.15 is an assumption, not a measurement; no field data for these laser form factors pins it down, and the paper describes the experiment that would.
• Give confident-but-wrong forecasts a floor. Each component steps aside when unsure, so the result is no worse than current systems for a stale forecast — but a confident yet wrong forecast has no such floor yet.
• Quantify fast recovery. Fast recovery through pre-wired hot spares narrows the gap to 99.999% "by an amount we have not computed," and it only shortens failures on pre-spared modules (4 of 18 modules see about 22% of failures in an exhausted window under exchangeable failures).
• Validate at better laser and restock regimes. At 50 FIT and a 90-day restock the fixed reserve already covers nearly every window and the gain falls to about 1.3%, so the case rests on longer windows and stress rates; lower-FIT hardware and confirmation of realistic windows would test the framework's value.

Target Audience

Advanced readers will get the most from this: networking and systems researchers working on AI data center fabrics, optical interconnect and CPO architects, reliability engineers concerned with common-cause failure and sparing policy, and hardware engineers interested in the FPGA/130 nm realization. Data center capacity and power planners will find the goodput-to-megawatt translation useful, and the paper's plain-language framing of the correlation ceiling makes Sections III and IV accessible to graduate students entering the area.

Authors’ abstract

A frontier AI model is trained in an AI data center, on tens of thousands of accelerators joined by a dedicated network, and every training step ends with all of them exchanging data at once. Between racks, that network is optical, and each of its four layers, from lasers to switches, runs a fixed or after-the-fact policy. Lasers sharing an enclosure, cooling, and power can fail together. At a fixed spare ratio, correlated failure provably caps a pool's availability however large it grows. In this paper, we present a comprehensive predictive control plane framework, called Halley, that reads each resource's next state and acts before the failure occurs. It has four components, each carrying a formal guarantee: LightReserve that sizes the laser pool to provide a coverage bound, FoldFit that schedules the switch within $11/9$ of the optimal, ThermaLead that holds a microring with one-step-ahead control, and TriageGuard that protects the highest-risk modules first. Halley gives each layer a predictive controller that defers to the deployed policy when unsure. Composed from simulations at that point, effective goodput, the share of ideal throughput kept, rises from 65.2\% to 76.5\% if a window that exhausts its spares is lost. If only the dark channels' bandwidth is lost, it reaches 67.4\%. Per megawatt, that is up to 1.03--1.17 times the training; for the same training, up to 3.3--14.7\% less data center capacity. \textsf{Halley}'s fallback circuits are formally verified and are shown to achieve close timing on a commercial FPGA, and pass a 130~nm signoff.