Connecting the Dots: A Machine Learning Ready Dataset for Ionospheric Forecasting Models

AI summary

Overview

Research area: Space weather / heliophysics applied machine learning, specifically data infrastructure for ionospheric forecasting.

Technical level: Intermediate. The paper is a dataset-and-pipeline contribution rather than a new modeling technique; readers benefit from light familiarity with ionospheric concepts (TEC, geomagnetic indices) and standard ML tooling.

Scope: This paper describes an open, machine-learning-ready dataset that aligns multi-source ionospheric and heliospheric observations into one temporally and spatially consistent product, and benchmarks baseline forecasting models trained on it.

What This Paper Is About

Forecasting the ionosphere—the charged upper layer of Earth's atmosphere—is hard because the observations needed to do it come from many instruments with different formats, cadences, coverage, and missing-data conventions. Machine learning could help, but existing data products were not built for ML workflows, so researchers spend most of their time preprocessing instead of modeling. This paper (from the 2025 NASA Heliolab program) assembles those scattered sources into one aligned, queryable dataset, ships the alignment code publicly, and shows that standard spatiotemporal architectures can forecast total electron content (TEC) from it.

Key Contributions

1. A heterogeneous, time-aligned dataset. Eight source families—solar EUV imagery, solar irradiance indices (F10.7 and related), solar wind and interplanetary magnetic field measurements, geomagnetic indices (Kp, Ap, AE, SYM-H, and others), dense JPL global ionospheric maps, sparse GNSS-derived TEC from Madrigal, orbital mechanics features, and quasi-dipole coordinates—are unified into a single modular structure indexed by time. The alignment window is 2010-05-13 to 2024-08-01, chosen to match the SDO Foundation Model.

2. An open, reproducible preprocessing and modeling codebase. The released code standardizes missing values to NaN across sources, forward-fills short gaps up to a configurable maximum rewind time, drops columns with major gaps, resamples to common cadences, and provides PyTorch datasets, dataset-specific normalization schemes, and example training code. Data and code are hosted on Zenodo, AWS S3, and GitHub.

3. A geomagnetic storm event catalog. A simple Kp-based threshold scheme assigns each time interval a NOAA G-level (G0 calm through G5 extreme) combined with event duration in hours, producing labels such as G2H6. The catalog exists to support event-aware train/validation splits and prevent data leakage from the same storm appearing on both sides of a split. The appendix visualizes this as the "Monitoring Event Space-weather TEC Ionospheric Catalog Index" (MESTICI) scale.

4. Baseline forecasting benchmarks. The authors train three IonCast architectures—an LSTM baseline, a Spherical Fourier Neural Operator (SFNO) model, and a GraphCast-inspired model—at 15-minute cadence to predict dense JPL TEC maps, and report that they outperform persistence baselines up to 12-hour lead times.

Main Findings

• No prior resource filled this gap. The authors state that no existing data product aligns both sparse and dense TEC maps with solar and geomagnetic driver data for ML-based ionospheric modeling. This dataset is positioned as the first integrated product of its kind.

• Alignment is nontrivial but tractable. Sources span cadences from 15 seconds (SDO EUV embeddings) to daily (F10.7 and related indices) to yearly (quasi-dipole maps). The pipeline harmonizes these onto standard cadences, with the released version trained at 15-minute resolution.

• Missing-data conventions are a real obstacle. Different providers encode gaps differently—OMNI uses channel-dependent sentinel values, and some OMNI features have years of missing data. The authors normalize all gaps to NaN, remove heavily gapped columns, and forward-fill only short interruptions (rewind time equal to native cadence for most streams; 50 minutes for OMNI).

• The dataset includes unusual, sparse inputs. Alongside dense JPL GIM maps and GNSS network TEC, it incorporates TEC derived from crowdsourced Android smartphone measurements—a deliberately heterogeneous ingredient that broadens what models can learn from.

• Baseline models beat persistence. IonCast architectures produce accurate TEC forecasts up to 12-hour lead times under both quiet and geomagnetically active conditions, demonstrating the dataset is immediately usable for model development.

• Event-aware splitting matters. The storm catalog was created specifically to ensure model validation reflects distinct storm events rather than fragments of the same event, addressing a subtle but significant methodological pitfall in space weather ML.

Methodology in Plain English

The team started by selecting the data sources that physically drive or describe the ionosphere: what the Sun is emitting, what the solar wind is doing, how Earth's magnetic field is responding, and what the resulting electron content actually looks like over the globe. They then picked a common time window and resampled everything onto shared cadences.

To handle gaps, they applied a conservative rule: fill only small holes using the most recent valid value, and only if that value is recent enough (within a "rewind time" that is roughly the source's native sampling interval, or 50 minutes for OMNI). Larger gaps are left as missing rather than filled with stale information. They also computed geometric features from solar and lunar positions and added quasi-dipole coordinates, since the ionosphere is organized by Earth's magnetic field rather than purely by geographic latitude.

Next, they classified the whole timeline into storm and quiet periods using Kp thresholds and durations, so that training and validation sets could be split by event. Finally, they trained three neural network architectures on the aligned data to predict dense TEC maps at 15-minute steps, comparing against a persistence baseline (which simply assumes the future looks like the present). The result is a public dataset, a public codebase, and a reference benchmark.

Why This Matters

Research impact: Standardized, ML-ready datasets remove a major bottleneck in space weather machine learning. They enable fair, apples-to-apples comparisons between models, make results reproducible, and lower the barrier for researchers without deep data-engineering resources. The authors frame this as a step toward digital twins of the ionosphere.

Real-world applications:

• GNSS accuracy and integrity. Ionospheric TEC is the dominant error source for single-frequency GPS/GNSS positioning; better forecasts support corrections used in aviation, surveying, autonomous vehicles, and precision agriculture.
• Aviation safety. The FAA and international bodies already issue ionospheric disturbance advisories for high-frequency communication and navigation; improved lead times could sharpen these warnings.
• Satellite operations. LEO constellations face increased drag during geomagnetic storms, and operators need advance notice for orbit maintenance and collision avoidance.
• Power grids and radio communications. Geomagnetic storms induce currents in long conductors and disrupt HF radio; forecasts at 12+ hours support mitigation planning for utilities and communication providers.

Industry relevance: Commercial space weather providers, aviation authorities, satellite operators, survey and agricultural technology companies, and insurers all depend on ionospheric conditions. The dataset's open license and released code make it accessible to startups and small labs, not just large agencies—and it arrives alongside missions like NASA's TRACERS and ESA's Vigil, which will supply new data streams and demand better forecasting models.

Future Directions

• Extend and refresh the time range. The current product ends in 2024 to match the SDO Foundation Model window; continuing updates and incorporating newer missions (TRACERS, Vigil at L5) would keep the dataset operationally relevant.
• Improve gap handling. Forward-filling is deliberately simple. Learned imputation or gap-aware architectures could extract more signal from the missing portions of the OMNI record and other sparse streams.
• Broaden model evaluation. Only three architectures are benchmarked. Systematic comparison across many model families, plus probabilistic and uncertainty-aware forecasting, would clarify which approaches genuinely help under storm conditions.
• Refine event definition and splitting. The Kp-threshold catalog is coarse; alternative storm indices, regional event definitions, and standardized leakage-free splitting protocols are open questions for the community.
• Push toward operational deployment and digital twins. Moving from benchmark lead times to real-time, continuously updated forecasting services remains the larger goal this infrastructure supports.

Target Audience

This paper is most useful to machine learning researchers entering space weather and heliophysics, who need a ready dataset rather than a preprocessing project. It also serves ionospheric and space physics researchers who want to apply data-driven methods to TEC forecasting, graduate students and postdocs looking for a well-documented starting point, and applied engineers in GNSS, aviation, and satellite operations who need to understand what current forecasting capabilities can and cannot deliver. Operational forecasters and program managers at agencies such as NASA, NOAA, and ESA will find the event catalog and benchmark results directly relevant to planning.

Authors’ abstract

Operational forecasting of the ionosphere remains a critical space weather challenge due to sparse observations, complex coupling across geospatial layers, and a growing need for timely, accurate predictions that support Global Navigation Satellite System (GNSS), communications, aviation safety, as well as satellite operations. As part of the 2025 NASA Heliolab, we present a curated, open-access dataset that integrates diverse ionospheric and heliospheric measurements into a coherent, machine learning-ready structure, designed specifically to support next-generation forecasting models and address gaps in current operational frameworks. Our workflow integrates a large selection of data sources comprising Solar Dynamic Observatory data, solar irradiance indices (F10.7), solar wind parameters (velocity and interplanetary magnetic field), geomagnetic activity indices (Kp, AE, SYM-H), and NASA JPL's Global Ionospheric Maps of Total Electron Content (GIM-TEC). We also implement geospatially sparse data such as the TEC derived from the World-Wide GNSS Receiver Network and crowdsourced Android smartphone measurements. This novel heterogeneous dataset is temporally and spatially aligned into a single, modular data structure that supports both physical and data-driven modeling. Leveraging this dataset, we train and benchmark several spatiotemporal machine learning architectures for forecasting vertical TEC under both quiet and geomagnetically active conditions. This work presents an extensive dataset and modeling pipeline that enables exploration of not only ionospheric dynamics but also broader Sun-Earth interactions, supporting both scientific inquiry and operational forecasting efforts.