CIPHER: Scalable Time Series Analysis for Physical Sciences with Application to Solar Wind Phenomena

AI summary

Overview

Research area: Machine learning for scientific time series analysis, specifically heliophysics and space weather (solar wind physics).

Technical level: Intermediate — assumes familiarity with clustering, time series analysis, and space physics concepts, but the pipeline itself is conceptually accessible.

Scope: The paper introduces CIPHER, a framework that combines symbolic time series compression, density-based clustering, and expert-in-the-loop labeling to classify large volumes of unlabeled physics data, demonstrated on solar wind measurements from the OMNI dataset.

What This Paper Is About

Physics produces enormous quantities of continuous sensor data — decades of solar wind measurements, seismic records, climate observations — but labeling that data requires scarce, expensive, and inconsistent human expertise. CIPHER tackles this bottleneck by automatically grouping similar time series segments into clusters, then asking a domain expert to label only a few representative examples per cluster and propagating those labels across all members. The goal is to make expert-driven classification of massive physics datasets tractable without sacrificing physical validity.

Key Contributions

1. A four-stage labeling pipeline (CIPHER) that chains preprocessing (detrending, smoothing, normalization), iSAX symbolic compression, HDBSCAN density-based clustering, and human-in-the-loop validation with cluster-wide label propagation.

2. A scalable symbolic representation strategy using iSAX with configurable "chunk size" (window length) and "word size" (temporal resolution), enabling indexing and comparison of millions of subsequences that would be computationally intractable at full resolution.

3. A demonstration on solar wind physics showing CIPHER recovers physically meaningful phenomena — coronal mass ejections (CMEs) and stream interaction regions (SIRs) — from OMNI data using fairly simple inputs.

4. A publicly released codebase and configuration set (github.com/spaceml-org/CIPHER) to support reproducibility and reuse in other physical science domains.

Main Findings

• CME recovery from a single parameter: Clustering smoothed and detrended proton density sequences produced a cluster containing the March 11–12, 2021 CME, complete with correctly ordered substructures (forward shock, compressed sheath, magnetic ejecta, trailing solar wind) confirmed by an expert examining multiple OMNI parameters simultaneously.

• Cross-parameter physical consistency: Even when clustering was driven by one primary parameter (flow speed), the resulting clusters showed coherent behavior across proton density and temperature, indicating the clusters correspond to genuine physical categories rather than artifacts.

• Preprocessing reveals hidden structure: Raw solar wind sequences appear too noisy and irregular for a human to group manually, but the preprocessed (smoothed, detrended) versions show narrow confidence intervals and clear shared patterns, demonstrating that the compression and preprocessing steps expose structure invisible in the raw data.

• Sparse labeling goes a long way: A small number of expert annotations on representative windows can be reliably propagated across entire clusters, drastically reducing expert labor compared to labeling each sequence individually.

• Noise handling matters: HDBSCAN's ability to mark low-density points as noise (tunable via min_samples) and an optional re-clustering step for unassigned points helped keep clusters physically coherent.

Methodology in Plain English

CIPHER works in four steps:

1. Clean up the data. Each time series is optionally detrended (removing slow drifts) and smoothed (removing high-frequency noise), then normalized so that comparable events across different time periods can be compared fairly.

2. Compress into symbols. The cleaned series are split into fixed-length windows (e.g., 35 hours) and each window is further divided into smaller segments (word size 8). Each segment is converted into a symbolic letter based on where its average value falls relative to statistical breakpoints. This turns a continuous curve into a short "word" — like a compact fingerprint. This is the iSAX step, and it makes large-scale indexing and comparison fast.

3. Group similar fingerprints. HDBSCAN, a clustering algorithm that groups items by density, is run on the symbolic index. It forms clusters of similar windows and labels sparse outliers as noise. Two key knobs are the minimum cluster size and a sensitivity parameter controlling what counts as noise.

4. Have an expert label a few, then propagate. A domain scientist inspects a handful of representative windows from each cluster — checking multiple physical parameters to confirm the cluster makes physical sense — and assigns a label (e.g., "CME"). That label is then applied to every window in the cluster, producing a systematically labeled dataset at scale.

The experiment used 1-minute OMNI solar wind data covering flow speed, proton density, proton temperature, and magnetic field components. Chunk size was 35 hours, word size 8, minimum cluster size 5, and noise-sensitivity 5.

Why This Matters

Impact on research: Expert annotation is the dominant bottleneck in applying modern machine learning to physics data. CIPHER shows that a small amount of expert time, intelligently targeted at cluster representatives, can produce ordered, physically validated labels across datasets that would otherwise be unlabelable. This shifts labeling from an O(N) human task to an O(number of clusters) human task.

Real-world applications:

• Space weather forecasting: Identifying CMEs and SIRs automatically supports operational alerts for satellite operations, power grid protection, and astronaut safety.

• Seismology: Grouping and classifying seismic waveforms to detect earthquake types, aftershock patterns, or volcanic tremors without labeling every record.

• Plasma physics and fusion research: Classifying diagnostic signals from tokamaks or other plasma experiments where labeled disruption events are rare.

• Climate and ocean science: Identifying recurring patterns (e.g., specific storm types, ENSO phases) in long satellite or buoy records.

Industry relevance: Any sector generating large volumes of unlabeled time series — energy grid monitoring, industrial equipment telemetry, aerospace, defense sensing — faces the same label-scarcity problem. CIPHER's combination of compression, clustering, and targeted human review is a general template that could reduce annotation cost in applied settings, not just scientific ones. The involvement of Intel Labs, Google Cloud, and NVIDIA in the underlying Heliolab program suggests direct industrial interest in scalable scientific ML pipelines.

Future Directions

• Joint multi-parameter clustering: The current work clusters primarily on one parameter at a time and cross-validates with others. Clustering on several parameters simultaneously could catch subtler phenomena that single-parameter approaches miss.

• Automated hyperparameter selection: Chunk size, word size, and HDBSCAN settings currently require manual tuning and sensitivity testing. Automating this would make the pipeline easier to deploy in new domains.

• Streaming and real-time operation: Integrating CIPHER with live data streams would let it flag phenomena as they occur, which is critical for operational space weather forecasting.

• Cross-domain transfer: Testing the framework on seismology, climate, or plasma physics would validate the claim that the approach generalizes beyond heliophysics. Additional data modalities (images, spectra) could also be folded in.

Target Audience

This paper is most useful for:

• Heliophysicists and space weather researchers who need to build labeled datasets from OMNI, Parker Solar Probe, or similar archives.

• Machine learning researchers working on scientific applications, especially those interested in time series clustering, symbolic representations, or human-in-the-loop annotation.

• Data scientists in adjacent physical sciences (seismology, climate, plasma physics) looking for a reusable blueprint for label-scarce classification problems.

• Practitioners building operational monitoring systems who need interpretable, expert-validated classification rather than pure black-box models.

Readers should have basic familiarity with clustering concepts and time series analysis; the space physics content is explained enough to follow, but a heliophysics background helps interpret the CME/SIR validation details.

Authors’ abstract

Labeling or classifying time series is a persistent challenge in the physical sciences, where expert annotations are scarce, costly, and often inconsistent. Yet robust labeling is essential to enable machine learning models for understanding, prediction, and forecasting. We present the \textit{Clustering and Indexation Pipeline with Human Evaluation for Recognition} (CIPHER), a framework designed to accelerate large-scale labeling of complex time series in physics. CIPHER integrates \textit{indexable Symbolic Aggregate approXimation} (iSAX) for interpretable compression and indexing, density-based clustering (HDBSCAN) to group recurring phenomena, and a human-in-the-loop step for efficient expert validation. Representative samples are labeled by domain scientists, and these annotations are propagated across clusters to yield systematic, scalable classifications. We evaluate CIPHER on the task of classifying solar wind phenomena in OMNI data, a central challenge in space weather research, showing that the framework recovers meaningful phenomena such as coronal mass ejections and stream interaction regions. Beyond this case study, CIPHER highlights a general strategy for combining symbolic representations, unsupervised learning, and expert knowledge to address label scarcity in time series across the physical sciences. The code and configuration files used in this study are publicly available to support reproducibility.