EVEREST: An Evidential, Tail-Aware Transformer for Rare-Event Time-Series Forecasting

AI summary

Overview

Research area: Machine learning for rare-event forecasting in multivariate time series, combining transformer sequence modelling, evidential uncertainty quantification, and extreme-value theory.

Technical level: Advanced. The paper assumes familiarity with transformer architectures, calibration metrics (ECE, Brier score), evidential learning (Normal–Inverse–Gamma), and peaks-over-threshold extreme-value theory.

Scope (one sentence): The paper introduces EVEREST, a compact transformer whose training-time auxiliary heads (evidential NIG, Generalized Pareto exceedance, and precursor supervision) improve calibrated, tail-aware rare-event forecasting, demonstrated on nine solar-flare tasks plus the industrial SKAB anomaly benchmark.

What This Paper Is About

Forecasting rare, high-impact events from multivariate time series is hard for three reasons the authors name directly: severe class imbalance, long-range dependencies that dilute early precursors, and the need for calibrated probabilities plus explicit tail-risk estimates when decisions are thresholded. Standard training objectives give little gradient signal to extremes and offer no input-conditional uncertainty. The goal of EVEREST is to address discrimination, calibration, and tail sensitivity jointly inside one compact transformer that costs nothing extra at inference.

Key Contributions

1. A single-query attention bottleneck for temporal focusing. A learned scorer w ∈ ℝ^d produces one soft attention distribution over time, pooling the encoder states into a single vector z. It adds only +d parameters and O(Td) FLOPs, and the authors report that replacing it with mean pooling loses 0.427 TSS on the hardest task.

2. Three training-only auxiliary heads that regularise a shared representation. An evidential Normal–Inverse–Gamma (NIG) head predicts (μ, v, α, β) over the logit for closed-form aleatoric/epistemic uncertainty; an extreme-value head predicts Generalized Pareto (ξ, σ) for logit exceedances above a high batchwise quantile u (90% by default); and a lightweight precursor head applies binary cross-entropy with the same label as anticipatory supervision. None of the three is used at inference.

3. A composite training objective. The loss is L = λ_f L_focal + λ_e L_evid + λ_t L_evt + λ_p L_prec, with reference weights (0.8, 0.1, 0.1, 0.05) and focal γ annealed linearly from 0 to 2 over the first 50 epochs, motivated by the Information Bottleneck principle.

4. A single-head deployment with no inference overhead. Deployment uses only the classification head p̂ = σ(l), so runtime matches a standard transformer of comparable size (approximately 0.81M parameters; 814.10k in the appendix budget table).

Main Findings

• Headline solar-flare discrimination. Across nine SHARP–GOES tasks, EVEREST reaches TSS of 0.973/0.970/0.966 for ≥C at 24/48/72 hours, 0.898/0.920/0.906 for ≥M, and 0.907/0.936/0.966 for ≥M5. All nine tasks exceed the reported baseline TSS values, with the paper citing gains of +0.251 TSS for ≥C–48 h and +0.237 for ≥M5–72 h.

• Baseline context. Comparisons are against Liu et al. (2019), Sun et al. (2022), and Abduallah et al. (2023, SolarFlareNet, described as the strongest published baseline). The paper reports that significance testing is applied within its own models, not against the baselines; baseline comparisons are to values reported in prior work.

• Calibration. On the most imbalanced task (M5–72 h) the model obtains ECE = 0.016 with a near-diagonal reliability curve, with similar trends for C–72 h and M–72 h. Precision is reported as 0.994/0.993/0.992 for ≥C at 24/48/72 h, a TSS decay of only 0.007 from 24 h to 72 h.

• Ablations on the hardest task (M5–72 h). Effect sizes reported are: attention bottleneck +0.427 TSS over mean pooling; EVT head +0.285 TSS; evidential NIG head +0.064 TSS; composite schedule (γ annealing and stable joint training) +0.045 TSS. Removing the precursor head degrades performance by −0.650 TSS despite being discarded at inference. Mixed precision (AMP) is described as indispensable, with FP32 runs diverging or underperforming.

• Robustness of the auxiliary regularisers. A 5×5 log-scale sweep over evidential and EVT loss weights keeps TSS and ECE stable. An EVT quantile sweep over u ∈ {0.85, 0.90, 0.95} on M5–72 h yields TSS in [0.903, 0.932] and ECE in [0.0119, 0.0164].

• Decision analysis under asymmetric costs. For M5–72 h with C_FN : C_FP = 20 : 1, the minimum-cost threshold is τ = 0.240*, distinct from the balanced-score threshold τ = 0.460.

• Cross-domain transfer on SKAB. With the architecture described as unchanged, EVEREST reaches mean TSS = 0.964 and F1 = 98.16% (the summary table lists F1 = 98.2 ± 1.7, precision 97.7 ± 2.9, recall 98.6 ± 3.2, TSS = 0.964 ± 0.028 across eight valve scenarios). The appendix notes that SKAB uses a reduced width (d = 96) and depth (L = 4), which differs from the "unchanged" phrasing in the main text.

• Effect size versus model size. EVEREST has 814k parameters and 16.6M FLOPs per window, giving 20.4 FLOPs per parameter, compared with SolarFlareNet at 6,120k parameters, 0.62M FLOPs, and 0.10 FLOPs per parameter. The six-layer backbone accounts for 97.6% of parameters and 97.7% of FLOPs; the auxiliary heads together contribute 0.91k parameters (0.11%) and 0.02M FLOPs (0.12%).

• Training efficiency. Mean epoch times are approximately 24 s on an RTX A6000 and approximately 69 s on an M2 Pro. Full energy and carbon accounting is referenced as appearing in the supplement, and is not reported in the content provided here.

• Interpretability and prospective validation. Saliency analysis shows true positives with coordinated increases in USFLUX and MEANGAM in the final hours before the forecast horizon, while true negatives and false positives show flatter or noisier signatures. On the unseen 6 Sep 2017 X9.3 flare (NOAA AR 12673), the largest event of Solar Cycle 24, data from 3–7 September 2017 were excluded from training and threshold calibration; the probability trace and lead-time statistics are in the appendix, not in the main text.

Methodology in Plain English

The researchers framed rare-event forecasting as binary prediction on fixed windows: each example is a window X ∈ ℝ^(T×F) with label y ∈ {0, 1} indicating whether an event occurs within a forecast horizon, and the model outputs a logit l and probability p̂ = σ(l) compared to a threshold τ.

The architecture has four stages. Raw inputs are projected to 128-dimensional tokens with sinusoidal positional codes scaled by a learnable factor. Six transformer encoder blocks (4 attention heads, FFN width 256, dropout 0.20) process the sequence. A single-query attention bottleneck then collapses the sequence into one vector z, rather than averaging. A shallow shared 128-dimensional MLP feeds four parallel heads: the primary classification logit plus the three training-only auxiliaries.

Each auxiliary targets a different weakness. Focal loss reweights hard examples to counter extreme rarity. The evidential NIG head learns a distribution over the logit so the model represents uncertainty in closed form without sampling. The EVT head fits a Generalized Pareto distribution to the batch's logit exceedances above a high quantile, which shifts gradient mass toward the risky upper tail. The precursor head reuses the same label purely as anticipatory supervision.

Data preparation follows the SHARP–GOES protocol: nine selected SHARP vector-magnetogram parameters aligned to GOES labels, quality masks (QUALITY = 0, |CMD| ≤ 70°, observer radial-velocity filter), and the same HARPNUM-stratified split as prior work so that each active region appears in exactly one split. To prevent leakage, the exact split from prior work is reused. Evaluation uses TSS as the primary metric, with precision/recall/F1, AUROC, PR-AUC, Brier score, and ECE over 15 equal-frequency bins. Thresholds are chosen by grid search over {0.10, ..., 0.90} using a balanced score (40% TSS, 20% F1, 15% precision, 15% recall, 10% specificity). Five seeds are trained, with 95% CIs from a 10^4-draw bootstrap stratified by NOAA active-region identifier.

Why This Matters

Impact on research. The paper's argument is that encoder improvements and calibration improvements are usually pursued separately, and that tailoring a compact architecture to calibration and tail sensitivity at once is a gap. Its distinctive move is making extreme-value theory a training-time regulariser on logits rather than a post-hoc or residual-based tail estimator, and combining that with evidential uncertainty in the same shared representation.

Real-world applications (named in the paper):

• Space weather forecasting and satellite diagnostics, where C-, M-, and M5-class flare prediction supports operational alerting.
• Industrial monitoring and anomaly detection, demonstrated on SKAB valve-sensor traces.
• Power systems, cited as a domain where missed high-impact events carry asymmetric cost.
• Weather forecasting and satellite health monitoring, listed among high-stakes target domains.

Industry relevance. Deployment uses one classification head with no auxiliary inference cost, and the model is small (approximately 0.81M parameters) with compute-dense forward passes (16.6M FLOPs), trainable on commodity hardware at roughly 24 s per epoch on an RTX A6000. The cost–loss sweep with C_FN : C_FP = 20 : 1 producing a different optimal threshold from the balanced score shows the model supports asymmetric decision-making without retraining. The authors frame the mixed-precision, small-model design as consistent with "Green AI" norms.

Future Directions

1. Streaming or indefinite context. The authors identify the fixed context window as a limitation that may miss slow precursor dynamics, and propose streaming/state-space memory or compressive transformers.

2. Multimodal fusion. Inputs are unimodal; the paper proposes fusing SHARP with EUV/radio data under cadence-aware alignment, and lists image and radio modalities as explicitly out of scope here.

3. Federated or continual training. Proposed to mitigate cross-cycle drift between training and deployment periods and to work around institutional data silos.

4. Compression and richer explanation. Quantisation/distillation and hardware-aware compilation for edge deployment, plus counterfactual and time-series integrated-gradient explanation methods to support operational trust and post-hoc auditing. The paper also flags integrating a contrastive forecasting variant of the precursor head as a natural extension.

Target Audience

Researchers and practitioners working on imbalanced time-series classification, uncertainty calibration, or extreme-value methods in machine learning; space-weather and heliophysics groups needing operational flare forecasting; and engineers applying transformer models to industrial monitoring, power systems, or satellite diagnostics under asymmetric misclassification costs. The paper is also relevant to readers interested in small-model, training-only-auxiliary design patterns, since the ablation structure isolates the marginal value of each component.

Authors’ abstract

Forecasting rare events in multivariate time-series data is challenging due to severe class imbalance, long-range dependencies, and distributional uncertainty. We introduce EVEREST, a transformer-based architecture for probabilistic rare-event forecasting that delivers calibrated predictions and tail-aware risk estimation, with auxiliary interpretability via attention-based signal attribution. EVEREST integrates four components: (i) a learnable attention bottleneck for soft aggregation of temporal dynamics; (ii) an evidential head for estimating aleatoric and epistemic uncertainty via a Normal--Inverse--Gamma distribution; (iii) an extreme-value head that models tail risk using a Generalized Pareto Distribution; and (iv) a lightweight precursor head for early-event detection. These modules are jointly optimized with a composite loss (focal loss, evidential NLL, and a tail-sensitive EVT penalty) and act only at training time; deployment uses a single classification head with no inference overhead (approximately 0.81M parameters). On a decade of space-weather data, EVEREST achieves state-of-the-art True Skill Statistic (TSS) of 0.973/0.970/0.966 at 24/48/72-hour horizons for C-class flares. The model is compact, efficient to train on commodity hardware, and applicable to high-stakes domains such as industrial monitoring, weather, and satellite diagnostics. Limitations include reliance on fixed-length inputs and exclusion of image-based modalities, motivating future extensions to streaming and multimodal forecasting.