Frequency Matters: When Time Series Foundation Models Fail Under Spectral Shift

AI summary

Overview

Research area: Time series foundation models (TSFMs), transfer learning, and spectral/frequency analysis of temporal data, with an industrial case study from mobile gaming.

Technical level: Intermediate. The paper assumes familiarity with self-supervised pretraining, transformer backbones, and standard evaluation metrics (AUC, MSE, MAE), but its central argument — that frequency composition drives transfer failure — is explained without heavy mathematics in the main text.

Scope: The paper argues, through one industrial deployment case and a set of controlled synthetic experiments, that TSFMs fail to transfer to downstream tasks when the dominant frequency bands of those tasks fall outside the bands represented during pretraining.

What This Paper Is About

Time series foundation models report strong numbers on public benchmarks, prompting comparisons to a "BERT moment" for time series, yet their usefulness in real industrial settings is uncertain. This paper investigates why a pretrained TSFM (MOMENT-small) underperforms domain-adapted baselines on a large-scale player engagement prediction task drawn from Candy Crush Saga. The authors propose spectral shift — a mismatch between the dominant frequency components in downstream tasks and those present during pretraining — as a primary explanation, and test that hypothesis with controlled synthetic data that deliberately separates "seen" from "unseen" frequency bands.

Key Contributions

1. An industrial-scale negative result. The authors benchmark MOMENT-small against domain-adapted baselines (PatchTST, XGBoost, TabNet) on a Player Engagement Prediction (PEP) task built from 824,208 multivariate time series samples from approximately 40,000 Candy Crush Saga players, and show the TSFM trailing all of them.

2. A frequency-based diagnosis. They compare the dominant frequencies in their downstream dataset with those of datasets used to pretrain MOMENT (FordA and FaultDetectionA from the UCR repository) and identify a gap, framing spectral shift as a driver of transfer degradation alongside covariate shift, nonstationarity, and label misalignment.

3. Controlled synthetic probing of the mechanism. They construct paired "seen" and "unseen" band datasets derived from six pretraining datasets (FordA, FordB, ElectricDevices, SmallKitchenAppliances, FaultDetectionA, FaultDetectionB), and evaluate a frozen MOMENT-small encoder with lightweight trained heads on both, in regression and classification form.

4. Practical guidance and protocol proposals. They recommend quantifying spectral overlap between pretraining corpora and downstream data, applying frequency-aware augmentation or light adaptation when overlap is low, and adopting benchmarks that explicitly stress-test robustness under spectral shift.

Main Findings

• The TSFM loses to domain-adapted baselines on the industrial task. On the PEP benchmark, MOMENT-small reaches Accuracy 0.758 (player-holdout) and 0.701 (temporal-holdout), AUC 0.791 and 0.749, MSE 2.250 and 2.854, and MAE 1.151 and 1.324. PatchTST — fine-tuned on the task — reaches Accuracy 0.939 and 0.921, AUC 0.982 and 0.975, MSE 0.518 and 0.711, and MAE 0.489 and 0.586. XGBoost (Accuracy 0.841 / 0.801, AUC 0.915 / 0.883) and TabNet (0.836 / 0.795, AUC 0.911 / 0.852) also outperform the TSFM, despite MOMENT-small being the only pretrained foundation model in the comparison.

• A visible spectral gap between pretraining and deployment data. The spectral analysis in Appendix B shows the industrial dataset exhibits fundamentally different dominant frequencies from the pretraining corpora (FordA and FaultDetectionA), suggesting MOMENT was not exposed to such frequency patterns.

• Frozen encoders degrade on unseen frequency bands in regression. Using MOMENT-small with only a regressor trained, every tested dataset shows higher error on unseen bands, e.g. ElectricDevices MSE 0.644 ± 0.002 (seen) vs 0.952 ± 0.003 (unseen) and MAE 0.559 ± 0.001 vs 0.791 ± 0.004; FaultDetectionA MSE 0.689 ± 0.001 vs 0.942 ± 0.004; FaultDetectionB MSE 1.129 ± 0.172 vs 2.005 ± 0.266. Smaller but consistent gaps appear for FordA (0.333 ± 0.010 vs 0.366 ± 0.005), FordB (0.333 ± 0.010 vs 0.358 ± 0.008), and SmallKitchenAppliances (0.691 ± 0.059 vs 0.877 ± 0.017).

• Classification shows the same direction, with exceptions. The largest drops on unseen bands occur for ElectricDevices (Accuracy 0.785 ± 0.003 vs 0.650 ± 0.003; AUC 0.890 ± 0.001 vs 0.716 ± 0.002), FaultDetectionA (0.766 ± 0.003 vs 0.651 ± 0.003; AUC 0.858 ± 0.000 vs 0.707 ± 0.003), ECG5000 (0.809 ± 0.020 vs 0.591 ± 0.028; AUC 0.898 ± 0.007 vs 0.692 ± 0.007), and SwedishLeaf (0.689 ± 0.020 vs 0.600 ± 0.040; AUC 0.796 ± 0.021 vs 0.643 ± 0.028). The trend is not uniform: on FordB, SmallKitchenAppliances, and FaultDetectionB the unseen-band numbers are comparable to or slightly above the seen-band ones, and FaultDetectionB results carry very large standard deviations (AUC 0.519 ± 0.306 seen vs 0.476 ± 0.190 unseen).

• Frequency is a harder transfer axis than semantics. The authors contrast time series with text: linguistic structure and semantics share patterns across domains, whereas small changes in dominant frequency band, sampling rate, or seasonality can radically change a signal's character, undermining the generalization that LLM-style transfer relies on.

• Stated limitations. The evidence comes from a single industrial domain (mobile gaming) and a single TSFM configuration, with broader validation described as ongoing. The synthetic probes use sinusoidal signals that do not fully capture irregular sampling, burstiness, or regime shifts.

Methodology in Plain English

The work proceeds in two stages.

First, an industrial benchmark. The authors define Player Engagement Prediction as forecasting a player's future behavioral and transactional outcomes over a 30-day horizon from their past interaction history. Each input is a multivariate time series of up to 512 completed game rounds within a lookback window of at most 30 days, described by 32 univariate features spanning progression (level difficulty, success/fail, attempt count), gameplay (time between rounds, number of moves), resources (purchase/use/balance of lives and boosters), strategy (participation in special features), and context (hour of day, days since install, device type). Two labels are predicted: a binary Purchase vs No Purchase flag and a continuous Engagement Score reflecting in-game behavioral intensity, normalized to protect business-sensitive information. Features are scaled with min-max or log transformations with application-specific capping; commercially sensitive financial signals are normalized and anonymized; player-identifiable data are not used. The primary dataset spans 226 days with 824,208 samples from roughly 40,000 selected players, split into player-holdout validation and test sets of 178,279 and 176,632 samples, plus a temporal-holdout set covering 28 days about two months after the primary window, with 34,279 samples from roughly 7,000 players. Temporal-holdout evaluation is zero-shot: no tuning of weights or hyperparameters. Baselines are XGBoost and TabNet on temporally aggregated features, PatchTST as a fine-tuned task model, and MOMENT-small assessed under linear probing with lightweight heads.

Second, controlled synthetic probes. Starting from datasets used to pretrain MOMENT, the authors compute each series' FFT and retain the top-5 dominant frequencies, giving a band [f_low, f_high]. They then generate "seen" signals whose frequencies are sampled uniformly from that band and "unseen" signals whose frequencies come from a shifted, disjoint interval [f_low + δ, f_high + δ], also constrained to stay within the overall frequency range of the original series. For example, if the dominant range is [0, 20] Hz, unseen signals are drawn from [20 + δ, 40 + δ]. Each signal is a sum of sinusoids with random phases and light additive noise. Regression targets are the z-score normalized sum of the sampled frequencies, which removes scale effects and prevents the model from solving the task from spectral displacement alone. Classification labels bin adjacent frequency ranges (for instance [0, 10] vs [10, 20] Hz), with the binning threshold set at the median of the training split's frequency range to balance classes. Splits use a fixed 70/15/15 partition over single-channel sequences of fixed length L, typically 512. The MOMENT backbone — patch embedder and transformer encoder — is frozen; only a linear regression or binary classification head is trained, with Adam at learning rate 10⁻³ for 50 epochs, MSE loss for regression and binary cross-entropy for classification, and model selection on validation MSE. Each experiment is run 3 times and average and standard deviation are reported. Compute included CPUs, NVIDIA GPUs (L4 24GB, T4 16GB), and an Apple MPS device (M1 MAX 32GB), with up to eight GPUs under distributed data parallelism for the MOMENT runs on the gaming dataset.

Why This Matters

Impact on research. The paper challenges the assumption that benchmark success on curated public datasets predicts industrial transfer, and reframes TSFM generalization as partly a question of spectral coverage rather than model capacity or architecture. It suggests that the field's evaluation protocols and pretraining objectives should explicitly account for spectral diversity, and that a model performing well in-domain may simply have memorized frequency bands rather than learned general temporal structure.

Real-world applications:

• Mobile gaming and player analytics, where engagement and purchase forecasting over horizons such as 30 days directly informs live-service decisions, and where player telemetry has irregular, multi-scale rhythms unlike standard benchmark data.
• Energy and utilities, where consumption signals have strong, domain-specific periodicities; the authors' own author list includes an affiliation with Fever Energy.
• Healthcare and sensor monitoring, where physiological and machine-sensor signals such as ECG and fault-detection traces have dominant bands far from those in general pretraining corpora.
• Finance and demand forecasting, where sampling rate and seasonality shift across instruments, markets, and business units, making spectral mismatch between a general pretrained model and a specific deployment likely.

Industry relevance. The practical message for teams considering a foundation model off the shelf is to check spectral overlap with their own data before committing, and to expect a fine-tuned domain-specific model such as PatchTST to be a strong default. The paper's own benchmark makes this concrete: on the gaming task the TSFM was beaten not only by the fine-tuned transformer but by tabular baselines, XGBoost and TabNet, that operate on aggregated features. Frequency-aware augmentation or light adaptation is offered as a lower-cost remedy when overlap is low.

Future Directions

• Broaden the evidence base. The authors state they are conducting broader validations beyond one industrial domain and a single TSFM configuration; testing other backbones (for example Chronos, Moirai, TimesFM, or Lag-Llama, which the related work section surveys) and other industries would test whether spectral shift generalizes as an explanation.

• Move beyond sinusoidal probes. The synthetic setup uses clean sums of sinusoids, which the authors acknowledge do not capture irregular sampling, burstiness, or regime shifts. More realistic synthetic and semi-synthetic generators would strengthen the causal claim.

• Develop frequency-aware pretraining. The paper suggests auxiliary objectives that predict dominant frequency bands, frequency-based data augmentation, and frequency-based sampling to expand frequency coverage during pretraining, and hypothesizes these could improve zero-shot and few-shot performance.

• Standardize spectral robustness benchmarks and overlap metrics. Concrete protocols for quantifying spectral overlap between a pretraining corpus and a downstream dataset, plus benchmarks that deliberately stress-test performance under spectral shift, are proposed as needed infrastructure for the field.

Target Audience

This paper is most valuable to machine learning engineers and applied scientists deciding whether to deploy a time series foundation model in production, particularly in domains with distinctive periodicities such as gaming, energy, healthcare, and finance. It also speaks to TSFM researchers interested in pretraining objectives and evaluation design, and to benchmark designers who want to move beyond curated public datasets toward protocols that test robustness under distributional and spectral shift. Readers need only a working understanding of self-supervised pretraining and standard regression and classification metrics to follow the main argument.

Authors’ abstract

Time series foundation models (TSFMs) have shown strong results on public benchmarks, prompting comparisons to a "BERT moment" for time series. Their effectiveness in industrial settings, however, remains uncertain. We examine why TSFMs often struggle to generalize and highlight spectral shift (a mismatch between the dominant frequency components in downstream tasks and those represented during pretraining) as a key factor. We present evidence from an industrial-scale player engagement prediction task in mobile gaming, where TSFMs underperform domain-adapted baselines. To isolate the mechanism, we design controlled synthetic experiments contrasting signals with seen versus unseen frequency bands, observing systematic degradation under spectral mismatch. These findings position frequency awareness as critical for robust TSFM deployment and motivate new pretraining and evaluation protocols that explicitly account for spectral diversity.