SEMPO: Lightweight Foundation Models for Time Series Forecasting

AI summary

Overview

Research area: Machine learning for time series forecasting, specifically lightweight foundation models (pre-trained general-purpose forecasters).

Technical level: Intermediate. The paper assumes familiarity with Transformers, self-attention, frequency-domain analysis (FFT), and the zero-shot / few-shot evaluation paradigm, though the two core ideas are explainable in plain language.

Scope: The paper introduces SEMPO, a 6.5M-parameter time series foundation model pre-trained on 83M time points, and evaluates it against 17 open-source baselines on 16 datasets across the TSLib and GIFT-Eval benchmarks.

What This Paper Is About

Time series foundation models (FMs) have become the dominant approach for general-purpose forecasting, but they typically need enormous architectures and billions of pre-training time points to work well, which makes them hard to deploy where compute or data is limited. This paper asks whether both the model size and the pre-training data size can be cut dramatically while keeping strong zero-shot and few-shot forecasting. The proposed answer is SEMPO, which pairs an energy-aware spectral decomposition module with a mixture-of-prompts Transformer.

Key Contributions

1. SEMPO, a time series foundation model with substantially reduced model size (6.5M parameters) and pre-training scale (83M time points), reported to beat state-of-the-art FMs with hundreds of millions of parameters pre-trained on billions of time points.

2. EASD (energy-aware spectral decomposition), which the authors introduce after identifying a bias toward high-energy frequency signals in the pre-training of current time series FMs. EASD partitions spectra by a learnable energy threshold and applies dual-branch frequency masking to preserve low-energy but informative signals.

3. MoPFormer (mixture-of-prompts enabled Transformer), a Transformer that learns a pool of small, dataset-specific prompts and routes each time series token to prompt-based experts, described as a lightweight alternative to Transformers with large mixture-of-experts networks.

4. Extensive evaluation on two large-scale benchmarks covering 16 datasets, reporting average reductions of 12% and 22% in forecasting errors under zero- and few-shot scenarios in the abstract, with larger per-metric reductions reported in the experiments section.

Main Findings

• Zero-shot performance on TSLib: SEMPO is reported as the best performer on the majority of datasets, with average reductions of 23.1% in MSE and 10.8% in MAE across all methods. MSE/MAE are averaged over horizons H ∈ {96, 192, 336, 720}.

• Comparison to large FMs: Against Chronos L (710M parameters, 84B pre-training time points), SEMPO achieves a reported 19.9% MSE improvement; against Time-MoE L (453M, 309B), 19.2%; against Moment (385M, 1.13B), 36.0%. SEMPO uses 83M time points versus 1.13B to 309B for these models.

• Comparison to lightweight FMs: Compared to Moirai S (14M parameters, 27B pre-training time points), SEMPO is reported as markedly superior despite the smaller parameter count.

• GIFT-Eval zero-shot: SEMPO achieves the best performance in 6 out of 9 selected datasets when compared against the Chronos family.

• Few-shot with 5% of training data: SEMPO outperforms 11 recent deep models with an average MSE reduction of 32.3%. It outperforms Time-LLM (11.4%), GPT4TS (15.9%), and S2IP-LLM (10.7%), and surpasses the MLP-based FM TTM by 4.6%.

• Marginal gains from fine-tuning on small targets: Compared with zero-shot results, fine-tuning yields only marginal gains on ETTh1 and ETTh2, which the authors attribute to limited scale or variability in the target dataset.

• Ablation on spectral masking: Replacing the default dual-branch spectral masking raises average MSE from 0.350 to 0.399 with multi-band masking (A.1) and to 0.416 with random patch masking (A.2). The A.2 degradation is attributed to its unstructured nature disrupting temporal dependencies, while A.1 is said to let dominant high-energy components eclipse low-energy signals.

• Ablation on MoP: Replacing MoP with a sparse MoE (3 experts, 1 activated, 8.5M parameters) underperforms SEMPO despite the larger model size. Removing the adaptive router in favor of conventional prefix tuning degrades performance further.

• Low-energy modeling: In spectrum visualizations on the Electricity dataset with H = 336, Chronos S largely neglects subtle low-energy components, and Moirai L shows non-negligible low-energy responses that do not consistently align with ground-truth spectral peaks. SEMPO is reported to attend to signals across the full energy spectrum.

• Prompt routing behavior: Visualized gating scores show that within a dataset, different patch tokens attend to different subsets of prompt-based experts. Same-domain datasets (e.g., ETTh1 and ETTm2) show similar routing patterns, while cross-domain datasets (e.g., Traffic and Weather) show distinct token-to-prompt mappings.

Methodology in Plain English

SEMPO is an encoder-decoder model built from four parts: the EASD module, patchify-and-project, the MoPFormer backbone, and reconstruction and prediction heads.

Splitting signals by energy (EASD). Each univariate series is normalized and transformed into the frequency domain with an FFT. The energy of a frequency is defined as the squared magnitude of its Fourier coefficient. A learnable threshold τ, adapted to the input's own spectral characteristics, splits the spectrum into a high-energy component and a low-energy component. This matters because low-energy frequencies are easy for a model to overlook when high-energy ones dominate.

Masking by frequency band. For each energy branch, the method samples a cutoff frequency and a direction (mask below or above the cutoff), producing multi-band masks. The masked high- and low-energy representations are recombined and transformed back to the time domain with an inverse FFT. Crucially, the two branches are masked independently, with no shared sampling parameters, which the authors say promotes spectral diversity and counters energy bias. The masked series is then cut into P non-overlapping patches of length 64, linearly projected, and given positional embeddings.

Handling diverse data with prompts (MoPFormer). Rather than scaling up the Transformer, SEMPO keeps a pool of 128 small prompt-based experts. A router computes softmax gating scores over this pool from each input token, and the experts are merged as a weighted combination. A two-layer MLP plus a reshape turns the merged representation into key-value pairs, which are concatenated onto the key and value matrices of self-attention. The backbone uses standard recent components: RMSNorm, pre-normalization, and SwiGLU.

Two-stage training. First, energy-aware pre-training trains on multi-domain data with a self-supervised reconstruction objective (MSE between input and reconstruction), without prompts. Second, MoP tuning freezes the Transformer backbone and trains only the mixture of prompts and the prediction heads, using multiple forecasting horizons plus a reconstruction term. At inference, only the frozen backbone, the prompts, and the prediction head are used.

Experimental setup. Pre-training used a ~83M-time-point subset of the UTSD collection with a 9:1 train/validation split. The full two-stage process took 10 hours on 4 A6000-48G GPUs with BF32 precision and batch size 2,048. Default settings: 6 layers, 16 heads, latent dimension 256, patch size 64, 128 prompts, 4 masks. The lookback window is L = 512 on TSLib. Baselines number 17 open-sourced models across three groups: pre-trained time series FMs, LLM-based FMs, and task-specific models.

Why This Matters

Impact on research. The paper challenges the assumption that foundation-model performance for time series requires scaling both parameters and pre-training data. It identifies a concrete mechanism—a bias toward high-energy frequency components—and offers a targeted architectural fix, plus a prompt-based alternative to expensive mixture-of-experts routing. If the results hold, they reframe "small data, small model" as a viable research direction rather than a compromise.

Real-world applications (domains named in the paper):

• Energy demand estimation
• Inventory optimization
• Urban computing
• Climate system modeling

Industry relevance. The paper explicitly targets deployment in resource-constrained environments, contrasting SEMPO's 6.5M parameters and 83M pre-training time points against models ranging from 14M to 710M parameters and 27B to 309B pre-training time points. Few-shot settings using 5% and 10% of training data are directly relevant to organizations with limited historical data for a specific target domain. Code and data are released at the project's GitHub repository (https://github.com/mala-lab/SEMPO).

Future Directions

• Scaling the lightweight design. The paper states that scaling analysis of the model, along with hyperparameter analysis of the prompt number (I), mask number (N_M), and latent dimension (D_p), is provided in the appendices, leaving open how far this small-model approach can be pushed before the efficiency advantage erodes.

• Better use of adaptation data. Fine-tuning on 5% and 10% of target data produced only marginal gains over zero-shot results on ETTh1 and ETTh2. Understanding when and how MoP tuning helps—and when it does not—remains unresolved in the main text.

• Generalization beyond the evaluated domain mix. Zero- and few-shot datasets were drawn from entirely new domains not present in pre-training, and the model relies on channel independence to handle multivariate input. Whether cross-variate dependency modeling would improve results is not addressed.

• Robustness of the prompt-routing interpretation. The paper visualizes expert gating patterns and notes same-domain versus cross-domain differences, but the analysis is descriptive. What the individual prompt-based experts actually specialize in, and whether the routing generalizes to unseen domains, are open questions.

Target Audience

This paper is most useful to machine learning researchers and practitioners working on time series forecasting, particularly those interested in foundation models, efficient/lightweight architectures, mixture-of-experts routing, and frequency-domain representation learning. It is also relevant to applied engineers who need forecasting under compute, memory, or data constraints—for example in energy, traffic, weather, and retail inventory settings—and to readers tracking the ongoing debate over whether scaling data and parameters is the only path to strong generalization.

Authors’ abstract

The recent boom of large pre-trained models witnesses remarkable success in developing foundation models (FMs) for time series forecasting. Despite impressive performance across diverse downstream forecasting tasks, existing time series FMs possess massive network architectures and require substantial pre-training on large-scale datasets, which significantly hinders their deployment in resource-constrained environments. In response to this growing tension between versatility and affordability, we propose SEMPO, a novel lightweight foundation model that requires pretraining on relatively small-scale data, yet exhibits strong general time series forecasting. Concretely, SEMPO comprises two key modules: 1) energy-aware SpEctral decomposition module, that substantially improves the utilization of pre-training data by modeling not only the high-energy frequency signals but also the low-energy yet informative frequency signals that are ignored in current methods; and 2) Mixture-of-PrOmpts enabled Transformer, that learns heterogeneous temporal patterns through small dataset-specific prompts and adaptively routes time series tokens to prompt-based experts for parameter-efficient model adaptation across different datasets and domains. Equipped with these modules, SEMPO significantly reduces both pre-training data scale and model size, while achieving strong generalization. Extensive experiments on two large-scale benchmarks covering 16 datasets demonstrate the superior performance of SEMPO in both zero-shot and few-shot forecasting scenarios compared with state-of-the-art methods. Code and data are available at https://github.com/mala-lab/SEMPO.