A Unified Shape-Aware Foundation Model for Time Series Classification

AI summary

Overview

• Research area: Machine learning / time series classification, specifically foundation models (large pre-trained models) adapted for classification rather than forecasting.
• Technical level: Advanced. The paper assumes familiarity with transformers, contrastive learning, prototypes, shapelets, and attention pooling.
• Scope: The paper introduces UniShape, a shape-aware foundation model pre-trained on 1.89 million time series samples and evaluated for classification accuracy and interpretability across 158 univariate datasets.

What This Paper Is About

Most time series foundation models are built for forecasting, where the goal is to predict future numeric values from long context windows. Classification instead requires finding short, class-discriminative local patterns (called shapelets) inside fixed-length samples, and existing foundation models largely ignore this distinction as well as the interpretability that domains such as healthcare need. The paper's goal is a single pre-trained model that captures multi-scale discriminative subsequences, transfers well to new target domains, and points to the specific time regions that drove each prediction.

Key Contributions

1. UniShape, a unified foundation model for time series classification that captures multiscale shapelet features and automatically selects the most relevant subsequence scales through a shape-aware adapter.
2. A prototype-based pretraining module that jointly learns representations at the instance level (class tokens) and the shape level (shape tokens), improving generalization to target domains.
3. A large-scale, multi-domain pretraining corpus of approximately 1.89 million univariate samples, assembled from the UCR archive, the UEA archive, and eight additional datasets, with multivariate series decomposed into channel-independent univariate series and all inputs resized to a fixed length of 512.
4. Extensive empirical validation on 158 univariate datasets (128 UCR datasets plus 30 additional datasets), reporting state-of-the-art classification accuracy along with interpretability and ablation analyses.

Main Findings

• Best fully supervised accuracy on 128 UCR datasets: UniShape reaches an average accuracy of 0.8708 and an average rank of 2.71 with 3.1 million parameters. The next best average accuracy is MR-H (MultiRocket-Hydra) at 0.8621, followed by Mantis at 0.8441, SoftShape at 0.8388, MiniRocket at 0.8545, RDST at 0.8571, NuTime at 0.8353, InceptionTime at 0.8315, Rocket at 0.8487, and TS2Vec at 0.8016.
• Weak task-agnostic foundation models: GPT4TS (0.7100), MOMENT (0.7020), UniTS (0.7357), PatchTST (0.6500), and TimesNet (0.6897) fall well below most non-deep-learning and domain-specific methods, suggesting that foundation models not designed for classification adapt poorly to it.
• Statistical significance: UniShape significantly outperforms every baseline on the 128 UCR datasets according to Wilcoxon signed-rank tests, with reported P-values including 7.80E-06 (Rocket), 3.25E-03 (MiniRocket), 8.58E-03 (RDST), 2.96E-02 (MR-H), 2.55E-11 (InceptionTime), 3.68E-32 (SoftShape), 3.20E-23 (UniTS), 2.08E-10 (NuTime), and 1.69E-06 (Mantis).
• Bigger pretraining data helps: On 18 UCR datasets, accuracy rises monotonically across pretraining scales: UCR alone (about 60K samples), UEA (about 1.39M), UCR+UEA (about 1.45M), and ALL (1.89M samples).
• Labels are not a major bottleneck: With pretraining at different labeling ratios on the same 18 UCR datasets, average accuracy is 0.8395 at 0%, 0.8410 at 1%, 0.8529 at 10%, 0.8574 at 50%, and 0.8588 at 100%. Gains beyond 10% labeling are described as marginal and statistically insignificant (P-value > 0.05), so 10% is the default.
• Strong zero-shot feature extraction on 30 unseen datasets: UniShape leads with average accuracy 0.7262 and average rank 3.07, ahead of Mantis (0.7052, rank 3.67), MOMENT (0.6972, rank 4.17), NuTime (0.6917, rank 3.53), Chronos (0.6793, rank 4.10), the raw-series RandomForest baseline (0.6930, rank 3.77), Moirai (0.5691, rank 6.37), GPT4TS (0.5600, rank 6.37), and UniTS (0.3431, rank 8.90).
• Interpretability aligns with known discriminative regions: On ECGFiveDays, the adapter assigns high attention to the delayed T-wave in the [75, 95] interval, previously identified as the key discriminative segment; on GunPoint, it highlights [30, 60] and [90, 110], the intervals associated with motion overshoot when distinguishing Gun from NoGun gestures.
• Ablations show every component matters: On 18 UCR datasets, removing pretraining drops accuracy to 0.8365 (with a Transformer encoder), removing the adapter to 0.8428, removing the instance-prototype loss to 0.8444, removing the shape-prototype loss to 0.8470, and removing both prototype losses to 0.8411, versus 0.8529 for full UniShape.
• Replacement choices degrade performance: Swapping the adapter's CNN for a Transformer gives 0.8446 and for an MLP 0.8431; replacing the Transformer encoder with a CNN gives 0.8512 and with an MLP 0.5651, the worst ablation result and worse than training from scratch.
• Transformer encoder beats CNN under the foundation-model setting: Although prior work found CNNs outperform Transformers in domain-specific training, here the Transformer encoder achieves a better average rank (3.94 for the CNN replacement versus 3.00 for UniShape) once pretraining and fine-tuning are used.
• Instance-level pretraining matters more than shape-level: The drop from removing the instance-prototype loss (-0.85%) exceeds the drop from removing the shape-prototype loss (-0.59%), though their combination shows the two are complementary.

Methodology in Plain English

UniShape works in two stages: pretraining on a large mixed-domain corpus, then fine-tuning on each target dataset. A shared adapter is applied to the same time series many times using sliding windows of five different lengths (window lengths and strides of 64, 32, 16, 8, and 4), producing segments the authors call "shapes." Each segment is normalized and encoded with 1D convolutional networks plus embedded statistics (local means and standard deviations), then compressed into a shape token. An attention head scores each shape token, and the weighted sum becomes a class token for that scale; class tokens are fused from coarse to fine across scales, so the model effectively learns which subsequence length is most informative for a given dataset.

The resulting class and shape tokens pass through a Transformer encoder. A set of learnable prototype vectors, one per class, is updated with an exponential moving average from labeled class tokens, and unlabeled samples are assigned pseudo-labels by nearest prototype under cosine similarity. Two contrastive losses pull class tokens and the top-scoring shape tokens toward their correct prototypes, and these are combined with a MoCo v3 self-supervised contrastive loss between two randomly cropped views of the same input. Fine-tuning uses cross-entropy on the final class token plus the shape-prototype loss as an auxiliary objective to keep the learned shapelets aligned with class labels. The attention scores are what make the model interpretable: they show which time regions the model treats as discriminative.

Why This Matters

This work argues that foundation models for time series need to be designed around classification rather than borrowed from forecasting, and it shows that doing so yields both better accuracy and explanations that experts can inspect.

• Healthcare monitoring: Interpreting ECG signals, where the paper itself shows the model attending to the delayed T-wave region on ECGFiveDays.
• Motion and gesture recognition: Distinguishing gestures such as Gun versus NoGun from sensor traces in GunPoint.
• Industrial and IoT sensor analytics: Classifying machine or device behavior from sensor streams where labels are scarce and local anomalies matter.
• Low-annotation domains generally: Because pretraining with only 10% labeled data performs close to 100%, the approach suits fields where expert labeling is expensive.

Industry relevance: UniShape uses 3.1 million parameters, far fewer than GPT4TS (84.1M) or MOMENT (341.2M), while outperforming them, and it can act as a frozen feature extractor with a small downstream classifier such as a Random Forest. That makes it attractive for deployment where compute is limited and for settings requiring auditable explanations of predictions.

Future Directions

• Extending to multivariate time series: The paper explicitly limits itself to univariate classification and states that modeling multivariate dependencies for more generalizable foundation models is the intended next step.
• Improving shape-level pretraining: The shape-prototype loss contributes less than the instance-prototype loss, so how to make local shapelet learning carry more of the transferable signal remains open.
• Reducing pretraining cost: Full 1.89 million-sample pretraining is described as time-consuming per run, which is why the data-scale and labeling analyses were limited to smaller setups.
• Handling negative transfer from model substitutions: The failure of MLP replacements (0.5651 accuracy, worse than training from scratch) raises the question of what architectural properties are essential for pretrained shapelet representations to transfer at all.

Target Audience

Researchers and practitioners working on time series classification, time series foundation models, or interpretable machine learning, particularly those interested in shapelets and prototype-based learning. It is also relevant to applied scientists in healthcare, sensor analytics, and industrial monitoring who need accurate classifiers under limited labels and want to inspect which parts of a signal drove the prediction. Readers without a background in transformers, contrastive learning, or time series representation learning will need to consult the cited background work first, since the method section is written for an advanced machine learning audience.

Authors’ abstract

Foundation models pre-trained on large-scale source datasets are reshaping the traditional training paradigm for time series classification. However, existing time series foundation models primarily focus on forecasting tasks and often overlook classification-specific challenges, such as modeling interpretable shapelets that capture class-discriminative temporal features. To bridge this gap, we propose UniShape, a unified shape-aware foundation model designed for time series classification. UniShape incorporates a shape-aware adapter that adaptively aggregates multiscale discriminative subsequences (shapes) into class tokens, effectively selecting the most relevant subsequence scales to enhance model interpretability. Meanwhile, a prototype-based pretraining module is introduced to jointly learn instance- and shape-level representations, enabling the capture of transferable shape patterns. Pre-trained on a large-scale multi-domain time series dataset comprising 1.89 million samples, UniShape exhibits superior generalization across diverse target domains. Experiments on 128 UCR datasets and 30 additional time series datasets demonstrate that UniShape achieves state-of-the-art classification performance, with interpretability and ablation analyses further validating its effectiveness.