Be Wary of Your Time Series Preprocessing

AI summary

Overview

Research area: Time series representation learning with Transformer-based models, specifically the largely unexamined role of input normalization and scaling (Standard and Min-Max, in instance-based and global variants) as a design choice rather than a routine preprocessing step.

Technical level: Advanced. The core of the paper is a formal expressivity analysis with two theorems and proofs (Appendix A and Appendix B), built on dot-product self-attention, weight-norm bounds, and Markov's inequality. The empirical sections are more accessible.

Scope: A single paper presenting the first formal analysis of how instance-based versus global scaling affects the expressivity of Transformer-based time series models, paired with classification and forecasting benchmarks.

What This Paper Is About

Almost every Transformer-based time series model scales or normalizes its input before attention, usually justified as a way to stabilize training — but the effect of that scaling on what the model can represent has not been rigorously studied. This paper asks how the choice of normalization strategy (Standard versus Min-Max, applied either per channel or globally) changes a model's ability to separate similar and dissimilar time series in its learned representation space. The authors derive theoretical expressivity bounds for these strategies and then test the practical consequences on classification and forecasting benchmarks across several models.

Key Contributions

1. A theoretical framework for analyzing how input scaling affects the expressivity of Transformer-based models (TBMs) for time series — formalizing expressivity as the probability that a model separates two inputs within an ε-similar neighborhood by more than a threshold σ.
2. Theoretical bounds (Theorem 1 and Theorem 2) for two widely used normalization families — Standard and Min-Max scaling — in both instance-based and global forms, showing that expressivity depends on the model's weight norms and on the scaling range induced by the normalization.
3. Derivation of task- and data-dependent guidance from those bounds, including when global normalization may help (dominant-channel tasks) and when instance-based normalization is preferable (fine-grained multi-channel variation).
4. Empirical validation across classification and forecasting tasks using multiple Transformer-based architectures (Transformer, PatchTST, Autoformer, TimesNet), showing that no single normalization method consistently wins and that omitting normalization can sometimes perform best.

Main Findings

• Expressivity is driven by variance, not the mean. Because self-attention is translation invariant, the mean component of Standard normalization does not affect the bound; the variance is the controlling factor. Under global Standard normalization, a channel with a dominant scale can diminish the model's sensitivity to other channels.

• Global normalization ties sensitivity to dominant channels. The instance-based bound scales with a sum over per-channel inverse variances, whereas the global bound depends on a single shared variance (or, for Min-Max, on the single global range x_max − x_min). This means global scaling can suppress useful information from less prominent channels.

• The task determines which strategy is preferable. For tasks where a large variation in a specific channel matters — the paper gives anomaly detection, where a spike in one sensor may signal a fault — global normalization may enhance expressivity by amplifying the dominant signal. For tasks like motion recognition, where multiple channels (for example, different spatial axes) carry complementary information, global normalization may obscure critical patterns and instance-based scaling becomes preferable.

• Classification results are highly context-dependent. On UEA benchmark datasets, global Standard normalization often performed better overall, but instance-based beat it on datasets such as PEMS-SF (Transformer: 88.05 ± 2.68 instance versus 85.36 ± 1.79 global) and Handwriting (37.57 ± 0.53 instance versus 37.18 ± 0.29 global). On UWaveGestureLibrary — where inputs are already normalized by design — global, instance-based, and no normalization gave comparable results (86.77 ± 0.39 across all three for the Transformer).

• No normalization can be the best choice. Across models and datasets, the "None" setting sometimes achieved the top accuracy, contradicting the assumption that normalization is always beneficial. For the Transformer on SpokenArabicDigits, for instance, None reached 98.59 ± 0.04, ahead of Standard (Instance) at 98.27 ± 0.26.

• Forecasting behaves differently from classification. On the ETT-Small dataset, global Standard normalization consistently outperformed instance-based Standard normalization for every model tested: Transformer 0.7617 ± 0.0512 versus 1.0023 ± 0.0041, Autoformer 0.4593 ± 0.0023 versus 0.9818 ± 0.0281, and PatchTST 0.4046 ± 0.0004 versus 0.8010 ± 0.0029. The authors suggest forecasting benefits from preserving global scale because it relies on longer-range trends rather than channel-specific detail.

• Omitting normalization hurts forecasting far more than classification. The None setting produced the worst MAE by a wide margin on ETT-Small: Transformer 2.9003 ± 0.1844, Autoformer 1.7321 ± 0.0362, PatchTST 1.4864 ± 0.0047. Robust scaling was strongest for the Transformer (0.7327 ± 0.0879) and Autoformer (0.4583 ± 0.0039), while global Standard remained best for PatchTST (0.4046 ± 0.0004).

• There is no free lunch in preprocessing. The authors state explicitly that no universal normalization strategy guarantees optimal performance across all tasks and datasets.

Methodology in Plain English

The authors first build a way to talk about how "expressive" a time series Transformer is. They define a small neighborhood around each input time series, bounded by a similarity budget ε, containing inputs considered semantically similar. A model is expressive in a region if it maps similar inputs to representations that differ by more than a threshold σ; the probability of that happening is the expressivity measure, and a model is (ε, σ, γ)-expressive when that probability stays below γ.

They then analyze a simplified but tractable model: a single-layer Transformer with H attention heads, using dot-product self-attention, and non-linear activations assumed to be 1-Lipschitz continuous. Because normalization is applied before the model, the full function can be written as a composition of the scaling function and the Transformer, so the final bound is simply the product of the two contributions. They apply this to Standard normalization (mean and standard deviation) and Min-Max normalization (rescaling to [0,1]), each in an instance-based form and a global form, and derive bounds in terms of the scaling parameters, the input dimension, and norms of the model's query, key, value, output, and feed-forward weights (collected into the terms C₁ and C₂). The proofs appear in Appendix A and Appendix B.

For the empirical side, they test classification and forecasting. Classification uses benchmark datasets from the UEA time series classification repository, with each experiment repeated five times to control for random initialization. Forecasting uses the ETT-Small dataset, with additional trends also examined on Traffic and Weather. Models include a standard Transformer, PatchTST, Autoformer, and TimesNet. All models are trained with the Adam optimizer; classification optimizes cross-entropy loss, while forecasting uses ℓ2 loss and reports Mean Absolute Error, with results given as mean and standard deviation.

Why This Matters

Impact on research. Normalization has been treated as an implementation detail across time series architecture, pretraining, and tokenization research. This paper reframes it as a design choice with formal consequences for representational capacity, and provides an interpretable lens for reasoning about how scaling decisions propagate through a model.

Real-world applications (as identified in the paper):

• Anomaly detection, where a spike in one sensor may indicate a fault — a setting where global normalization may amplify the signal of interest.
• Motion recognition, where multiple channels such as spatial axes carry complementary information that global normalization might obscure.
• Time series classification on heterogeneous sensor and activity benchmarks, where the best strategy differs per dataset.
• Long-term forecasting on datasets like ETT-Small, Traffic, and Weather, where the choice of preprocessing produces clear MAE differences.

Industry relevance. Recent time series foundation models such as TimesFM and MOMENT are adapted to many downstream tasks via head-only fine-tuning, full fine-tuning, or parameter-efficient methods like LoRA. If the normalization choice shifts results as much as these experiments suggest, then it is a practical configuration decision for anyone deploying or adapting such models — not a step that can be inherited from a default implementation.

Future Directions

• Design adaptive or task-aware scaling methods. The authors state they do not propose a new normalization method that performs uniformly well; building principled preprocessing that adapts to the data and task is left open.
• Extend the theory beyond the simplified setting. The analysis assumes a single-layer Transformer with self-attention and 1-Lipschitz activations; generalizing the bounds to deeper models and other architectures is a natural next step.
• Automate the selection of a normalization strategy. Since the best choice is context-dependent, a procedure that infers the right strategy from dataset characteristics and task objectives would be valuable.
• Connect expressivity to downstream performance more tightly. The empirical work shows large performance swings, but the link between a specific bound value and a specific accuracy or MAE outcome remains an open question.

Target Audience

Researchers and practitioners working on time series modeling with Transformers, especially those building or fine-tuning time series foundation models and choosing preprocessing configurations. The theoretical sections suit readers comfortable with attention formulations and Lipschitz-based generalization bounds, while the experimental sections are directly useful to applied machine learning engineers who need to justify a normalization choice. Readers looking for a new normalization algorithm will not find one here — the paper's value is diagnostic rather than prescriptive.

Authors’ abstract

Normalization and scaling are fundamental preprocessing steps in time series modeling, yet their role in Transformer-based models remains underexplored from a theoretical perspective. In this work, we present the first formal analysis of how different normalization strategies, specifically instance-based and global scaling, impact the expressivity of Transformer-based architectures for time series representation learning. We propose a novel expressivity framework tailored to time series, which quantifies a model's ability to distinguish between similar and dissimilar inputs in the representation space. Using this framework, we derive theoretical bounds for two widely used normalization methods: Standard and Min-Max scaling. Our analysis reveals that the choice of normalization strategy can significantly influence the model's representational capacity, depending on the task and data characteristics. We complement our theory with empirical validation on classification and forecasting benchmarks using multiple Transformer-based models. Our results show that no single normalization method consistently outperforms others, and in some cases, omitting normalization entirely leads to superior performance. These findings highlight the critical role of preprocessing in time series learning and motivate the need for more principled normalization strategies tailored to specific tasks and datasets.