Paloma: Phase-Conditioned Residual Modulation for Time Series Forecasting

AI summary

Overview

Research area: Time series forecasting, specifically periodicity-aware modeling and signal-processing-inspired neural architecture design.

Technical level: Intermediate. The paper assumes familiarity with forecasting benchmarks (ETT, Electricity, Traffic, Solar-Energy, Weather, PEMS), residual decomposition, and basic linear algebra (covariance, orthogonalization, rotation), but its central idea can be grasped without deep signal-processing background.

Scope: Paloma argues that the residual left after removing a learned periodic component is still structured and phase-dependent, and introduces two lightweight phase-conditioned modules — rotation and amplitude scaling — that model those deviations and can also be plugged into other forecasting architectures.

What This Paper Is About

Many efficient forecasters (e.g., CycleNet) decompose a series into a learnable repeating "cycle" plus a residual, then hand the residual to a simple linear model or shallow MLP on the implicit assumption that it is unstructured noise. The authors show — visually on real data and through two propositions — that this residual still varies systematically depending on where you are within the cycle, in both direction and magnitude. Paloma's goal is to model that leftover structure explicitly, at low cost, and in a way that transfers to existing backbones.

Key Contributions

1. An empirical and theoretical case against "residual = noise." The authors visualize learned CycleNet cycles and residual mean/standard deviation heatmaps on the Electricity dataset (cycle length 168), plus a counterfactual Gaussian-noise control, and prove in Proposition 2 that the phase-conditional covariance of the residual equals that of the raw signal — so second-order phase structure survives even an optimal MSE-fitted prototype.

2. A Phase Rotation module. Residual features are projected into two latent representations, orthogonalized via Gram–Schmidt, and rotated by a phase-conditioned angle Θ = π·tanh(Linear(P̃)) derived from a learnable phase control matrix P ∈ ℝ^{W×N}.

3. An Amplitude Scaling module. A learnable amplitude control matrix A ∈ ℝ^{W×N} produces phase-conditioned scale (Sigmoid) and shift terms applied affinely to an embedded residual, capturing magnitude variation across phases.

4. A reusable plugin ("Paloma technique") and full evaluation. The two modules are abstracted into a component that can be inserted into other architectures and applied directly to input sequences; the paper reports experiments on twelve datasets, ablations on six, five-seed stability runs, and plugin experiments on four backbones.

Main Findings

• Residuals are phase-structured, not noise. In Figure 1(b), even where the learned cycle closely matches the empirical mean, residual mean and standard deviation still vary by phase, notably near phase indices 20, 90, and 140. The Gaussian-noise control in Figure 1(c) shows a nearly phase-invariant standard deviation, unlike the real data, which rises around peak regions.

• Theory explains why this persists. Proposition 1 shows the MSE-optimal prototype is the phase-conditional mean, Q*_k = E[x_l | i_l = k]. Proposition 2 shows the resulting residual has E[r_l | i_l = k] = 0 but Cov(r_l | i_l = k) = Cov(x_l | i_l = k), so phase-dependent covariance is preserved exactly. An MAE counterpart is given in Appendix B.3.

• State-of-the-art aggregate performance. With lookback T = 96 and results averaged over four horizons, Paloma records the best result in 7 of 24 MSE slots and 11 of 24 MAE slots across the twelve datasets — e.g., ETTm1 0.363 MSE / 0.372 MAE, ETTh2 0.361 / 0.386, Electricity 0.166 / 0.251, Weather 0.231 / 0.262, Solar-Energy 0.197 / 0.234, PEMS03 0.094 / 0.194, PEMS04 0.085 / 0.182, PEMS07 0.078 / 0.164, PEMS08 0.111 / 0.194.

• A known weak spot on Traffic. Paloma reports 0.461 MSE / 0.260 MAE on Traffic — the best MAE but a noticeably higher MSE than RAFT (0.414), DistDF (0.417), FreDF (0.421), and iTransformer (0.428). The authors attribute this to complex spatiotemporal correlations and lag effects that a channel-independent design does not model as well as channel-interaction methods.

• Both modules matter, and they are complementary. On ETTm1, removing Phase Rotation gives 0.368 MSE / 0.374 MAE, removing Amplitude Scaling gives 0.369 / 0.374, and removing both gives 0.376 / 0.380, versus 0.363 / 0.372 for full Paloma. The same monotone degradation appears on ETTh1, Electricity, Weather, Traffic, and Solar-Energy.

• The plugin transfers across architectures. Applied on Electricity with lookback 96, Paloma technique improves iTransformer by 6.25% MSE / 2.62% MAE on average over four horizons, PatchTST by 9.85% / 5.23%, FilterNet by 5.78% / 2.24%, and DLinear by 19.76% / 8.63%. Gains are largest on short-horizon PEMS: DLinear improves by 62.57% MSE / 43.56% MAE on PEMS04 and 57.74% / 36.70% on PEMS03 (averaged across horizons 12, 24, 48, 96).

• Stability. Standard deviations across five runs with different random seeds are reported in Table 7, supporting the claim of robust performance.

• Linear cost. Overall complexity is O(N(TD + D² + Dτ)) — linear in lookback T and channels N.

Methodology in Plain English

The pipeline starts by normalizing each input series (instance normalization) and assigning every time step a phase index — its position within a cycle of known length W, computed as i_l = [(t₀ mod W) + l] mod W.

Three learnable tables are indexed by that phase: a prototype matrix Q holding the reference pattern per phase, a phase control matrix P, and an amplitude control matrix A. The residual is simply R = X − Q̃, where Q̃ is the phase-aligned lookup of Q.

The residual then goes down two parallel paths. In the Phase Rotation path, R is linearly mapped into two hidden representations, one of which is made orthogonal to the other by subtracting its projection (Gram–Schmidt). The lookup from P produces a phase-dependent angle (bounded to ±π via tanh and scaled by π), and the two representations are mixed by a standard cosine/sine rotation. This lets the model turn the feature geometry differently at different points in the cycle. In the Amplitude Scaling path, the lookup from A generates a per-phase multiplicative gate and additive offset applied to an embedded residual, then a linear projection — capturing how strongly the signal deviates at each phase.

The two outputs are summed, passed through a linear layer plus GELU, added to an embedding of the raw input as a shortcut, projected to the forecast horizon, and de-normalized. Because the two modules only need a phase-indexed lookup and a few linear maps, they can be lifted out as the "Paloma technique" and inserted into other models, either on residuals or directly on the input.

Why This Matters

For research: The paper challenges a widely used design shortcut — "model the cycle, let a linear layer handle the rest." Proposition 2 gives a clean reason why that shortcut leaks information, and the modulation framing offers an alternative to frequency filtering (FilterNet, TimeEmb, FreqCycle) and phase tokenization (PhaseFormer) as a way to use phase information. The plugin result also provides a cheap, architecture-agnostic upgrade path for existing baselines.

Real-world applications:

• Energy and electricity load forecasting, where daily and weekly cycles dominate but deviation magnitude swings with peak hours — the paper's own motivating example uses the Electricity dataset.
• Traffic flow prediction, where PEMS results show strong short-horizon gains (0.085 MSE on PEMS04 versus 0.119 for CycleNet).
• Solar and renewable generation forecasting, where the Solar-Energy dataset is used with a reported periodicity of 144.
• Weather and sensor-network monitoring, where the Weather benchmark is used and where structured residual variance around peaks would matter for risk-aware planning.

Industry relevance: The linear complexity in both lookback length and channel count, plus the fact that the modules drop into existing backbones with no architectural redesign, makes this attractive for production forecasting where retraining cost and latency matter. The DLinear gains (19.76% MSE on Electricity, over 57% on PEMS04/PEMS03) suggest the largest payoff is on already-cheap linear models.

Future Directions

• Channel interaction. Paloma's channel-independent design is the stated reason for its weaker Traffic MSE. Extending phase-conditioned modulation to cross-channel structure — or combining it with attention across channels — is the obvious next step.

• Choosing the cycle length W. The method requires W as a priori input (168 for Electricity, 144 for Solar-Energy in the visualizations). How to learn or select W automatically, and how sensitive results are to misspecification, is not resolved in the visible content.

• Beyond second-order structure. The theory covers the conditional mean and covariance. Whether phase-conditioned higher-order or cross-phase dependencies carry additional exploitable signal is left open.

• Where in the network to inject the plugin. The reported experiments apply Paloma technique to whole input sequences or residual streams in four baselines; whether placement (per-layer, pre- vs. post-decomposition) matters systematically is not established in the content available.

Target Audience

Forecasting researchers and practitioners working on efficient, periodicity-aware models — especially those building on CycleNet, DLinear, or SparseTSF-style decompositions. It will also interest readers exploring signal-processing-inspired inductive biases (modulation, phase, filtering) in deep learning, and engineers who want a low-cost add-on to improve an existing production forecaster without replacing it. Readers needing implementation specifics (hidden dimension D, training schedule, full hyperparameters) should consult the appendices, which are referenced but not included in the accessible content; the t-SNE visualization analysis in Section 5.4 is also truncated in the provided text.

Authors’ abstract

Explicitly modeling periodicity has proven to be an efficient approach for time series forecasting, where the residual, obtained after removing the periodic component from the time series, is typically treated as unstructured noise. However, in this work, we show that such residuals still exhibit structured, \textit{phase-dependent} deviations, as supported by both empirical observations and theoretical analysis. This suggests that residuals should not be treated merely as unstructured noise, but instead as \textit{phase-conditioned signals} whose variations arise in both direction and magnitude. Motivated by this perspective and inspired by phase and amplitude modulation in classical signal processing, we propose \textit{Paloma}, a simple yet effective framework for modeling residual deviations. Specifically, Paloma comprises two complementary modules: \textit{a Phase Rotation module}, which performs phase-conditioned rotations to capture directional changes in the feature space, and \textit{an Amplitude Scaling module}, which applies phase-conditioned affine transformations to model magnitude variations. We further abstract the two modules into a general-purpose plugin, termed the \textit{Paloma technique}, which can be readily integrated into existing forecasting architectures to modulate input sequences in a phase-aware manner, enabling backbone models to capture phase-dependent variations in both direction and magnitude. Extensive experiments on twelve datasets demonstrate that Paloma achieves state-of-the-art and robust forecasting performance. Additional results show that the Paloma technique consistently improves a wide range of baseline models, validating its effectiveness as a general-purpose component. Our Code is available at \url{https://github.com/TSTS13/Paloma}.