TimePerceiver: An Encoder-Decoder Framework for Generalized Time-Series Forecasting

AI summary

Overview

Research area: Machine learning for multivariate time-series forecasting, specifically neural architecture design (attention-based encoder-decoder models).

Technical level: Intermediate. The paper assumes familiarity with Transformer attention, patching, and standard forecasting benchmarks, but its central idea — generalizing what counts as a "forecasting" task — is conceptually simple.

Scope: The paper proposes TimePerceiver, a unified attention-based encoder-decoder framework that reframes forecasting as a generalized temporal reconstruction problem covering extrapolation, interpolation, and imputation, and reports state-of-the-art results across eight standard benchmarks.

What This Paper Is About

Most time-series forecasting research has concentrated on designing better encoders — the part of the model that reads past data — while treating how the model decodes predictions and how it is trained as separate, secondary concerns. This paper argues those three pieces should be designed together, and proposes a framework where the model is trained not just to predict the future, but to fill in arbitrary missing or held-out time segments anywhere along the timeline. The goal is a single model that learns richer temporal structure end-to-end, without the usual two-stage pretrain-then-finetune pipeline.

Key Contributions

1. A generalized forecasting formulation. Instead of the standard setup (fixed-length past window → fixed-length future horizon), the paper defines forecasting as predicting any set of target time indices given any set of observed indices. This single objective subsumes extrapolation (future prediction), interpolation (filling gaps between observed points), and imputation (recovering missing values), and the standard formulation becomes a special case.

2. A latent bottleneck encoder. Rather than running full self-attention over all input tokens, a small set of learnable latent tokens first attends to the input, then refines itself through self-attention, then writes information back to the input tokens. This jointly captures temporal and cross-channel dependencies while reducing attention cost from quadratic, O(N²), to linear in the number of latents, O(N·M).

3. A query-based decoder. Each target timestamp gets a learnable query built from its temporal and channel positional embeddings. These queries cross-attend to the encoded input to retrieve only the information relevant to their position, then project to predicted patch values — a mechanism that naturally supports arbitrary, non-contiguous target sets.

4. Strong empirical results plus design analysis. The framework achieves 55 best and 17 second-best scores out of 80 benchmark settings, with the best average rank (1.375 MSE, 1.550 MAE), and the paper includes ablations isolating the formulation, encoder attention choice, and positional-embedding sharing, plus attention-map visualizations.

Main Findings

• State-of-the-art across benchmarks. On eight datasets (ETTh1, ETTh2, ETTm1, ETTm2, Weather, Solar, Electricity, Traffic) and four prediction horizons, TimePerceiver records the best average rank of 1.375 in MSE and 1.550 in MAE. It outperforms the channel-dependent baseline CARD by 5.6% MSE and 4.4% MAE on average, and iTransformer by 8.5% MSE and 7.3% MAE.

• Generalizing the task improves accuracy. Training with the generalized formulation instead of the standard past-to-future setup improved results on every dataset tested, averaging 5.0% better MSE and 3.4% better MAE — for example, ETTh1 MSE dropped from 0.420 to 0.404, and Electricity from 0.169 to 0.157.

• The generalized formulation transfers to other architectures. Swapping PatchTST's linear decoder for TimePerceiver's query-based decoder and training with the generalized objective improved PatchTST on ETTh1 (MSE 0.423 → 0.415), suggesting the idea is not tied to one encoder design.

• Latent bottleneck beats full attention. On ETTh1, latent bottleneck encoding (0.404 MSE) outperformed both joint full self-attention (0.425) and decoupled axis-separated self-attention (0.423). The paper attributes this to lower memory cost and less diluted attention, since joint attention over all tokens spreads focus too thinly.

• Target sampling strategy matters and is dataset-dependent. Three sampling schemes were tested: fully contiguous targets, fully disjoint (random) targets, and mixed (two equal-length segments). All beat the standard formulation, but the best choice varied by dataset — ETTh1 favored contiguous sampling while others favored disjoint or mixed. The mixed strategy was the most reliable default without tuning.

• Shared positional embeddings work better. Using the same temporal and channel positional embeddings in both encoder and decoder generally outperformed separate embeddings, supporting the intuition that encoder inputs and decoder queries must be temporally aligned. The Solar dataset was a minor exception on MAE.

• Attention maps show interpretable structure. Visualizations on ETTh1 and ETTm1 show how latent tokens distribute attention across input patches and how decoder queries align with the input positions most relevant to their predicted timestamps.

Methodology in Plain English

The researchers start by redefining the problem. Instead of "given the last L time steps, predict the next H," they say: "given any set of observed time points, predict any set of unobserved ones." This is done by splitting the timeline into fixed-length patches, randomly choosing some patches as input and the rest as prediction targets, and scoring the model only on the targets.

To make this work, the model needs to handle inputs and outputs at arbitrary positions. The architecture has three parts. First, input patches are converted into embeddings, with separate learned signals added for when a patch occurred (temporal position) and which variable it belongs to (channel position). Second, an encoder compresses these into a small fixed set of latent tokens that attend to the input, refine among themselves, and then feed back to the input tokens — an efficient information bottleneck. Third, a decoder creates one query per target timestamp, again built from temporal and channel positional embeddings, and has each query pull relevant context from the encoded input via cross-attention before projecting to predicted patch values. The whole thing is trained end-to-end with plain mean squared error on the target patches.

Evaluation covers eight real-world datasets from weather, energy, and traffic domains, with results averaged over multiple input lengths (96, 384, 768) and horizons (96, 192, 336, 720) to avoid cherry-picking favorable configurations.

Why This Matters

Impact on research. The paper challenges the field's encoder-centric habit, arguing that task formulation, architecture, and training objective should be co-designed. Its generalized formulation is a reusable idea — any patch-based forecasting model can adopt it by replacing its linear prediction head with a query-based decoder. The latent bottleneck also offers a concrete efficiency argument against full self-attention for multivariate series.

Real-world applications.

• Weather forecasting, where sensor stations report irregularly and missing readings must be filled before numerical models can run.
• Electricity load forecasting, which needs both future demand prediction and gap-filling when smart meters drop out.
• Traffic flow prediction, where detector outages create gaps that must be imputed while simultaneously forecasting congestion ahead.
• Energy and industrial monitoring (e.g., solar generation), where a single model handling past reconstruction, gap-filling, and future prediction reduces the number of specialized systems to maintain.

Industry relevance. The two-stage pretrain-finetune pipeline is expensive to operate. A single model that learns useful representations through one generalized objective, and that handles irregular or missing data natively rather than requiring preprocessing, is operationally simpler. The O(N·M) attention complexity also matters for deployments with many channels (the Traffic dataset has hundreds), where quadratic attention becomes impractical.

Future Directions

• Automatic sampling strategy selection. The paper shows the best target-patch sampling scheme varies by dataset, but leaves it as a hyperparameter. A principled or learned method for choosing it — possibly conditioned on dataset statistics — is an open problem.

• Scaling to very large channel counts and sequence lengths. The latent bottleneck reduces cost but the number of latent tokens M is a fixed hyperparameter. How M should scale with the number of channels, and whether the approach holds at foundation-model scale, remains untested.

• Extending beyond numeric multivariate series. The formulation is domain-agnostic in principle. Applying it to multivariate streams with irregular real-world missingness (not synthetic masking), or to domains like medical monitoring and financial tick data, would test how far the generalization carries.

• Unifying with time-series foundation models. Since the generalized formulation turns forecasting into a masked-reconstruction task, it raises the question of whether this objective could serve as the pretraining recipe for a general-purpose forecasting foundation model, rather than as an architecture-specific training scheme.

Target Audience

Researchers and graduate students working on time-series forecasting or sequence modeling, particularly those familiar with Transformer-based architectures such as PatchTST and iTransformer. Practitioners building forecasting systems for energy, traffic, or weather who face missing data and want one model instead of separate imputation and prediction pipelines will also find the formulation directly useful. Readers without background in attention mechanisms will need to consult the preliminaries section, but the core conceptual shift — what counts as the prediction task — is accessible to anyone who understands standard forecasting.

Authors’ abstract

In machine learning, effective modeling requires a holistic consideration of how to encode inputs, make predictions (i.e., decoding), and train the model. However, in time-series forecasting, prior work has predominantly focused on encoder design, often treating prediction and training as separate or secondary concerns. In this paper, we propose TimePerceiver, a unified encoder-decoder forecasting framework that is tightly aligned with an effective training strategy. To be specific, we first generalize the forecasting task to include diverse temporal prediction objectives such as extrapolation, interpolation, and imputation. Since this generalization requires handling input and target segments that are arbitrarily positioned along the temporal axis, we design a novel encoder-decoder architecture that can flexibly perceive and adapt to these varying positions. For encoding, we introduce a set of latent bottleneck representations that can interact with all input segments to jointly capture temporal and cross-channel dependencies. For decoding, we leverage learnable queries corresponding to target timestamps to effectively retrieve relevant information. Extensive experiments demonstrate that our framework consistently and significantly outperforms prior state-of-the-art baselines across a wide range of benchmark datasets. The code is available at https://github.com/efficient-learning-lab/TimePerceiver.