UrbanAI 2025 Challenge: Linear vs Transformer Models for Long-Horizon Exogenous Temperature Forecasting

AI summary

Overview

Research area: Time series forecasting for smart buildings, specifically benchmarking linear models against Transformer-family architectures for long-horizon temperature prediction under an exogenous-only constraint.

Technical level: Intermediate.

Scope: This paper reports an empirical, head-to-head comparison of six forecasting models (Linear, NLinear, DLinear, Transformer, Informer, Autoformer) submitted to the UrbanAI 2025 Challenge, evaluating them on training, validation, and official blind test splits.

What This Paper Is About

The UrbanAI 2025 Challenge asks participants to forecast indoor temperature over long horizons using only exogenous inputs — weather and setpoint variables — because ground-truth temperatures are unavailable at inference time in production settings. The author's goal is to test whether attention-based architectures (Vanilla Transformer, Informer, Autoformer) justify their added complexity in this constrained regime, or whether simple linear baselines (Linear, NLinear, DLinear) generalize better. The paper answers this with a controlled comparison using identical train/validation/test splits and standardized metrics.

Key Contributions

1. A direct, apples-to-apples comparison of three linear baselines (Linear, NLinear, DLinear) against three Transformer-family architectures (Vanilla Transformer, Informer, Autoformer) under the LTSF-Linear evaluation protocol, adapted to the challenge's exogenous-only rule.
2. A demonstration that linear models dominate in the long-horizon, exogenous-only regime on the official blind test set, with NLinear achieving the best reported test accuracy (MAE 0.2461, MSE 0.5220).
3. Evidence of a generalization gap in Transformer-family models: the Vanilla Transformer recorded the lowest training MSE in the table (0.0330) yet collapsed on the official test set (MSE 1.4371).
4. A fully documented benchmark setup (horizons, optimizer settings, normalization, metrics, compute environment) intended to support reproducibility for future challenge baselines.

Main Findings

• NLinear was the strongest model overall. It recorded the best validation metrics (MAE 0.2529, MSE 0.4665) and the best official test metrics (MAE 0.2461, MSE 0.5220) among the reported entries. The paper states it achieved the best overall accuracy across all splits; note that the training-split table lists the Vanilla Transformer with a lower training MSE (0.0330) than NLinear (0.0619).
• DLinear was consistently second among linear models. Validation MAE 0.2784 / MSE 0.4744; official test MAE 0.2811 / MSE 0.5489. The paper describes it as underperforming NLinear by a small margin.
• Linear was the weakest of the three linear baselines but still beat the Transformer family. Validation MAE 0.2843 / MSE 0.4882; official test MAE 0.2862 / MSE 0.5634.
• The Vanilla Transformer generalized poorly. Training MSE 0.0330, validation MAE 0.3375 / MSE 0.5302, but official test MAE 0.8342 / MSE 1.4371 — the paper describes this as a collapse under the exogenous-only setting.
• Informer underperformed across the board. Its training and validation MAE and MSE are reported as missing (dashes) in Table 1, and its official test numbers match the Transformer exactly (MAE 0.8342, MSE 1.4371). The paper states it matched the Transformer's weak test metrics.
• Autoformer landed between the two groups. Training MSE 0.0844; validation MAE 0.3493 / MSE 0.6201; official test MAE 0.3598 / MSE 0.7368 — better than Transformer and Informer on test, worse than all three linear baselines.
• Best-to-worst on the official test set (MAE): NLinear 0.2461, DLinear 0.2811, Linear 0.2862, Autoformer 0.3598, then Transformer and Informer tied at 0.8342.

Methodology in Plain English

The author took a single dataset from the Smart Buildings Control Suite (SBCS) and split it into a training block of 36,105 samples, a validation block of 10,275 samples used only for early stopping, and an official blind test set whose labels were never accessed during model design or tuning. Technical issues meant only 10,563 of the intended (larger) test samples were actually evaluated.

Every model was given the same setup: 96 consecutive exogenous input steps (weather, setpoints) to predict the next 96 temperature steps, roughly four days in and four days out, matching a direct multi-step forecasting scheme. Features were normalized per variable using training-set statistics, and all errors are reported in that normalized space rather than physical temperature units, following the LTSF-Linear protocol.

The three linear models are structurally simple. Linear is a single layer projecting the input window to the horizon. NLinear first subtracts the last input timestep from the whole window, applies a learnable linear projection, then adds that last value back — a centering trick for robustness to nonstationarity. DLinear splits the series into a trend component (estimated by moving average) and a seasonal component, then forecasts each with its own linear layer.

The Transformer-family models use their published designs: the Vanilla Transformer uses encoder–decoder multi-head self-attention with sinusoidal positional encodings and teacher-forced decoding; Informer uses ProbSparse attention with a distillation operation and a generative decoder; Autoformer uses an auto-correlation mechanism with built-in seasonal-trend decomposition. Hyperparameters were swept within stated ranges — d_model in {256, 512}, heads in {4, 8}, encoder/decoder layers in {2, 3}, dropout in [0.05, 0.2] for the Transformer; factor in {3, 5} for Informer; decomposition kernel sizes in {3, 5, 7} for Autoformer. All Transformer-family runs used Adam with learning rate 1e-4 or 5e-4, weight decay 1e-4, batch size in {16, 32}, and early stopping on validation MAE with patience 3. Everything was implemented in PyTorch and run on a Google Colab environment with an NVIDIA T4 GPU.

Primary metrics are MAE and MSE; for probabilistic submissions the challenge uses KL divergence, though the paper's own results are point predictions.

Why This Matters

This paper is a cautionary data point for a widely observed pattern in forecasting research: added architectural capacity does not automatically translate into better generalization. In an exogenous-only, long-horizon setting, a one-layer linear projection with a centering trick beat three attention-based designs on a blind test set — by a large margin against the Transformer and Informer (MAE 0.2461 vs 0.8342). The comparison also shows how strongly a model can look good in training and validation while failing on held-out data, which matters for how benchmarks are reported.

Real-world applications:

• Smart building HVAC control: indoor temperature forecasts drive heating and cooling setpoints, and these models must run without observed indoor temperature at inference time.
• Energy optimization and cost reduction: accurate temperature prediction supports scheduling equipment to reduce consumption, as the paper notes in its broader-impacts discussion.
• Occupant comfort management: maintaining comfortable conditions while minimizing energy use, which the paper identifies as a primary motivation.
• Carbon reduction at scale: the paper cites improved forecasting as a route to energy savings and lower emissions across building portfolios.

Industry relevance: Buildings are instrumented with abundant exogenous sensors (weather feeds, setpoints, occupancy schedules) but do not always expose reliable internal temperature readings at prediction time. A model that performs well under that constraint, and that runs on modest hardware such as a Google Colab NVIDIA T4 GPU, is directly deployable in building management systems where interpretability and low compute cost are operational advantages.

Future Directions

1. Add nonlinearity carefully. The conclusion proposes integrating modest nonlinearity as a next step, building on linear models rather than replacing them.
2. Apply transfer learning. The paper names transfer learning as a promising direction for leveraging growing datasets across buildings.
3. Report statistical variability. The paper's own checklist acknowledges that no error bars or significance statistics are reported, and states that future work will add variability across seeds and runs.
4. Complete the benchmark and release artifacts. Only 10,563 of a larger official test set were evaluated due to technical issues, and the checklist states that code, configs, and dataset instructions will be released in anonymized supplemental material.
5. Explore probabilistic outputs. The challenge supports histograms and mean/std submissions scored by KL divergence, but this paper reports only point-prediction MAE and MSE.

Target Audience

Researchers and practitioners working on time series forecasting, particularly those interested in the ongoing debate over whether Transformer-based architectures are necessary for long-horizon prediction. It is also useful for challenge participants and organizers who need strong, reproducible baselines, and for building-automation engineers who require accurate temperature forecasts from exogenous signals alone on limited compute. Readers looking for a rigorous theoretical treatment should look elsewhere: this is an empirical benchmark comparison with no new theorems.

Authors’ abstract

We study long-horizon exogenous-only temperature forecasting - a challenging univariate setting where only the past values of the indoor temperature are used for prediction - using linear and Transformer-family models. We evaluate Linear, NLinear, DLinear, Transformer, Informer, and Autoformer under standardized train, validation, and test splits. Results show that linear baselines (Linear, NLinear, DLinear) consistently outperform more complex Transformer-family architectures, with DLinear achieving the best overall accuracy across all splits. These findings highlight that carefully designed linear models remain strong baselines for time series forecasting in challenging exogenous-only settings.