Benchmarking Catastrophic Forgetting Mitigation Methods in Federated Time Series Forecasting

AI summary

Overview

• Research area: Continual learning (CL) and federated learning (FL) applied to time series forecasting, specifically the problem of catastrophic forgetting (CF) in decentralized, non-i.i.d. regression settings.
• Technical level: Intermediate. The paper is readable for someone familiar with basic machine learning and neural network training, but assumes comfort with federated aggregation, regularization losses, and error metrics such as RMSE.
• Scope in one sentence: The authors build and release the first benchmarking framework for CF mitigation in Federated Time Series Forecasting (FTSF), then compare Replay, Elastic Weight Consolidation, Online EWC, Learning without Forgetting, and Synaptic Intelligence against a naive fine-tuning baseline across 12 clients and 11 seasonal tasks on the Beijing Multi-Site Air Quality dataset.

What This Paper Is About

Most work on catastrophic forgetting — where a model loses earlier knowledge as it learns new tasks — has been done on image classification in centralized settings. The authors argue that the regression-based forecasting setting common in IoT and edge deployments, combined with federated learning's non-i.i.d. client data, has no systematic benchmark at all. Their goal is to provide that benchmark: a reproducible pipeline that partitions real multivariate environmental time series into chronologically ordered seasonal tasks per client, trains LSTM forecasters under FedAvg, and measures forgetting, plasticity, overall performance, and compute cost for several CL methods.

Key Contributions

1. A new benchmark for CF in federated time series forecasting. The framework integrates FL, CL, and multivariate regression, and supports lagged sequence generation, client-wise partitioning, and seasonal task scheduling. The code is publicly released at https://github.com/khaledhallakk/cf-federated-timeseries.
2. Adaptation and evaluation of core CL methods in a continual forecasting pipeline. Replay, EWC, SI, and LwF (the latter labeled as KD in the results) are adapted to LSTM-based forecasting across decentralized clients.
3. Fine-grained evaluation metrics. The paper introduces task-wise measurement of average forgetting, average plasticity, and computation time (CPU) to expose the stability–plasticity trade-off.
4. Validation on real-world non-stationary, non-i.i.d. data. The framework is exercised on a multivariate environmental dataset with heterogeneous client distributions and three target variables of differing temporal character.

Main Findings

• Continual adaptation beats a static model. Using the Temperature target as an illustration, the paper reports that Naive-CL consistently outperforms the static BaseModel (trained once and never updated), and the gap widens as more tasks are introduced.
• Naive fine-tuning forgets heavily. Average Forgetting (AF) values for NaiveCL rise with target difficulty: 1.32 for Temperature, 5.25 for PM2.5, and 8.44 for Wind Speed. Its Average Plasticity (AP) values are 43.28, 80.52, and 136.69 respectively.
• Replay is the most consistently effective method. It achieves the lowest forgetting across all three targets, with negative AF values in every case (−0.63 for Temperature, −0.03 for PM2.5, −0.16 for Wind Speed), which the authors describe as positive backward transfer.
• Replay costs the most compute. Replay uses 673 ± 40 CPU time units on Temperature, 730 ± 21 on PM2.5, and 690 ± 20 on Wind Speed, exceeding the next most expensive method by over 20%. NaiveCL is the cheapest (442 ± 34, 437 ± 30, 428 ± 32).
• EWC and O-EWC trade plasticity for stability. Both reduce forgetting, but EWC records the weakest plasticity and average performance of the evaluated methods. The paper attributes O-EWC's better plasticity to its online Fisher information matrix updates.
• LwF (KD) and SI offer a middle ground. Both show slightly higher forgetting than Replay while preserving good plasticity, with SI reaching AF of 0.01 ± 0.03 on Temperature and 0.02 ± 0.03 on PM2.5.
• Optimal replay buffer size is target-specific. The best replay sample ratio was 15% for Temperature, 20% for PM2.5, and 25% for Wind Speed; larger ratios degraded plasticity.
• Hyperparameters were tuned per method and target. Selected values included λ_KD of 30/60/120, λ_SI of 3/12/14, λ_Replay of 0.6/0.7/0.8 (Temperature/PM2.5/Wind Speed), λ_O-EWC of 0.8×10^6, 1.5×10^6, and 1.3×10^6, and λ_EWC of 10^10, 10^6, and 10^8. The stated search goal was an AF below 0.1.

Methodology in Plain English

The researchers treat each air quality monitoring station in the Beijing Multi-Site Air Quality dataset as a separate client in a federated system, which naturally produces heterogeneous local data. Each client's history (March 1, 2013 to February 28, 2017) is cut into chronological, mutually exclusive time windows: one base task spanning March 2013 to March 2014 that covers all four seasons, followed by 11 continual tasks, each corresponding to a single season between 2014 and early 2017. Data from a finished task is treated as unavailable unless deliberately retained, mirroring privacy and storage limits.

Preprocessing is uniform across clients: forward and backward fill for missing values, sine/cosine cyclical encoding of periodic features such as hour and day of week, and robust min–max scaling based on the 1st and 99th percentiles computed globally. Inputs are lagged windows of 12 past hourly steps forecasting 6 future steps, with an 80%–20% temporal train/test split per task.

Training happens in two stages. First, all clients jointly train a shared LSTM with Federated Averaging over 500 communication rounds (1 local epoch per round) on the base task. That model initializes every subsequent strategy. Second, each client runs through the 11 seasonal tasks in order, training 30 communication rounds per task with 1 local epoch each, aggregating via FedAvg, and applying the CL mechanism under test.

Performance is recorded as a matrix where each entry is the RMSE of a model trained up to task i evaluated on task j. From this matrix the authors compute Average Forgetting, Average Plasticity, and Average Performance, all scaled by 10^3 for readability. Each experiment is repeated over five random seeds, and results are reported as mean ± standard deviation.

Why This Matters

• Research impact: The paper fills a stated gap — no prior work had proposed a rigorous benchmarking setup targeting catastrophic forgetting in federated time series forecasting. It gives the field a shared protocol, standardized metrics, and open code for comparing CL methods in a regression-based, decentralized, temporally ordered setting rather than the i.i.d. classification benchmarks (Split-MNIST, class-incremental CIFAR-100) that dominate the literature.
• Real-world applications:
  • Environmental monitoring, where a model that forgets wintertime pollution patterns after training on summer data may fail to trigger timely public health interventions.
  • Smart infrastructure and smart cities, where sensors produce seasonally drifting data across geographically dispersed sites.
  • Healthcare, where long-horizon historical patterns retain diagnostic or predictive value and data cannot be centralized.
  • Energy and industrial monitoring, where demand and process patterns shift seasonally and client revisitation is rare.
• Industry relevance: The paper explicitly targets IoT, edge-AI, and privacy-constrained deployments. Its compute-time metrics are directly relevant to practitioners on constrained hardware, and its finding that Replay is the strongest but most expensive method — while relying on stored data that raises privacy and scalability concerns in federated settings — matters for anyone designing production systems where raw data cannot be retained.

Future Directions

1. Replay-free methods that match Replay's retention. The authors call for novel CL strategies that achieve Replay-level memory retention without storing any old data, plus exploration of latent replay, prototype memories, and lightweight generative surrogates.
2. Dynamic, task-aware CL. Adaptive replay strategies that adjust buffer size based on task difficulty, data drift, or information gain, and methods tailored specifically to the sequential structure of time series.
3. Automated hyperparameter selection. The paper notes that method performance depends heavily on tuning and suggests principled sensitivity analysis or meta-learning to set these values automatically.
4. Broader benchmarking protocols. Extending evaluation to longer prediction horizons, variable input lengths, client-specific adaptations, and explainable mechanisms for interpreting forgetting dynamics, along with standardized open toolkits.

Target Audience

This paper is most useful to researchers working on continual learning, federated learning, or both, especially those moving beyond image classification into regression and forecasting. It also serves applied engineers building privacy-preserving forecasting systems for IoT, edge, environmental, or industrial monitoring, and benchmark-focused researchers who need a reproducible protocol and released code for comparing forgetting-mitigation methods under non-i.i.d., temporally ordered data.

Authors’ abstract

Catastrophic forgetting (CF) poses a persistent challenge in continual learning (CL), especially within federated learning (FL) environments characterized by non-i.i.d. time series data. While existing research has largely focused on classification tasks in vision domains, the regression-based forecasting setting prevalent in IoT and edge applications remains underexplored. In this paper, we present the first benchmarking framework tailored to investigate CF in federated continual time series forecasting. Using the Beijing Multi-site Air Quality dataset across 12 decentralized clients, we systematically evaluate several CF mitigation strategies, including Replay, Elastic Weight Consolidation, Learning without Forgetting, and Synaptic Intelligence. Key contributions include: (i) introducing a new benchmark for CF in time series FL, (ii) conducting a comprehensive comparative analysis of state-of-the-art methods, and (iii) releasing a reproducible open-source framework. This work provides essential tools and insights for advancing continual learning in federated time-series forecasting systems.