Benchmarking Probabilistic Time Series Forecasting Models on Neural Activity

AI summary

Overview

Research area: Machine learning for time series forecasting, applied to systems neuroscience (mouse cortical neural activity).

Technical level: Intermediate.

Scope: A systematic benchmark of probabilistic forecasting models — from classical statistical methods to deep learning architectures and time series foundation models — on spontaneous mouse cortical activity recorded with widefield calcium imaging, evaluated across multiple forecast horizons and with explicit uncertainty quantification.

What This Paper Is About

Deep learning has driven rapid progress in general time series forecasting, but neural activity has been largely absent from established forecasting benchmarks, and the few neuroscience-specific forecasting models that exist rarely compare themselves against the broader forecasting literature. This paper asks whether the models that win on general benchmarks (economics, energy, weather, retail) also win on brain data, which differs sharply in sampling rate, structure, and noise. The authors run a head-to-head comparison of probabilistic forecasters on spontaneous mouse cortical activity and measure both accuracy and the reliability of the predicted uncertainty.

Key Contributions

1. A systematic benchmark of 12 probabilistic forecasting models plus 2 baselines on neural data. The evaluated methods span four classical statistical models (AR, ARIMA, AR-HMM, Theta), six deep learning models (DeepAR, DLinear, TFT, PatchTST, TiDE, WaveNet), and two time series foundation models (Chronos, Moirai, evaluated both zero-shot and fine-tuned), alongside Naive and Average baselines.

2. Evaluation across multiple forecast horizons rather than a single fixed horizon. The task predicts activity over [i, i+L) from history [i-H, i) for L = 18, 35, and 70 steps, corresponding to roughly 0.5, 1, and 2 seconds, with H tuned per model and per L.

3. Explicit uncertainty quantification via a quantile-based metric. Performance is reported as Mean Weighted Quantile Loss (MWQL), and the authors additionally track how the standard deviation of the predicted distribution relative to the training data standard deviation evolves over forecast steps.

4. A finding about the practical limit of neural activity predictability with current models. The authors identify a horizon beyond which even the best model is no better than predicting the mean and standard deviation of the training data.

Main Findings

• Deep learning models beat classical methods across horizons. PatchTST, TiDE, and fine-tuned Chronos consistently achieved higher accuracy than classical approaches at every prediction horizon tested. All methods outperformed the Naive and Average controls in aggregated performance.

• The best forecasts stay informative out to roughly 1.5 seconds. Forecasts were most reliable within 35 steps (1 second), and informative predictions may extend to about 50 steps (1.5 seconds). Beyond that, even PatchTST performed no better than predicting the mean and standard deviation of training data.

• Foundation models transferred poorly zero-shot. Chronos and Moirai, pretrained on general-domain data (weather, finance, transportation, and others), performed badly on neural activity without fine-tuning, suggesting neural time series have characteristics distinct from their pretraining data. Chronos became competitive after fine-tuning on neural activity, indicating its architecture is flexible enough to capture neural dynamics.

• Autoregressive and transformer errors converge toward the mean-prediction baseline. Naive performance continued to worsen with horizon, while AR and PatchTST converged toward the Average model's score. PatchTST and AR losses exceeded 95% of the Average model loss after 1.80 s and 1.28 s respectively. This convergence is expected for AR models on stationary series, where the h-step-ahead forecast converges to the mean as h grows.

• PatchTST's advantage is horizon-dependent at the step level. Aggregate scores across all steps can mask where the advantage comes from. For the 70-step horizon, the Average model outperformed Naive but still not the other models, so a model's aggregate win may stem from better performance at earlier steps.

• Predicted uncertainty grows with horizon. For both AR and PatchTST, the ratio of predicted distribution standard deviation to training data standard deviation increased with forecast horizon and approached 1 — again expected for AR. The authors note narrower intervals are not automatically better, since they may miss sources of uncertainty.

• Models miss large fluctuations. At the level of individual test samples, some features of activity were captured, but large fluctuations existed that no model predicted.

• PatchTST's strong showing echoes general benchmarks. The authors note its performance aligns with a recent benchmarking study on time series from domains such as economics, energy, and retail.

• Naive and Average were close to each other. Theta and Average performances are omitted from the main comparison figure because they are close to Naive and worse than all other models shown.

Methodology in Plain English

The authors recorded spontaneous cortical activity from five mice across five experimental sessions using widefield calcium imaging at an effective rate of 35 Hz (the true camera rate is 70 Hz, with alternating 405 nm and 470 nm excitation used for hemodynamic correction). Sessions lasted 24.5 ± 3.7 minutes, corresponding to 51,495 ± 7,784 timesteps. Data were registered to the Allen Mouse Brain Common Coordinate Framework (CCFv3), and average activity traces were extracted from four brain regions: somatosensory (SS), motor (MO), visual (VIS), and retrosplenial (RSP). Three sessions were purely spontaneous activity; in the other two, 10% sucrose rewards were given at random 2–5 second intervals to keep mice awake and alert, and any receptive-field mapping was performed before the analyzed window.

Each time series was split chronologically: the earliest 60% of timesteps for training, the next 20% for validation, and the final 20% for testing. Validation and test samples used sliding windows with non-overlapping targets, so forecast targets stayed fixed while history length varied. The authors tested L = 18, 35, and 70 steps, treating history length H as a hyperparameter tuned for each L and each model. Results were aggregated over test samples and reported in MWQL, with error bars reflecting mean ± standard deviation across five random seeds. Statistical comparisons used one-sided paired t-tests with p < 0.05 against the AR model chosen by validation MWQL (AR(valQL)), except where noted otherwise.

Why This Matters

Impact on research: Neural activity has been largely missing from forecasting benchmarks, and probabilistic uncertainty quantification has been essentially unexplored for neural time series despite substantial measurement and intrinsic noise in neural data. This work imports strong backbones from the forecasting literature and gives the neuroscience community a tested starting point, while also highlighting that zero-shot transfer from general-domain foundation models does not work here — a caution for anyone planning to apply such models to brain data.

Real-world applications:

• Closed-loop neural control: Accurate short-horizon forecasts are directly applicable to designing closed-loop control systems, particularly given optogenetic tools that allow targeted manipulation of mouse brain activity.
• Therapeutic brain stimulation: Forecasting could inform stimulation paradigms for clinical intervention.
• Brain–machine interfaces: Prediction of upcoming neural state is central to BMI design.
• Early intervention for neurological disorders: Forecasting methods may support earlier detection or intervention strategies.

Industry relevance: The paper's core technical question — whether general-purpose forecasting architectures and foundation models transfer to a new, structurally different data domain — is directly relevant to any organization deploying pretrained time series models. The result that zero-shot foundation models underperform but fine-tuned ones become competitive is a useful data point for practitioners deciding between pretraining transfer, fine-tuning, and training from scratch.

Future Directions

• Develop a neural time series forecasting foundation model. The authors propose combining the strong backbones identified here (PatchTST, TiDE, fine-tuned Chronos) with neuroscience-specific innovations such as cross-subject training.
• Determine whether the forecast horizon limit is a modeling or a data property. It remains an open question whether the ~1.5 second informative limit comes from design constraints of existing methods or reflects intrinsic variability and time scales in neural activity.
• Use closed-loop control experiments as a probe of neural temporal structure. Control experiments informed by forecasting performance may help reveal temporal structure inherent to the neural system.
• Generalize beyond the current data setting. The benchmark covers univariate forecasts from four averaged regions; extension to other modalities such as Neuropixels single-neuron recordings, behavior tracking, cell-type labeling, and connectivity mapping is framed as a route to expand what forecasting can address.

Target Audience

Neuroscientists and machine learning researchers interested in neural dynamics, closed-loop control, and brain–machine interfaces; time series forecasting practitioners curious about how standard architectures behave outside the usual benchmark domains; and engineers evaluating whether pretrained forecasting foundation models can be transferred to specialized scientific data. Readers need only a working familiarity with forecasting concepts such as horizons, quantile loss, and autoregressive versus direct multi-step prediction.

Authors’ abstract

Neural activity forecasting is central to understanding neural systems and enabling closed-loop control. While deep learning has recently advanced the state-of-the-art in the time series forecasting literature, its application to neural activity forecasting remains limited. To bridge this gap, we systematically evaluated eight probabilistic deep learning models, including two foundation models, that have demonstrated strong performance on general forecasting benchmarks. We compared them against four classical statistical models and two baseline methods on spontaneous neural activity recorded from mouse cortex via widefield imaging. Across prediction horizons, several deep learning models consistently outperformed classical approaches, with the best model producing informative forecasts up to 1.5 seconds into the future. Our findings point toward future control applications and open new avenues for probing the intrinsic temporal structure of neural activity.