Dynamical modeling of nonlinear latent factors in multiscale neural activity with real-time inference

AI summary

Overview

Research area: Computational neuroscience, neural decoding, multimodal machine learning, and dynamical latent variable models for brain activity. Technical level: Advanced. Scope: This paper introduces MRINE, a nonlinear dynamical modeling framework that fuses multiscale neural modalities with different timescales, distributions, and missing samples to enable real-time recursive decoding.

What This Paper Is About

Brain activity is often measured simultaneously with multiple modalities, such as discrete spiking activity and continuous local field potentials (LFPs). These modalities have different sampling rates, different statistical distributions, and can be missing at some time steps, which makes it difficult to combine them for real-time decoding of behavior or other target variables. The paper aims to build a nonlinear latent factor model that aggregates information across such multiscale modalities in real time while handling missing samples and timescale differences without relying on suboptimal zero-padding.

Key Contributions

1. Introduces MRINE, a nonlinear dynamical modeling framework for multimodal neural time-series that supports real-time recursive inference and non-causal smoothing while handling different timescales, distinct distributions, and missing samples.
2. Designs a multiscale encoder that uses modality-specific neural networks and linear dynamical systems with Kalman filtering to learn within-modality dynamics, predict missing samples forward in time, and then nonlinearly fuse modality-specific embeddings.
3. Develops a multiscale linear dynamical system backbone with modality-specific decoders for Poisson-distributed spikes and Gaussian-distributed continuous signals, enabling causal filtering, smoothing, and multi-horizon prediction.
4. Trains the model with multi-horizon k-step-ahead prediction, smoothed reconstruction, smoothness regularization extended to Poisson observations, and time-dropout, and demonstrates improved decoding over linear and nonlinear multimodal benchmarks.

Main Findings

• Simulation validation: In stochastic Lorenz attractor simulations, fusing Poisson and Gaussian channels improved latent reconstruction accuracy, with larger gains when the primary modality had fewer channels.
• Nonhuman primate behavior decoding: In two nonhuman primate datasets, adding LFP channels to spike channels and spike channels to LFP channels both improved decoding of 2D reach velocities. Improvements were larger in low-information regimes.
• Specific decoding gains: For grid reaching, adding 20 LFP channels to 5, 10, and 20 spike channels improved accuracy by 14.7%, 10.4%, and 5.3%; adding 20 spike channels to 5, 10, and 20 LFP channels improved accuracy by 49.8%, 34%, and 29.6%. For center-out reaching, the corresponding gains were 47.3%, 24.4%, 9.8% and 30.1%, 21.6%, 20.6%.
• Benchmark superiority: MRINE significantly outperformed MSID, mmPLRNN, MMGPVAE, and MVAE across all tested channel regimes for both nonhuman primate datasets. Several baselines required zero-padding for missing LFP time steps or used non-causal inference.
• Timescale flexibility: MRINE handled spikes sampled every 10 ms and LFPs sampled every 50 ms, and also improved decoding when both modalities had the same 10 ms timescale.
• Additional dataset: On a high-dimensional visual stimuli dataset containing Neuropixels spikes and calcium imaging data, MRINE also outperformed comparison methods.
• Ablation contributions: Smoothness regularization, smoothed reconstruction, the multiscale encoder design, and time-dropout each contributed to improved performance.

Methodology in Plain English

The method separates the problem into three parts: within-modality dynamics, cross-modality fusion, and shared multiscale dynamics. First, spikes are modeled with a Poisson distribution and continuous signals such as LFPs with a Gaussian distribution. Each modality is passed through its own small neural network to create modality-specific embedding factors. Each modality also has its own linear dynamical system, and a Kalman filter estimates that modality’s latent state at each time step. If a modality is missing at a time step, its dynamical system predicts the state forward using only its own past dynamics. The filtered modality-specific embeddings are then concatenated and passed through a fusion network to form multiscale embedding factors. These factors become observations for a central linear dynamical system, where Kalman filtering produces real-time multiscale latent factors and Kalman smoothing produces non-causal estimates. Modality-specific decoders map the multiscale latent factors back to spike and LFP distributions. Behavior is decoded from the inferred latent factors. Training uses multi-horizon prediction, smoothed reconstruction, smoothness regularization across time for spikes, LFPs, and latent factors, and time-dropout to improve robustness to missing modalities.

Why This Matters

This work provides a practical way to combine complementary neural signals for real-time decoding while directly addressing missing samples and different sampling rates, which prior nonlinear multimodal models did not handle. It advances both neural latent variable modeling and multimodal machine learning for heterogeneous time-series. Real-world applications include:

• Brain-computer interfaces and neuroprosthetics for restoring movement or communication.
• Closed-loop neuromodulation and clinical monitoring using spikes and field potentials.
• Neuroscience research on how behavior is encoded across spatial and temporal scales.
• Multimodal time-series fusion in other domains with different sampling rates and missing data, such as wearable sensors, physiological monitoring, and industrial sensing.

Industry relevance spans neurotechnology, medical devices, real-time AI systems, and any application that needs to fuse multiple asynchronous or heterogeneous data streams causally.

Future Directions

• Testing MRINE in closed-loop human brain-computer interfaces and clinical neurotechnology settings.
• Scaling the framework to more modalities, larger channel counts, and longer recordings while reducing computational cost for implantable real-time hardware.
• Improving robustness to nonstationarity, electrode drift, and changing missing-data patterns across sessions.
• Extending the model to other likelihood distributions and jointly modeling behavior within the generative framework.
• Developing theoretical guarantees for identifiability, stability, and robustness of nonlinear multimodal latent factor models.

Target Audience

Computational neuroscientists, neural engineers, brain-computer interface researchers, and machine learning researchers working on state-space models, multimodal time-series, or real-time inference. The paper is also relevant to neurotechnology industry developers and advanced students interested in neural decoding and dynamical latent variable models.

Authors’ abstract

Real-time decoding of target variables from multiple simultaneously recorded neural time-series modalities, such as discrete spiking activity and continuous field potentials, is important across various neuroscience applications. However, a major challenge for doing so is that different neural modalities can have different timescales (i.e., sampling rates) and different probabilistic distributions, or can even be missing at some time-steps. Existing nonlinear models of multimodal neural activity do not address different timescales or missing samples across modalities. Further, some of these models do not allow for real-time decoding. Here, we develop a learning framework that can enable real-time recursive decoding while nonlinearly aggregating information across multiple modalities with different timescales and distributions and with missing samples. This framework consists of 1) a multiscale encoder that nonlinearly aggregates information after learning within-modality dynamics to handle different timescales and missing samples in real time, 2) a multiscale dynamical backbone that extracts multimodal temporal dynamics and enables real-time recursive decoding, and 3) modality-specific decoders to account for different probabilistic distributions across modalities. In both simulations and three distinct multiscale brain datasets, we show that our model can aggregate information across modalities with different timescales and distributions and missing samples to improve real-time target decoding. Further, our method outperforms various linear and nonlinear multimodal benchmarks in doing so.