Cross-Modal Representational Knowledge Distillation for Enhanced Spike-Informed LFP Modeling

AI summary

Overview

Research area: Neural signal processing and representation learning, at the intersection of deep learning (transformers, knowledge distillation) and computational neuroscience / brain-computer interfaces.

Technical level: Advanced. The paper assumes familiarity with transformer architectures, masked autoencoding, and knowledge distillation objectives, plus basic understanding of intracortical neural recordings (spikes and local field potentials).

Scope: The paper introduces a cross-modal knowledge distillation framework that transfers representational knowledge from multi-session spike models to LFP-only models, improving downstream behavior decoding across six monkeys and three datasets.

What This Paper Is About

Local field potentials (LFPs) are routinely recorded alongside spiking activity, and they offer practical advantages over spikes for brain-computer interfaces — greater long-term stability, robustness to electrode degradation, and lower power needs. However, LFP-based models consistently underperform spike-based models at decoding behavior because LFPs are aggregate, population-level signals with redundant spatial patterns and high noise correlations. This paper's goal is to close that performance gap by transferring knowledge from a well-trained spike model into an LFP model, so that the resulting LFP model works using only LFP signals at inference time.

Key Contributions

1. A novel unsupervised cross-modal knowledge distillation framework that aligns the latent representations of an LFP student model to those of a pretrained multi-session spike teacher model, evaluated across motor cortical data from 6 monkeys spanning 3 datasets (both raw LFP and LFP power signals).
2. A multi-session extension of the distillation procedure that pools paired spike-LFP data across many sessions, improving performance further when behavior labels are scarce.
3. A multi-session LFP-only baseline (MS-LFP) built with session-specific spatial patch tokenization, enabling rigorous comparison of distilled versus non-distilled LFP models.
4. An improved multi-session spike model (MS-Spike) with a new neural tokenization scheme (session-specific space embeddings) that outperforms the recent state-of-the-art NDT2 baseline on downstream behavior decoding.

Main Findings

• Large, consistent decoding gains: In the fully unsupervised setting, Distilled LFP models achieved an average behavior decoding R² of 0.71, versus 0.27 for MS-LFP and 0.24 for single-session LFP models (n=51 sessions, p < 2.6×10⁻¹⁰, one-sided Wilcoxon signed-rank test).
• Distillation can match or beat the teacher: Distilled LFP models slightly outperformed their own spike-based teacher models (0.71 vs. 0.69 average R², p < 1.7×10⁻³), suggesting alignment helps extract shared, behavior-predictive information more effectively.
• Better than multimodal concatenation: Multimodal models trained on concatenated spike and LFP inputs (and even when spikes were zeroed at inference) reached only 0.27–0.33 average R², well below the distilled LFP models.
• Representations become aligned: t-SNE visualizations clustered MS-Spike and Distilled LFP representations together, far from MS-LFP representations. Quantitative alignment metrics confirmed this: top-1 retrieval accuracy of 0.67, top-5 of 0.90, mean rank 4.99, and Centered Kernel Alignment (CKA) of 0.89 between spike and distilled LFP representations, versus near-random retrieval for MS-LFP pairs.
• Generalization to unseen sessions: A Distilled LFP model trained on a single session's spike–LFP alignment outperformed all LFP baselines on other sessions never used in pretraining, fine-tuning, or distillation.
• Supervised settings also benefit: Even when the spike teacher was fine-tuned with behavior supervision, Distilled LFP models still beat all LFP-only baselines, though spike models naturally remained the upper bound when directly supervised.
• Multi-session distillation adds further gains: Pooling paired data across sessions improved supervised fine-tuning (0.82 vs. 0.80) and fully-supervised fine-tuning (0.83 vs. 0.81) over single-session distillation.
• Scalable to smaller models: Distilled LFP models with 10× fewer parameters than baseline LFP models still outperformed all baselines, indicating the gains come from representation quality rather than model capacity.

Methodology in Plain English

The approach is a teacher–student setup across two neural modalities.

Step 1 — Build a strong spike teacher. The authors train a multi-session transformer on 226 sessions from 16 subjects across 6 public datasets. Neural activity is tokenized by grouping neurons into spatial patches; each patch gets a "value embedding" (shared across sessions) plus a session-specific "space embedding" that captures electrode/neuron arrangement. This differs from prior work (NDT2), which used session tokens instead of session-specific spatial embeddings. The model is pretrained with masked autoencoding: 60% of tokens (across both time and space) are hidden and must be reconstructed.

Step 2 — Fine-tune the teacher on the target session using either unsupervised MAE, joint MAE + behavior regression, or fully supervised behavior regression.

Step 3 — Train an LFP student. A fresh LFP transformer is initialized with the same patch tokenization (using a dilated causal convolutional layer instead of learnable embeddings, since LFP values are continuous). The student is trained with a combined objective: (a) an autoencoding loss that reconstructs the observed LFP signal, and (b) a representation alignment loss that maximizes cosine similarity between the student's latent representations and the frozen teacher's spike representations at each timestep. A scaling hyperparameter λ balances the two terms. The teacher stays frozen throughout.

The final model operates on LFP signals alone at inference, which is the key practical advantage over multimodal fusion approaches.

Why This Matters

Impact on research: This work shows that cross-modal distillation can transfer knowledge between fundamentally different neural measurement scales — spiking activity (individual neurons) and LFPs (population-level fields). Beyond engineering better decoders, it provides a new scientific tool for aligning and comparing latent structure across spatiotemporal scales of brain activity, which the authors demonstrate quantitatively via t-SNE, representation retrieval, and CKA analyses. It also provides a rigorous multi-session LFP baseline that the field previously lacked.

Real-world applications:

• Long-term brain-computer interfaces: Since LFP signals remain stable as spike recordings degrade from electrode encapsulation, distilled LFP decoders could maintain BCI performance over years rather than months.
• Motor neuroprosthetics: More accurate decoding of movement variables (e.g., velocity) from LFP could improve control of robotic limbs or cursor interfaces for paralyzed patients.
• Low-power implantable devices: LFPs require less power than spike processing, making them more suitable for wireless, battery-constrained implants.
• Clinical neural monitoring: Better LFP representations could assist in seizure detection, Parkinson's disease monitoring, and other conditions where field potentials are the primary recorded signal.

Industry relevance: Directly relevant to neurotechnology companies developing chronic implants (e.g., Blackrock Neurotech, Neuralink, Synchron, Paradromics) that must balance signal longevity against decoding accuracy, and to medical device makers building closed-loop neuromodulation systems. The 10× smaller model result also matters for on-device inference with strict compute limits.

Future Directions

1. Scale up pretraining data: Extend to larger and more diverse multi-session, multi-subject datasets to learn richer latent representations and improve robustness.
2. Explore alternative pretraining objectives: Move beyond masked autoencoding to other self-supervised tasks that may yield higher-quality teacher representations.
3. Extend beyond modality transfer: Test the same distillation framework for transfer between brain regions, or between different recording technologies (e.g., iEEG or EEG).
4. Validate under degraded signal conditions: Confirm that distilled LFP models are genuinely reliable replacements for spike-based decoders under controlled chronic degradation, which the authors acknowledge is needed before full clinical deployment.
5. Build more sophisticated multimodal architectures: The current multimodal baseline was a simple proof of concept; richer fusion models could complement the distillation approach when both modalities are available.

Target Audience

This paper is most valuable to neural engineers and machine learning researchers working on brain-computer interfaces and neural decoding models, especially those interested in self-supervised representation learning for neural signals. Computational neuroscientists studying population-level dynamics will benefit from the representation-alignment analyses. Practitioners building deployable neurotechnology (implant developers, clinical BCI groups) will find the practical framing — LFP-only inference, long-term stability, low power, small models — directly actionable. Graduate students entering the fields of neural data modeling or cross-modal representation learning will find the methodological template useful for their own work.

Authors’ abstract

Local field potentials (LFPs) can be routinely recorded alongside spiking activity in intracortical neural experiments, measure a larger complementary spatiotemporal scale of brain activity for scientific inquiry, and can offer practical advantages over spikes, including greater long-term stability, robustness to electrode degradation, and lower power requirements. Despite these advantages, recent neural modeling frameworks have largely focused on spiking activity since LFP signals pose inherent modeling challenges due to their aggregate, population-level nature, often leading to lower predictive power for downstream task variables such as motor behavior. To address this challenge, we introduce a cross-modal knowledge distillation framework that transfers high-fidelity representational knowledge from pretrained multi-session spike transformer models to LFP transformer models. Specifically, we first train a teacher spike model across multiple recording sessions using a masked autoencoding objective with a session-specific neural tokenization strategy. We then align the latent representations of the student LFP model to those of the teacher spike model. Our results show that the Distilled LFP models consistently outperform single- and multi-session LFP baselines in both fully unsupervised and supervised settings, and can generalize to other sessions without additional distillation while maintaining superior performance. These findings demonstrate that cross-modal knowledge distillation is a powerful and scalable approach for leveraging high-performing spike models to develop more accurate LFP models.