MEIcoder: Decoding Visual Stimuli from Neural Activity by Leveraging Most Exciting Inputs

AI summary

Overview

Research area: Machine learning for neuroscience, specifically decoding (reconstructing) visual images from single-cell neural population activity in the primary visual cortex (V1), with connections to brain-machine interfaces and neuroprosthetics.

Technical level: Intermediate — readers should be comfortable with deep learning basics (CNNs, GANs, loss functions, transfer learning) and with basic visual neuroscience concepts such as receptive fields and V1.

Scope: The paper introduces MEIcoder, a decoder that combines neuron-specific most exciting inputs (MEIs), a log-SSIM reconstruction loss, and adversarial training to reconstruct visual stimuli from neural activity, and evaluates it on three datasets plus a proposed benchmark of over 160,000 samples.

What This Paper Is About

Decoding images from brain activity is hard because the data are scarce and the recorded neurons are few, so the neural signal is a highly compressed, noisy version of the original stimulus. Previous methods either train from scratch on limited single-subject data and produce low-fidelity reconstructions, or they lean on pre-trained generative models and produce sharp but unreliable images that contain hallucinated content or miss fine spatial detail. The goal of this paper is a decoding method that produces high-fidelity, spatially accurate reconstructions using only small numbers of recorded neurons and limited training data.

Key Contributions

1. A new decoding method (MEIcoder) that achieves state-of-the-art performance in reconstructing visual stimuli from V1 neural population activity, which the authors present as evidence that high-fidelity decoding is feasible with currently available single-subject data.
2. A scaling analysis of how reconstruction quality depends on the number of recorded neurons and the amount of training data.
3. A unified decoding benchmark created by aggregating datasets from multiple sources, containing over 160,000 samples.
4. A biologically informed architecture in which each readin module injects MEIs as prior knowledge and combines them nonlinearly in a shared core, enabling transfer across heterogeneous multi-subject datasets.

Main Findings

• MEIcoder outperforms all five baselines across all three datasets. On the Brainreader test set it reaches SSIM .400, PixCorr .679, Alex(2) .998, Alex(5) .990; with multi-subject pre-training and fine-tuning it reaches SSIM .424, PixCorr .706, Alex(2) .999, Alex(5) .977. On SENSORIUM 2022 it reaches SSIM .331, PixCorr .503, Alex(2) .988, Alex(5) .896. On Synthetic Cat V1 it reaches SSIM .774, PixCorr .777, Alex(2) .994, Alex(5) .987. For comparison, the strongest baseline numbers reported are InvEnc on Synthetic Cat V1 (SSIM .771, PixCorr .833, Alex(2) .986, Alex(5) .978) and InvEnc on Brainreader (SSIM .321, PixCorr .611, Alex(2) .989, Alex(5) .896).

• Gains are largest in data- and neuron-scarce settings. The authors attribute this to the Brainreader and SENSORIUM 2022 datasets being the most constrained among the three.

• MEIs are the main driver of performance. In the ablation study averaged over the three datasets, removing MEIs from the readin caused the largest performance drop across all metrics, compared with ablating the SSIM loss (replacing it with MSE) or removing the learnable neuron embeddings.

• MindEye2 reconstructions are less consistent and occasionally hallucinate. The variance of Alex(2) and Alex(5) scores across individual test points was much higher for MindEye2 than for MEIcoder, and the paper shows examples of hallucinated content (a woman's face, a giraffe).

• Data efficiency: MEIcoder's reconstructions begin to capture ground-truth images after training with 1,000 or fewer training data points, and at 1,000 training points it outperforms the second-best method from the main table trained on all data. Quantitative metrics plateau while qualitative results keep improving.

• Neuron efficiency: MEIcoder reaches more than 95% two-way identification ability (Alex(2)) on all three datasets with only around 1,000 neurons, and can accurately distinguish digits with around 2,500 neurons. Pixel correlation keeps increasing for Brainreader and Synthetic Cat V1 and does not saturate even with the 46,875 neurons of the synthetic cat dataset.

• Neurons, not training data, appear to be the more limiting factor. The authors conclude that recording 1,000 to 2,500 neurons from mouse V1 is enough for fine-grained reconstructions with the power to discriminate handwritten digits.

• Generalization to out-of-distribution stimuli: Using responses predicted by a CNN encoder pre-trained on Brainreader, MEIcoder reconstructed artificial patterns it had never seen in training, and simple linear combinations of MEIs weighted by neural responses already captured basic characteristics of the originals.

• Concept-based interpretability analysis: Using non-negative matrix factorization to learn 32 feature bases ("visual concepts") from core feature maps plus a sensitivity analysis, the authors observed that (1) high-intensity regions of active concepts become smaller and more focused across decoder layers, suggesting a coarse-to-fine hierarchy anchored to the neuron's MEI location; (2) many neurons' top three feature bases include a concept encoding image-border brightness, suggesting some neurons encode global lighting while specializing in local structure; and (3) for a few neurons, increasing the response incrementally shifts a dark object in the reconstruction, mirroring prior findings of over-representation of "black-dominant" (OFF) neurons in macaque V1 layers 2/3.

• Transfer learning helps only on one dataset. Multi-subject pre-training followed by single-subject fine-tuning improved results on Brainreader but not on SENSORIUM 2022, which the authors list as a limitation.

Methodology in Plain English

MEIcoder is split into two parts. A shared core — a six-layer convolutional neural network with batch normalization, ReLU activations, and dropout — maps a latent representation into an image. Separately, one readin module is trained per single-subject dataset and converts that subject's neural activity into the core's latent space, so learning signals can be reused across subjects even when the number of neurons and the response-to-stimulus mapping differ.

The distinctive part is the readin. Instead of a plain feed-forward network, the authors inject MEIs — the images that most strongly excite each individual neuron, which carry information about that neuron's receptive field and are computed once from the training data. Each recorded neuron's response is embedded together with a learnable neuron embedding through a one-layer network into a context representation of image size. That context representation is multiplied pointwise with the neuron's precomputed MEI to give a "neural map." The neural maps of all neurons are then combined by a pointwise convolution into a fixed number of channels, so that readins with different neuron counts all feed the same core. The biological intuition is a linear one — overlay the MEIs of all neurons weighted by their responses — but because V1 coding is nonlinear, the actual combination is left to the nonlinear core.

Training is end-to-end from random initialization for 300 epochs with the AdamW optimizer, using the validation set for hyperparameter search and early stopping based on the Alex(5) score. The objective combines a negative log-SSIM loss (weight 0.9) with an adversarial loss (weight 0.1) in which a secondary CNN discriminator tries to distinguish reconstructions from real dataset images. Unlike a standard GAN, the decoder is not trained generatively and is conditioned only on neural responses.

Evaluation uses SSIM, Pearson pixel correlation, and two-way identification with ImageNet-pretrained AlexNet features at layers 2 and 5. Baselines are InvEnc, Energy Guided Diffusion, MonkeySee, CAE, and MindEye2, all run using the original authors' code with the same hyperparameter search procedure. The three datasets are Brainreader (mouse V1, 8,587 neurons, 4,500/500/100 train/validation/test split, 36×64 px grayscale images), SENSORIUM 2022 (mouse, 8,372 neurons, 4,984 images, test set of 100 images with 10-trial averaged responses), and a synthetic cat V1 dataset generated with a validated spiking model (50,250 ImageNet images encoded into responses of 46,875 neurons; 45,000/5,000/250 split). Experiments used one NVIDIA Tesla V100 GPU with less than 32 GB of VRAM; the longest MEIcoder run took about four days and the longest baseline run about six days, with main results averaged over three random seeds.

Why This Matters

Impact on research. The paper argues that high-fidelity visual decoding from V1 is feasible with currently available single-subject data, challenging the assumption that strong generative priors from GenAI are necessary. It also shows for the first time that MEIs — previously used to study the tuning properties of individual neurons — are a powerful prior for stimulus reconstruction, and it documents the hallucination and high-variance problems of GenAI-based decoders in this setting. The released benchmark of over 160,000 samples, aggregated from previously published sources, is intended to make future comparisons easier.

Real-world applications.

• Brain-machine interfaces and visual neuroprosthetics, particularly for individuals with acquired blindness, which the authors name as their primary motivation.
• Evaluating and calibrating the information content of neural recordings, helping experimenters decide how many neurons must be recorded for a given task.
• Using biologically realistic spiking models (such as the cat V1 model) to generate large synthetic datasets for developing and benchmarking decoders when biological data are scarce.
• Interpretability-driven neuroscience, using the decoder as a tool to probe how individual neurons combine into a coherent visual representation.

Industry relevance. The work is relevant to neurotechnology companies building visual prostheses or neural interface decoding pipelines, and more broadly to machine learning groups interested in parameter-efficient architectures, loss design (log-SSIM versus MSE versus perceptual losses), and adversarial objectives for low-data regimes.

Future Directions

• Extending to higher-order visual areas. The authors focused on V1 and did not test on areas such as V4, where receptive fields are more complex; they argue the nonlinear core and learnable neuron embeddings should let MEIcoder handle them.
• Improving multi-subject transfer learning. Performance gains from pre-training appeared on Brainreader but not on SENSORIUM 2022, leaving the conditions under which transfer helps unresolved.
• Developing better evaluation metrics. The authors note that quantitative measures plateau while visual quality keeps improving, indicating limitations of pixel correlation and two-way identification scores.
• Deeper interpretability analysis. The concept-based analysis with NMF and sensitivity analysis is presented as an initial step; the authors call further interpretability work an exciting avenue, including comparison against existing work on visual feature coding.

Target Audience

This paper is most useful to researchers in neural decoding, computational neuroscience, and brain-machine interfacing, particularly those working with V1 recordings in mice, monkeys, or humans. It also suits machine learning researchers interested in biologically informed priors, low-data decoding, and generative versus discriminative reconstruction objectives, as well as neurotechnology practitioners who need to judge how many neurons and how much training data a practical decoder requires.

Authors’ abstract

Decoding visual stimuli from neural population activity is crucial for understanding the brain and for applications in brain-machine interfaces. However, such biological data is often scarce, particularly in primates or humans, where high-throughput recording techniques, such as two-photon imaging, remain challenging or impossible to apply. This, in turn, poses a challenge for deep learning decoding techniques. To overcome this, we introduce MEIcoder, a biologically informed decoding method that leverages neuron-specific most exciting inputs (MEIs), a structural similarity index measure loss, and adversarial training. MEIcoder achieves state-of-the-art performance in reconstructing visual stimuli from single-cell activity in primary visual cortex (V1), especially excelling on small datasets with fewer recorded neurons. Using ablation studies, we demonstrate that MEIs are the main drivers of the performance, and in scaling experiments, we show that MEIcoder can reconstruct high-fidelity natural-looking images from as few as 1,000-2,500 neurons and less than 1,000 training data points. We also propose a unified benchmark with over 160,000 samples to foster future research. Our results demonstrate the feasibility of reliable decoding in early visual system and provide practical insights for neuroscience and neuroengineering applications.