NeuroBridge: Bio-Inspired Self-Supervised EEG-to-Image Decoding via Cognitive Priors and Bidirectional Semantic Alignment

AI summary

Overview

Research area: Computer vision and neural decoding — specifically self-supervised cross-modal learning that maps electroencephalography (EEG) signals to visual images.

Technical level: Intermediate. The paper assumes familiarity with contrastive learning (CLIP-style dual encoders, InfoNCE-style losses), but its core ideas — augmenting two modalities differently and projecting both into one shared space — are explained in accessible terms.

Scope: One-sentence scope: the paper proposes NeuroBridge, a self-supervised architecture combining Cognitive Prior Augmentation (CPA) and a Shared Semantic Projector (SSP) to improve zero-shot EEG-to-image retrieval on the THINGS-EEG benchmark, with additional evaluation on THINGS-MEG.

What This Paper Is About

The paper tackles the problem of decoding what image a person is looking at from their EEG brain activity alone. The core obstacle is a "modality gap": EEG signals are temporal, low-dimensional and noisy, while images are spatial, high-dimensional and semantically dense, and EEG responses to the same image vary across subjects, sessions and mental states. Because EEG datasets are small relative to computer vision datasets, the authors aim to build a self-supervised framework that closes this gap using cognitive priors and bidirectional alignment rather than collecting more data.

Key Contributions

1. NeuroBridge, a unified self-supervised framework for EEG-to-image decoding that aims to bridge the modality gap and generalizes across intra-subject and inter-subject settings.
2. Cognitive Prior Augmentation (CPA), an asymmetric augmentation strategy that applies multiple transformations to images but only a single transformation to EEG signals, simulating perceptual variability and enriching semantic diversity across both modalities.
3. Shared Semantic Projector (SSP), a bidirectional alignment module that projects image and EEG features into one trainable shared semantic space, replacing unidirectional alignment with a co-adaptive strategy.
4. Empirical validation on THINGS-EEG and THINGS-MEG, including ablations on transformations, feature fusion, projector design, encoder backbones, and regularisation strategy, plus an open-source release (https://github.com/feroooooo/NeuroBridge).

Main Findings

• Intra-subject gains: On the 200-way zero-shot retrieval task on THINGS-EEG, NeuroBridge reaches 63.2% Top-1 and 89.9% Top-5 averaged over 10 subjects, improvements of +12.3% and +10.2% over the previous state of the art. The strongest prior baseline reported, UBP, averaged 50.9% Top-1 and 79.7% Top-5.
• Inter-subject gains: Under leave-one-subject-out evaluation on THINGS-EEG, NeuroBridge averages 19.0% Top-1 and 45.9% Top-5, versus 14.0% Top-1 and 34.3% Top-5 for Neural-MCRL, the strongest prior inter-subject baseline listed.
• Augmentation asymmetry matters: For images, gaussian blur, gaussian noise, low resolution and mosaic improve retrieval, while color jitter, grayscale and random cropping reduce it. For EEG, only smoothing consistently helps; temporal shifting is reported to hurt.
• More fused views help up to a point: Top-1 rises from 50.9% with one fused image transformation to 62.1% with four, then declines to 58.5% (five), 58.1% (six) and 57.0% (seven).
• Asymmetric ℓ2 normalisation is critical: Normalising only the image features gives 63.2%/89.9%; normalising both ("Sym") drops to 46.4%/79.0%; normalising only EEG ("Inv-Asym") drops to 38.6%/71.7%; normalising neither ("Plain") gives 54.4%/85.5%.
• Projector design: A 512-dimensional linear projection performs best among the linear and MLP projector variants tested; higher-dimensional projectors are reported to overfit or become redundant, lower-dimensional ones too constrained.
• Ablation of components: With neither CPA nor SSP, accuracy is 40.5%/72.2%. Image prior alone raises it to 60.0%/89.1%; the EEG prior alone gives 40.8%/72.7%; SSP alone gives 41.5%/73.5%. The full combination reaches 63.2%/89.9%. The image prior is the single largest contributor.
• Robustness to hyperparameters: Best batch size reported is 1,024 (63.2%/89.9%); performance ranges from 54.6% Top-1 at batch size 32 to 62.2% at batch size 2,048. For temperature, 0.5 gives the highest numbers (63.6%/91.0%) and 0.001 the lowest (59.5%/88.9%); the default is 0.07.
• Transfer to MEG: On THINGS-MEG, NeuroBridge reaches 32.2% Top-1 / 60.8% Top-5 intra-subject (versus UBP at 26.7%/55.2% and NICE at 12.8%/36.0%) and 3.4% Top-1 / 12.8% Top-5 inter-subject (versus UBP at 2.2%/10.4%).
• Semantic consistency on Subject 8: The similarity matrix over all 200 test concepts shows strong alignment, and retrieved neighbours tend to share semantic category (animals retrieved for animal queries, food for food queries).

Methodology in Plain English

The setup is a paired dataset of images and simultaneously recorded EEG. A frozen, pretrained CLIP image encoder produces visual features, while an EEG encoder is trained from scratch. Two ideas do the heavy lifting:

First, Cognitive Prior Augmentation. Because people attend to different parts of the same image and their brain responses fluctuate, the authors augment the two modalities differently — many image transformations (gaussian blur, gaussian noise, low resolution, mosaic, and others) but only one EEG transformation (smoothing by default). The features from the multiple augmented image views are averaged into a single representative embedding, which keeps the computational cost and alignment problem manageable.

Second, the Shared Semantic Projector. Rather than aligning EEG directly to CLIP's language-influenced embedding space, both image and EEG features are passed through trainable projectors into a new, shared semantic space learned from data. Alignment is enforced with a contrastive loss that pulls matched EEG-image pairs together and pushes mismatched pairs apart. A distinctive detail: only the image features are ℓ2-normalised on the unit hypersphere, while EEG feature magnitudes are allowed to vary, so magnitude acts as a learnable confidence signal and the pretrained image structure is preserved.

Training uses PyTorch on two NVIDIA GeForce RTX 3090 GPUs, batch size 1,024, 50 epochs, AdamW with learning rate 1e-4 and weight decay 1e-4, temperature τ = 0.07, and results averaged over five independent runs. For the intra-subject THINGS-EEG setting, EEG input is restricted to 17 parietal/occipital electrodes (P7, P5, P3, P1, Pz, P2, P4, P6, P8, PO7, PO3, POz, PO4, PO8, O1, Oz, O2) with EEGProject as encoder; the inter-subject setting uses all 63 channels with TSConv. Preprocessing follows prior work: 0–1,000 ms epochs from stimulus onset, baseline correction from the 200 ms pre-stimulus window, downsampling to 250 Hz, and multivariate noise normalisation (MVNN), yielding 16,540 training samples and 200 test samples per subject.

Why This Matters

The work is a step toward decoding visual experience from cheap, portable, non-invasive brain recordings. Its main research impact is showing that architectural priors — asymmetric augmentation and a learned shared semantic space — can substitute substantially for the large paired datasets that EEG-image decoding lacks, and that the same framework transfers from EEG to MEG without redesign.

Potential applications:

• Brain-computer interfaces: non-invasive communication or control channels for users who cannot speak or type, driven by imagined or perceived visual content.
• Clinical assessment: objective measurement of visual processing pathways, potentially useful where patients cannot verbally report what they see.
• Cognitive neuroscience research: a quantitative tool for comparing how different people represent the same visual concepts, and for studying attention and perceptual variability.
• Brain-inspired AI and multimodal retrieval: the CPA/SSP ideas may transfer to other weakly aligned modality pairs (for example audio-image, video-text) that also lack large paired corpora.

Industry relevance: the method is deliberately lightweight in practice — a frozen CLIP backbone with trainable projectors and an EEG encoder of roughly 2.4M to 17.2M parameters depending on configuration — which is relevant for consumer-grade BCI hardware and for groups that lack large-scale compute.

Future Directions

• Adaptive augmentation: the authors note that manually designed augmentations may insufficiently capture cognitive variability, and suggest adaptive augmentation as a next step.
• Removing vision-language bias: reliance on pretrained visual encoders can introduce biases inherited from image-text pretraining; data-driven alignment is proposed as a mitigation.
• Scaling data: the authors suggest that larger EEG-image datasets could alleviate the trade-off observed in projector dimensionality, where 512 dimensions was optimal and higher dimensions overfit.
• Closing the inter-subject gap: inter-subject accuracy (19.0% Top-1 on THINGS-EEG, 3.4% on THINGS-MEG) remains far below intra-subject performance, leaving cross-subject generalisation a largely open problem.

Target Audience

Researchers and graduate students in neural decoding, brain-computer interfaces, and multimodal representation learning; computer vision practitioners interested in low-data cross-modal alignment; and neuroscientists who want a quantitative, self-supervised tool for comparing EEG and MEG responses to visual stimuli across subjects. Readers need some comfort with contrastive learning to follow the loss formulation, but the high-level strategy is accessible to a broader interdisciplinary audience.

Authors’ abstract

Visual neural decoding seeks to reconstruct or infer perceived visual stimuli from brain activity patterns, providing critical insights into human cognition and enabling transformative applications in brain-computer interfaces and artificial intelligence. Current approaches, however, remain constrained by the scarcity of high-quality stimulus-brain response pairs and the inherent semantic mismatch between neural representations and visual content. Inspired by perceptual variability and co-adaptive strategy of the biological systems, we propose a novel self-supervised architecture, named NeuroBridge, which integrates Cognitive Prior Augmentation (CPA) with Shared Semantic Projector (SSP) to promote effective cross-modality alignment. Specifically, CPA simulates perceptual variability by applying asymmetric, modality-specific transformations to both EEG signals and images, enhancing semantic diversity. Unlike previous approaches, SSP establishes a bidirectional alignment process through a co-adaptive strategy, which mutually aligns features from two modalities into a shared semantic space for effective cross-modal learning. NeuroBridge surpasses previous state-of-the-art methods under both intra-subject and inter-subject settings. In the intra-subject scenario, it achieves the improvements of 12.3% in top-1 accuracy and 10.2% in top-5 accuracy, reaching 63.2% and 89.9% respectively on a 200-way zero-shot retrieval task. Extensive experiments demonstrate the effectiveness, robustness, and scalability of the proposed framework for neural visual decoding.