ZEBRA: Towards Zero-Shot Cross-Subject Generalization for Universal Brain Visual Decoding

AI summary

Authors: Haonan Wang, Jingyu Lu, Hongrui Li, Xiaomeng Li (The Hong Kong University of Science and Technology) arXiv: 2510.27128v1 [cs.CV], 31 Oct 2025 | License: CC BY 4.0

Overview

Research area: Computational neuroscience and computer vision — specifically fMRI-to-image reconstruction (brain visual decoding), with a focus on cross-subject generalization.

Technical level: Intermediate. The paper combines an fMRI encoder, CLIP-aligned embeddings, diffusion priors, and adversarial training, which assumes familiarity with representation learning and generative models.

Scope: The paper proposes and evaluates ZEBRA, a framework that reconstructs viewed images from fMRI signals of subjects it has never been trained on, without any subject-specific fine-tuning.

What This Paper Is About

Most fMRI-to-image reconstruction systems are trained per person, or are pretrained on many people and then fine-tuned on each new subject. That fine-tuning step makes the technology impractical outside the lab, since each new individual requires expert involvement and roughly a day of computation. ZEBRA's goal is to skip that step entirely: train once on a set of subjects, then decode images directly from a brand-new subject's brain activity with no additional data or retraining.

Key Contributions

1. ZEBRA, described as the first zero-shot brain visual decoding framework. It generalizes to unseen subjects without additional fMRI data or subject-specific finetuning, requiring training only once on the training subjects and using only the invariant projection path at inference.
2. A disentanglement strategy for fMRI representations. The method decomposes brain features into subject-invariant and subject-specific components using residual decomposition plus adversarial training, and separately extracts semantic-specific features aligned to CLIP embeddings.
3. Two "Representation Preservation Anchors." An auxiliary fMRI reconstruction task (in the subject-invariant branch) and a three-way alignment to classification, CLIP vision, and CLIP text targets (in the semantic branch) prevent adversarial training from distorting the underlying brain feature space.
4. Empirical validation across training regimes. ZEBRA outperforms zero-shot baselines on every reported metric and reaches performance comparable to fully finetuned models on several metrics, supported by quantitative tables, qualitative reconstructions, feature visualizations, and ablations. Code and model weights are released at https://github.com/xmed-lab/ZEBRA.

Main Findings

• Large gains over the only zero-shot-compatible baseline. Averaged over subjects 1, 2, 5, and 7 of the Natural Scenes Dataset, ZEBRA raises PixCorr from 0.057 (NeuroPictor⋆) to 0.131 and SSIM from 0.297 to 0.375. The paper's abstract reports the PixCorr gain as +0.084 (0.153 vs. 0.069 of NeuroPictor), and an average improvement of +6.4 percentage points on Alex(5) (81.8% vs. 75.4%).
• Consistent high-level semantic improvements. Against NeuroPictor⋆, ZEBRA achieves 74.6% vs. 71.4% on AlexNet(2), 81.2% vs. 74.7% on AlexNet(5), +9.7 percentage points on Inception (72.2% vs. 62.5%), and +5.5 percentage points on CLIP (71.5% vs. 66.0%).
• Lower perceptual distance. EffNet-B distance drops from 0.939 to 0.837 and SwAV distance from 0.607 to 0.506 relative to NeuroPictor⋆.
• Comparable to fully finetuned models on some metrics. ZEBRA reports an SSIM of 0.384, close to the 0.375 of fully finetuned NeuroPictor, despite using no test-subject data. In the zero-shot setting it reaches 74.6% on AlexNet(2), against 78.9% for Takagi et al. and 87.7% for MindBridge under full finetuning.
• Ablations confirm every component matters. On subject 1, the base model reaches PixCorr 0.089, SSIM 0.325, and 63.2% CLIP. Adding adversarial training in SIFE yields +0.040 PixCorr and +3.6% CLIP; adding the anchor and the SSFE components yields progressively higher scores, with the full model at PixCorr 0.153, SSIM 0.384, Alex(2) 76.1%, Alex(5) 81.8%, Inception 73.4%, CLIP 72.3%, EffNet-B 0.814, and SwAV 0.490.
• More training subjects help. Increasing training subjects from 4 (2 to 5) to 7 (2 to 8) improves PixCorr from 0.109 to 0.153 and CLIP from 63.7% to 72.3%, with consistent trends across all metrics.
• Disentanglement is visible in feature space. UMAP and t-SNE visualizations show subject-invariant features mixed across all subjects with no subject-specific clustering, while subject-specific features cluster clearly by subject.
• Remaining weakness: semantics. ZEBRA's main limitation relative to few-shot methods is semantic accuracy; failure cases occur mainly on fine-grained semantic distinctions, especially rare object categories.
• Efficiency claim. The paper states decoding reaches 73.4% performance with approximately one second of inference per image, in contrast to subject-specific finetuning that often exceeds 12 hours per subject.

Methodology in Plain English

ZEBRA starts from a strong baseline: a ViT-based fMRI encoder (fMRI-PTE, pretrained on the UK Biobank dataset) turns each subject's fMRI scan into a unified 2D brain activation map of size 256×256 and then into a latent representation, which a diffusion prior converts into OpenCLIP vision embeddings used to guide image generation with Stable Diffusion. This baseline alone, however, remains sensitive to individual differences.

The core idea is to split the brain feature into two parts — what is common across people, and what is personal to each individual. A self-attention block produces the "invariant" part, and the "specific" part is simply what remains (the residual). To force the invariant part to actually be subject-agnostic, a subject discriminator tries to guess who the person is from it, while the extractor tries to make that impossible (adversarial training via a gradient reversal layer). Meanwhile a subject classifier is trained to correctly identify the individual from the residual, so that personal information is genuinely pushed into that branch.

Because this adversarial pushing can damage the original brain representation, the authors add a "preservation anchor": a small masked decoder reconstructs the input fMRI signal from the features using a mean absolute error loss, keeping the latent space faithful to the neural data.

A second module takes the invariant features and aligns them with semantic information. Brain features are projected into CLIP vision space through three linear layers with GELU activations, producing semantic-specific, semantic-invariant, and general embeddings. The semantic-specific embedding is aligned directly to ground-truth OpenCLIP vision embeddings with a BiMixCo contrastive loss, while a gradient reversal layer discourages the semantic-invariant branch from matching CLIP targets. Another preservation anchor aligns embeddings to image class labels (cross-entropy), CLIP vision, and CLIP text.

All losses are summed into one objective, with the diffusion prior loss weighted by λ = 30. Training used NSD (8 subjects, roughly 8,000–9,000 images each) for 60 epochs on 8 NVIDIA RTX H800 GPUs with a total batch size of 128 (16 per GPU), the AdamW optimizer, a learning rate of 1e-4, and the OneCycle schedule. For each test subject (1, 2, 5, or 7, each with 982 shared test images), the model is trained on the other 7 subjects. At inference, the predicted image latents are decoded into coarse images with SDXL unCLIP and then refined with base SDXL in image-to-image mode guided by predicted captions, starting from a noised coarse image and skipping the first 50% of diffusion steps. Evaluation follows MindEye2's metrics: PixCorr, SSIM, AlexNet(2), AlexNet(5), and high-level metrics from EffNet-B, SwAV, Inception, and CLIP.

Why This Matters

Impact on research. The paper reframes cross-subject fMRI decoding from a "pretrain-then-finetune" problem into a zero-shot generalization problem. By showing that competitively separating subject-invariant from semantic-specific information is possible, it suggests a shared latent space for brain representations across individuals — an idea with implications beyond image reconstruction, since the authors describe ZEBRA as modality-agnostic and extendable to text or video.

Real-world applications:

• Clinical and research use: Decoding tools could be deployed on new patients without an AI expert spending hours per subject on finetuning.
• Brain-computer interfaces: The paper cites real-time BCI and neurorehabilitation as fields held back by fine-tuning delays.
• Mental state interpretation: Translating BOLD responses into visual reconstructions supports studying what a person is perceiving or imagining.
• Scalable neuroscience studies: A subject-agnostic feature space enables broader exploration of cognitive function and individual variability across large cohorts.

Industry relevance. The efficiency argument — roughly one second of inference per image versus over 12 hours of subject-specific finetuning — is what turns brain decoding from a lab demonstration into a deployable pipeline. Companies and labs working on neurotechnology, assistive communication devices, and generative models conditioned on neural signals would benefit from a model that works on a new person out of the box. The authors also emphasize that ethical and regulatory frameworks for brain data privacy must accompany these technical advances, and report code and model weights as openly available on GitHub.

Future Directions

• Improving high-level semantic fidelity. The paper identifies this as the key remaining challenge: ZEBRA's reconstructions are competitive on low-level perceptual metrics but lag few-shot approaches on semantic accuracy, with failures concentrated on rare categories and fine-grained distinctions.
• Extending beyond static images. Since the framework is described as modality-agnostic, the authors propose combining ZEBRA with methods such as NeuroClips or Neurons to enable zero-shot fMRI-to-video generation, and possibly text.
• Scaling subject diversity. The current dataset has a limited number of subjects; the ablation showing consistent gains from 4 to 7 training subjects suggests that more subjects and more fMRI data should further improve robustness and zero-shot performance toward a universal brain decoder.
• Broader real-world coverage. Capturing everyday visual experience more comprehensively will require additional fMRI recordings and wider subject coverage, which the authors frame as a problem spanning machine learning, computer vision, neuroscience, and biomedical engineering.

Target Audience

This paper is most useful for researchers and engineers working on neural decoding, fMRI-to-image reconstruction, and brain-computer interfaces, especially those interested in cross-subject transfer and zero-shot generalization. It also suits practitioners in generative modeling and representation learning who want to see disentanglement and adversarial training applied to biological data, and computational neuroscientists studying how semantic information is shared across individuals. Clinicians and neurotechnology product teams evaluating whether decoding models can be deployed without per-patient fine-tuning will find the efficiency claims and the limitations section directly relevant.

Authors’ abstract

Recent advances in neural decoding have enabled the reconstruction of visual experiences from brain activity, positioning fMRI-to-image reconstruction as a promising bridge between neuroscience and computer vision. However, current methods predominantly rely on subject-specific models or require subject-specific fine-tuning, limiting their scalability and real-world applicability. In this work, we introduce ZEBRA, the first zero-shot brain visual decoding framework that eliminates the need for subject-specific adaptation. ZEBRA is built on the key insight that fMRI representations can be decomposed into subject-related and semantic-related components. By leveraging adversarial training, our method explicitly disentangles these components to isolate subject-invariant, semantic-specific representations. This disentanglement allows ZEBRA to generalize to unseen subjects without any additional fMRI data or retraining. Extensive experiments show that ZEBRA significantly outperforms zero-shot baselines and achieves performance comparable to fully finetuned models on several metrics. Our work represents a scalable and practical step toward universal neural decoding. Code and model weights are available at: https://github.com/xmed-lab/ZEBRA.