Brain-IT: Image Reconstruction from fMRI via Brain-Interaction Transformer

AI summary

Overview

• Research area: fMRI-to-image reconstruction (brain decoding), sitting at the intersection of computer vision, generative modeling, and computational neuroscience.
• Technical level: Intermediate to Advanced. The high-level ideas are accessible, but the pipeline assumes familiarity with diffusion models, CLIP/VGG features, and Transformer attention.
• Scope: The paper introduces Brain-IT, a brain-inspired pipeline built around a Brain Interaction Transformer (BIT) that maps clusters of functionally similar brain voxels to localized image features for diffusion-based image reconstruction.

What This Paper Is About

The core problem is reconstructing the image a person actually saw from their fMRI brain activity. Prior state-of-the-art methods, largely built on diffusion models, produce visually plausible images that are often unfaithful to the seen image — deviating in structure such as position and color, and sometimes missing or distorting semantic content. Brain-IT aims to close this gap by extracting brain representations in a way that better preserves the brain's distributed, retinotopic organization and by feeding two complementary feature types into the reconstruction process.

Key Contributions

1. Brain-IT pipeline: A brain-inspired approach for fMRI-to-image reconstruction that the authors report achieves state-of-the-art results both visually and on standard quantitative metrics.
2. Brain Interaction Transformer (BIT): A model that maps functional brain-voxel clusters directly to localized image features, enabling effective integration of information across multiple brains.
3. DIP-based low-level reconstruction: A new approach to low-level image reconstruction from fMRI using Deep Image Prior (DIP), which predicts the coarse image layout and serves as initialization for the diffusion process.
4. Efficient transfer learning: Meaningful reconstructions from just 15 minutes of fMRI recordings, and results at 1 hour that are comparable to prior methods trained on the full 40 hours.

Main Findings

• Quantitative dominance on 40 hours: Brain-IT outperforms all baselines in 7 of 8 reported metrics. On the NSD subjects 1, 2, 5, and 7, it reaches PixCorr 0.386, SSIM 0.486, Alex(2) 98.4%, Alex(5) 99.5%, Incep 97.3%, CLIP 96.4%, Eff 0.564, and SwAV 0.320, compared with MindEye2 at 0.322, 0.431, 96.1%, 98.6%, 95.4%, 93.0%, 0.619, and 0.344.
• One metric where it does not lead: NeuroVLA scores higher on CLIP (97.5% versus Brain-IT's 96.4%), making Brain-IT first in three high-level metrics and second in that one.
• Strong low-level fidelity: The paper attributes its large margin on all four low-level metrics (PixCorr, SSIM, Eff, SwAV) to the combination of BIT's VGG feature predictions and their Deep Image Prior inversion pipeline.
• Transfer learning with 1 hour: With only 1 hour of subject-specific data, Brain-IT reaches 0.331 PixCorr, 0.473 SSIM, 97.1% Alex(2), 98.6% Alex(5), 94.4% Incep, 93.0% CLIP, 0.648 Eff, and 0.370 SwAV — versus MindEye2 (1 hour) at 0.195, 0.419, 84.2%, 90.6%, 81.2%, 79.2%, 0.810, 0.468, and MindTuner (1 hour) at 0.224, 0.420, 87.8%, 93.6%, 84.8%, 83.5%, 0.780, 0.440.
• Gap widens under data scarcity: The performance gap over competing methods is larger at 1 hour of data than at 40 hours. The 1-hour Brain-IT results are reported as comparable to prior methods trained on the full 40-hour dataset.
• Extreme data efficiency: The authors state this is the first demonstration of high-quality reconstructions from as little as 15 minutes of fMRI data, with 30-minute results also reported.
• Branches are complementary (Table 2): The low-level branch alone scores SSIM 0.505, Alex(5) 99.1%, Incep 95.9%, CLIP 85.8%; the semantic branch alone scores 0.431, 99.4%, 96.1%, 95.2%; combined they reach 0.486, 99.5%, 97.3%, 96.4%. The low-level branch favors structural metrics, the semantic branch favors perceptual metrics, and the combination beats both on several metrics.
• External images help (Table T1): Adding roughly 120k unlabeled COCO images with predicted fMRI responses improves most metrics — for example PixCorr rises from 0.365 to 0.386, Alex(2) from 97.5% to 98.4%, and Eff improves from 0.592 to 0.564.
• Functional clustering beats anatomical clustering (Table T2): Functional clustering scores PixCorr 0.386, SSIM 0.486, and SwAV 0.320, compared with 3D-coordinate clustering (0.378, 0.491, 0.336), Schaefer-400 parcellation (0.371, 0.477, 0.333), and Schaefer-1000 parcellation (0.378, 0.475, 0.331).
• Brain tokens are spatially and semantically selective: Averaging BIT cross-attention maps across layers and heads shows brain tokens consistently contribute to specific spatial locations in predicted feature maps with a clear contralateral organization, and some tokens preferentially attend to regions associated with concepts such as faces, limbs, and text.

Methodology in Plain English

Clustering the brain. Each brain voxel is assigned to one of 128 functional clusters shared across all subjects, reducing roughly 40K voxels to 128 clusters. Cluster assignment is based on encoding "voxel embeddings" from the Brain Encoder of Beliy et al. (2024), clustered with a Gaussian Mixture Model. Because a cluster captures similar roles across individuals, the same building blocks can be used within and across brains.

Turning voxels into tokens. The Brain Tokenizer converts fMRI activations into one 512-dimensional Brain Token per cluster. Each voxel's scalar activation is multiplied by a learned 512-dimensional voxel embedding (capturing that voxel's function), and a learned 512-dimensional cluster embedding acts as the query in a single-head graph attention layer that aggregates information within the cluster.

Refining and predicting. A Cross-Transformer then lets Brain Tokens interact. The first cross-attention block has learned query tokens attending to the Brain Tokens, followed by five cross-transformer blocks, each with a self-attention layer (tokens interact with each other) and a cross-attention layer (tokens produce image features). All attention layers use eight heads, and a final projection maps to the output feature dimensionality.

Enriching sparse data. Training uses paired fMRI and image data from all subjects simultaneously, augmented with roughly 120k natural images from the unlabeled COCO portion whose fMRI responses are predicted by the Image-to-fMRI encoder of Beliy et al. (2024).

Two prediction targets. Two BIT models are trained: one predicts 256 spatial OpenCLIP ViT-bigG/14 tokens (semantic features) and one predicts multi-layer VGG features (low-level), with each position in each layer treated as a separate token and trained with an InfoNCE loss.

Two reconstruction branches. The semantic branch uses an unCLIP-style diffusion model (following Scotti et al., 2024) — SDXL adapted to condition on all 256 spatial CLIP tokens — trained in two stages: first aligning BIT's predictions to CLIP tokens with an L2 loss, then jointly training BIT and the diffusion model with the standard diffusion loss. The low-level branch inverts predicted VGG features through a Deep Image Prior: a DIP-generated image is passed through a frozen VGG network and optimized with an L2 loss so its activations match BIT's predictions.

Combining them at inference. The low-level image initializes the diffusion process with coarse structure, and the diffusion model refines it under semantic conditioning, motivated by the coarse-to-fine generation behavior reported by Kamb and Ganguli (2025). A final refinement stage passes the image through a pre-trained SDXL diffusion model without any text prompt.

Why This Matters

Impact on research. The paper argues that current limitations stem from how representations are extracted from fMRI, mapped to image features, and injected into generative models. Brain-IT replaces the common practice of compressing all voxels into a single global fMRI embedding — which discards the brain's distributed organization — with voxel-centric functional clusters that map directly to localized image features. This gives a template for cross-subject modeling at the voxel level rather than the scan level, and it demonstrates that only the per-voxel embeddings need to be adapted for a new subject.

Real-world applications discussed in the paper (cited as broader implications, not demonstrated here):

• Brain-computer interfaces that decode visual experience.
• Studying visual imagery and revealing dream content.
• Assessing disorders of consciousness.
• Neuroscience research into what is represented where in the brain and how inter-regional interactions support visual processing.

Industry relevance. Collecting fMRI is costly and time-consuming, so a method that reaches prior 40-hour performance with 1 hour of subject-specific data — and produces meaningful results from 15 minutes — substantially lowers the data barrier for deploying brain-decoding models. The shared-weight, voxel-centric design and the use of an external image corpus with predicted brain responses are directly relevant to any setting where per-subject neural data is scarce. The authors also plan to release full code with saved checkpoints and all reconstructed images, which supports benchmarking under new evaluation metrics.

Future Directions

• Improving on remaining failures: The authors state reconstructions remain imperfect, with semantics and fine-grained details sometimes inaccurate, and point to failure examples in Appendix Figure S16. They suggest exploring more expressive feature spaces where the current method fails, while noting some errors may reflect limitations of the fMRI signal itself.
• Broader neuroscience use of BIT: The authors propose applying the Brain-Interaction Transformer to other neuroscience applications, analyzing information flow between functional sub-regions to reveal what is represented where and how inter-regional interactions relate to perception.
• Adapting to other neural signals and datasets: The strong transfer-learning capability may make the pretrained model suitable for other studies and modalities, allowing adaptation with minimal data to answer task-specific neuroscientific questions.
• Open ablation questions: The appendix examines the number of clusters and their effect on reconstruction performance (Section A.3), but that content is truncated in the provided text, so the specific findings are not reported here.

Target Audience

This paper is most useful to researchers working on brain decoding and fMRI-to-image reconstruction, particularly those building on the NSD benchmark and diffusion-based decoding pipelines. It will also interest computer vision researchers studying how to condition generative models on structured, localized representations, and neuroscience groups interested in cross-subject alignment based on functional rather than purely anatomical voxel groupings. Readers without a background in diffusion models or neuroimaging will need to consult the referenced prior work (MindEye2, MindTuner, NeuroPictor, NeuroVLA) to place the comparisons in context.

Authors’ abstract

Reconstructing images seen by people from their fMRI brain recordings provides a non-invasive window into the human brain. Despite recent progress enabled by diffusion models, current methods often lack faithfulness to the actual seen images. We present "Brain-IT", a brain-inspired approach that addresses this challenge through a Brain Interaction Transformer (BIT), allowing effective interactions between clusters of functionally-similar brain-voxels. These functional-clusters are shared by all subjects, serving as building blocks for integrating information both within and across brains. All model components are shared by all clusters & subjects, allowing efficient training with a limited amount of data. To guide the image reconstruction, BIT predicts two complementary localized patch-level image features: (i)high-level semantic features which steer the diffusion model toward the correct semantic content of the image; and (ii)low-level structural features which help to initialize the diffusion process with the correct coarse layout of the image. BIT's design enables direct flow of information from brain-voxel clusters to localized image features. Through these principles, our method achieves image reconstructions from fMRI that faithfully reconstruct the seen images, and surpass current SotA approaches both visually and by standard objective metrics. Moreover, with only 1-hour of fMRI data from a new subject, we achieve results comparable to current methods trained on full 40-hour recordings.