AVID: A 5T fMRI Dataset for Benchmarking Auditory-induced Visual Mental Imagery Decoding

AI summary

Overview

Research area: Neuroimaging-based brain decoding (fMRI-to-image reconstruction) and vision–language benchmarking, at the intersection of cognitive neuroscience and computer vision.

Technical level: Intermediate — readers should be comfortable with fMRI preprocessing concepts, ROI-based neural inputs, and standard image/text similarity metrics. The paper is a dataset and benchmark resource, not a new model architecture.

Scope (one sentence): The paper releases AVID, a 5.0 Tesla fMRI dataset of auditory-cued visual mental imagery from nine participants (approximately 67 hours, 20,000 trials), together with fixed data splits, predefined neural inputs, and a two-axis evaluation protocol, and reports baseline results for four existing reconstruction pipelines.

What This Paper Is About

Decoding visual mental imagery — reconstructing what a person is imagining from brain activity — is harder than decoding perception, and the field lacks dedicated, training-scale datasets. Existing imagery paradigms often show participants pictures or ask them to recall recently seen images, which mixes genuine endogenous image construction with residual visual input and short-term memory replay. AVID addresses this by eliciting imagery purely through spoken Mandarin descriptions with eyes closed in a high-field 5T scanner, and by pairing the data with an evaluation protocol that separates visual similarity to the source image from semantic agreement with what was actually described to the participant.

Key Contributions

1. A visual-free, auditory-induced imagery paradigm. Participants hear spoken Mandarin scene descriptions while their eyes are closed, with no visual stimulus presented during the imagery window, intended to reduce visual contamination and short-term/working-memory replay confounds.

2. AVID, described as the first training-scale 5T fMRI dataset for visual mental imagery. It contains approximately 67 hours of recordings from nine participants across 1,200 stimuli (800 complex, 400 simple) and 20,000 total trials, structured into a scene-level complex split and a controlled simple split with explicitly specified attributes.

3. A standardized evaluation protocol that reports source-image consistency and spoken-description semantic alignment separately rather than as a single aggregate score, plus attribute-level scoring (color, position, visual identity) on the simple split.

4. Baseline results for four representative pipelines (LDM, MindEye1, MindEye2, MindBridge), treated as probes of the benchmark rather than as claims of model novelty, with released splits, ROI masks, and evaluation code.

Main Findings

• Pixel-level reconstruction is not the right target. PixCorr was very low across all methods — 0.034 to 0.056 on the complex split, and from −0.028 to 0.004 on the simple split — which the authors interpret as evidence that pixel-level recovery of the source image is an inappropriate expectation for auditory-induced imagery.

• Single-participant decoding captured the most information on the complex split. MindEye1 achieved the highest mean scores on CLIP-I (65.94%), CLIP-T (67.83%), SigLIP (70.63%), and AltCLIP (69.83%) on the complex split, while MindBridge obtained the highest mean Inception score (65.22%).

• The weakest baseline was the cross-participant transfer setting. MindEye2, evaluated with leave-one-participant-out pretraining followed by target-participant fine-tuning, scored lower on the complex split (Inception 55.75%, CLIP-I 57.81%, CLIP-T 57.95%, SigLIP 59.08%, AltCLIP 58.82%, TIFA 24.34%), pointing to cross-participant transfer and adaptation as an open problem.

• TIFA did not track the other semantic metrics on the complex split. LDM (32.56%) and MindBridge (32.25%) scored slightly above MindEye1 (30.27%), even though MindEye1 led on the identification-based text metrics.

• High SSIM on the simple split should not be read as accurate recovery. Two methods exceeded 0.90 SSIM (MindEye1 at 0.962, MindBridge at 0.906) even though PixCorr stayed near zero; the authors attribute this partly to large low-texture backgrounds inflating structural similarity.

• Simple-split TIFA was higher than complex-split TIFA but still far from ceiling, with LDM highest at 48.92%, followed by MindEye1 (44.34%), MindBridge (40.32%), and MindEye2 (39.34%).

• Attribute recovery was uneven. Against analytical chance levels of 25.76% (color), 20.00% (position), and 1.05% (visual identity), MindBridge had the highest mean color score (35.55 ± 4.04%), MindEye1 the highest mean position (35.34 ± 8.56%) and identity (3.38 ± 1.96%) scores, and identity accuracy across methods ranged only from 2.07% to 3.38%.

• The input is not dominated by early visual cortex. Early visual cortex accounted for approximately 4.5% (complex) and 5.8% (simple) of selected vertices in the primary input, which is distributed across frontoparietal, sensorimotor, temporal, and language-selective regions.

• Retained performance under a stricter language control. The authors also evaluated the ROI_m \ (ROI_s ∪ ROI_l) input, which additionally removes language-selective vertices, and report that retained performance suggests decoding beyond speech- and language-selective regions (Appendix B.13).

Methodology in Plain English

Data collection. Nine healthy native Mandarin speakers aged 18 to 25 were scanned on a 5T MRI scanner (TR = 1.5 s, 2.0 mm isotropic voxels, with 0.7 mm anatomical images). Each 12-second trial began with a spoken description, followed by an 8-second eyes-closed imagery window, a 250 ms beep, and a 4-second reset interval. Participants rated imagery vividness on a 4-point button box and were instructed to imagine a static scene from an observer perspective.

Two complementary stimulus sets. The complex split used 800 scene descriptions derived from Caption Scene Dataset image–caption pairs originally based on COCO-CN, synthesized into 4–6 s audio clips of 12–20 Mandarin characters, each shown twice. The simple split used 400 in-house image–caption pairs that explicitly specify object identity, alphanumeric content, color, count, and position on a neutral background, each shown twice.

Defining the neural input. A functional localizer with three auditory conditions (Mandarin sentences, non-word sequences, non-linguistic acoustic controls) plus a repeat-based noise-ceiling analysis produced three ROI masks: task-responsive, speech-responsive, and language-selective. The primary input is the task-responsive set minus the speech-responsive set, so all baselines are compared on identical pre-defined vertices instead of each choosing its own input.

Evaluation. Reconstructions are scored on two separate axes. Source-image consistency uses PixCorr, SSIM, Inception, and CLIP-I. Spoken-description alignment uses CLIP-T, SigLIP, AltCLIP, and TIFA. On the simple split, a frozen Qwen3-VL-8B model answers closed-set questions about color, position, and identity, scored as class-balanced accuracy across seven participants.

Baselines. Four existing pipelines were run largely with their original training and generation procedures, adapted only for data loading, input formatting, splits, and metrics: LDM and MindEye1 as single-participant models, MindEye2 as leave-one-participant-out transfer, and MindBridge as multi-participant joint training.

Why This Matters

Impact on research. AVID makes imagery decoding a comparable, reproducible problem: fixed splits, shared ROI definitions, and a protocol that acknowledges imagery has no unique pixel-level ground truth. The separation of source-image consistency from description alignment gives the field a way to report partial success honestly rather than collapsing everything into one number.

Real-world applications (as motivated or implied by the paper):

• Neuroprosthetic communication interfaces that could help externalize internal experience.
• Brain-to-image reconstruction systems that operate without any external visual prompt.
• Clinical and cognitive research on imagery vividness and individual differences in inner experience.
• Benchmark development for evaluating generative models against neural data.

Industry relevance. The dataset and code are released openly (CC BY 4.0, DOI 10.57760/sciencedb.33666, code at github.com/lishurui0612/AVID-benchmark) with a validated Croissant metadata file, in BIDS format with GLMSingle betas, FreeSurfer outputs, ROI masks, and split files — a usable substrate for groups building decoding or generative-model pipelines. The reported baseline gaps, especially in cross-participant transfer, mark concrete engineering targets.

Future Directions

• Models designed for imagery rather than borrowed from perception. The authors state that closing the gap with perception-oriented decoding will likely require architectures built specifically for visual mental imagery.

• Better cross-participant transfer and adaptation. The gap concentrated in MindEye2's leave-one-participant-out setting is flagged as an important direction.

• Improving attribute-level fidelity. Visual identity recovery barely exceeded chance (2.07% to 3.38% against 1.05%), and color and position recovery remained incomplete despite exceeding chance.

• Broadening population and language coverage. The release is limited to young, healthy, pre-screened Mandarin speakers with relatively vivid self-reported imagery (VVIQ 52 to 81), and the simple split came from a subset of the complex-split participants — so generalization to other languages, ages, lower-vividness individuals, and clinical groups is untested.

Target Audience

Researchers in brain decoding and fMRI-to-image reconstruction who need a training-scale imagery benchmark; cognitive neuroscientists studying mental imagery, imagery vividness, and the perception–imagery relationship; machine learning practitioners working on multimodal generative models evaluated against neural data; and benchmark or dataset curators interested in evaluation protocols for tasks without a unique ground-truth target.

Authors’ abstract

Visual mental imagery (VMI) decoding uses brain activity to reconstruct internally generated visual scenes, providing a potential pathway for externalizing internal experiences and developing neuroprosthetic communication interfaces. However, progress is limited by the lack of dedicated, training-scale VMI datasets. Current VMI paradigms often rely on visual prompts or short-term memory images, introducing visual and mnemonic confounds that make it difficult to disentangle neural activity associated with endogenous image construction from signals driven by prior or concurrent visual input. Furthermore, existing VMI datasets offer limited descriptive control over individual visual attributes, making it difficult to systematically assess fine grained visual details. To address these gaps, we introduce AVID, the first dedicated training-scale 5.0 Tesla functional magnetic resonance imaging (fMRI) resource for auditory-induced VMI decoding. AVID comprises approximately 67 hours of dense recordings from nine participants. The dataset is structured into two complementary splits: a scene-level complex split using naturalistic Mandarin descriptions and a controlled simple split with explicitly specified visual attributes. We further formalize a standardized evaluation protocol that decouples source-image consistency from semantic alignment, allowing for a more nuanced assessment of imagery reconstruction. Baseline qualitative and quantitative results show that current reconstruction pipelines remain limited on AVID, although the most successful reconstructions preserve recognizable scene level semantics from spoken descriptions. By combining dedicated training-scale neural recordings and standardized evaluation resources, AVID provides both a data foundation for imagery-specific model development and a benchmark for evaluating VMI decoding with substantially reduced sensory contamination.