MicroVQA++: High-Quality Microscopy Reasoning Dataset with Weakly Supervised Graphs for Multimodal Large Language Model

AI summary

Overview

• Research area: Computer Vision (cs.CV), specifically multimodal large language models (MLLMs) applied to microscopy and biomedical imaging, plus dataset construction and graph-based data filtering.
• Technical level: Intermediate. The paper assumes familiarity with CLIP embeddings, natural language inference, graph neural networks (GraphSAGE, GATv2), supervised fine-tuning, LoRA, and GRPO reinforcement learning.
• Scope: The paper builds a large-scale microscopy visual question answering (VQA) corpus, MicroVQA++ (20,000 training questions and 6,000 test questions), from BIOMEDICA image–caption pairs, filters it with a heterogeneous image–caption–QA graph called HiCQA-Graph, and shows that a 4B-parameter MLLM fine-tuned on it matches GPT-5 on the MicroVQA benchmark.
• arXiv: 2511.11407v1 [cs.CV], 14 Nov 2025. Authors: Manyu Li, Ruian He, Chenxi Ma, Weimin Tan, Bo Yan (Fudan University). License: CC BY-SA 4.0.

What This Paper Is About

Multimodal LLMs are increasingly used for biomedical images, but progress in microscopy reasoning is bottlenecked by the absence of a large, high-quality training set — the existing MicroVQA benchmark has only 1,042 questions and 255 images, which is enough to test models but not to improve them. The authors address this by converting expert-written figure–caption pairs from the BIOMEDICA archive into question–answer and multiple-choice data using an MLLM agent, then filtering out inconsistent or hallucinated samples with a graph that checks agreement between image, caption, and generated QA. The goal is a microscopy training corpus large enough to actually raise the reasoning performance of small open-source models.

Key Contributions

1. A QA and MCQ construction pipeline. Starting from peer-reviewed image–caption pairs, an MLLM agent extracts answers, generates grounded QA pairs across three reasoning capacities (Expert Visual Understanding, EU; Hypothesis Generation, HG; Experiment Proposal, EP), then produces multiple-choice questions with explicit chain-of-thought rationales.
2. HiCQA-Graph, a heterogeneous Image–Caption–QA graph for cross-modal consistency filtering. The graph fuses NLI-based textual entailment between caption and answer, CLIP-based vision–language alignment between image and QA, and agent-derived capacity signals, using message passing (GraphSAGE, GATv2) to flag unreliable samples. The authors state it is the first graph that jointly models (image, caption, QA) for cross-modal consistency filtering.
3. The MicroVQA++ dataset itself, with 20,000 training questions and 6,000 test questions, and 8,594 / 5,198 unique images in the two splits. Relative to MicroVQA, this is 19.2x and 5.8x more questions and 33.7x and 20.4x more images for train and test respectively; MicroVQA++ also carries chain-of-thought rationales (20,000 train / 6,000 test) where MicroVQA has none.
4. Evidence that data quality, not model size, drives microscopy reasoning. After SFT on the MicroVQA++ training split, InternVL3.5-4B-Instruct reaches 59.4 average accuracy on MicroVQA, matching GPT-5 (59.4) and setting state-of-the-art among open-source MLLMs.

Main Findings

• A 4B model matches the strongest closed models on MicroVQA. InternVL3.5-4B-Instruct fine-tuned on MicroVQA++ scores EU 61.0, HG 56.9, EP 61.3, average 59.4, versus GPT-5 at EU 63.3, HG 58.9, EP 53.9, average 59.4; o3 reaches 59.3 and the human baseline is 50.3 (random is 22.0).
• Fine-tuning yields large relative gains for small models. On MicroVQA, InternVL3.5-2B-Instruct improves by 22.7% and InternVL3.5-4B-Instruct by 27.5% relatively on average after SFT (44.4 to 54.5, and 46.6 to 59.4). Even LLaVA-Med-Mistral-7B rises from 43.0 to 53.7.
• The MicroVQA++ test set is deliberately harder and less redundant. It follows a higher Bloom's level difficulty distribution than MicroVQA, and its train/test overlap is intentionally low, so SFT yields only a modest +0.6 absolute gain for InternVL3.5-4B-Instruct there (to 41.3 average), which the authors read as reduced dataset bias and a stronger generalization requirement.
• Filtering helps, and the graph filters best. In the filtering comparison (2B/4B), the full unfiltered set gives 54.2 / 58.2 average; NLI- or CLIP-only filtering is inferior; NCLIP (a linear combination of both) at the top-75% keep ratio gives 54.5 / 58.2; HiCQA-Graph at the top-75% keep ratio gives the best result, 54.5 / 59.4.
• Every component of HiCQA-Graph contributes. Removing CLIP (54.1 / 58.6), NLI (53.2 / 58.3), the capacity signal (54.0 / 58.0), or the cross-modal consistency token (54.2 / 57.4) all reduce average accuracy relative to the full model at 54.5 / 59.4.
• Multiple-choice supervision beats free-form QA supervision. SFT with MCQ achieves 59.4 average versus 42.4 for SFT with QA; GRPO on MCQ reaches 57.4. GRPO on QA data without a format reward led to severe reward hacking.
• The pipeline is computationally cheap at the graph stage. Per image, CLIP forward takes 32.29 ms and NLI forward 77.05 ms; graph training is 3.56 ms forward and 10.21 ms backward with 129.32 ms end-to-end (the body text of Section 4.5 states 129.42 ms), and testing is 0.63 ms forward with 109.82 ms end-to-end, on a single RTX 3090.
• Three recurring failure modes. Visual grounding errors (mislocalizing small or overlapping structures), distractor susceptibility (choosing options by lexical overlap), and cross-modal priors (caption-like hallucinations such as unsupported infection/inflammation tropes).
• CLIP space shows the datasets are related but distinct. In t-SNE plots (perplexity 30, 1,000 iterations), MicroVQA++ images form a compact, uniform arrangement while MicroVQA forms two tight clusters; the text spaces show clear distributional boundaries between human expert and MLLM-agent questioning styles, with non-negligible overlap regions described as transfer-friendly zones.

Methodology in Plain English

The authors start from BIOMEDICA, an automatically curated biomedical vision–language dataset built from PubMed Central Open Access articles containing approximately 24M image–caption pairs, of which about 10.4% are microscopy-related — roughly 2.5M image–caption pairs. Because captions in published papers have already been written and checked by human experts, they are treated as trustworthy supervision seeds.

The pipeline runs in three stages. Stage one samples microscopy figure–caption pairs and uses an MLLM agent to extract answer spans from the caption and build initial QA pairs across the three reasoning capacities. Stage two passes each sample through HiCQA-Graph. In this graph, images and QA pairs are encoded with CLIP ViT-L/14 (768-dimensional, images resized to 224×224), and each gets an extra one-dimensional consistency score; captions are encoded with CLIP's text Transformer (content longer than 75 tokens is summarized first). Four directed relation types connect the nodes: Image–described_by–Caption, Image–asked_about–QA, Caption–supports–QA (edge weight is NLI entailment probability), and QA–similar–QA (edge weight is nonnegative cosine similarity). Two stacked hetero-convolution layers pass messages — GraphSAGE for cross-type image-to-text edges, GATv2 with four attention heads for edge-attribute-aware edges — and two MLP heads on QA nodes predict a soft "keep" score and a three-way EU/HG/EP capacity label. Stage three feeds the validated items to an MLLM agent to produce multiple-choice distractors and chain-of-thought rationales, followed by human screening for the test set. The paper reports no information leakage with respect to MicroVQA.

For experiments, models were fine-tuned with LLaMA-Factory using LoRA rank 16 on attention projections and MLP blocks, AdamW at 1e-4 with weight decay 0.1, cosine decay with 5% warm-up, 3 epochs in bfloat16, gradient clipping at 1.0, sequence packing with context length 4,096, and an effective batch size of 256 sequences; microscopy images were resized so the short side was 448 pixels. GRPO used the verl framework with four RTX 3090s, learning rate 5e-6, KL coefficient 0.05, entropy bonus 0.001, value-loss coefficient 0.5, 6 sampled completions per prompt, and 1 epoch over the MCQ subset. The NLI scorer is a fine-tuned XLM-R model trained on XNLI and MultiNLI, and the CLIP encoder is OpenCLIP ViT-L/14 initialized from BIOMEDICA-pretrained weights. Evaluation follows MicroVQA's protocol: regex-based MCQ accuracy for EU, HG, and EP.

Why This Matters

This work reframes the microscopy bottleneck as a data problem rather than a model-size problem, and shows that carefully constructed supervision can let a 4B-parameter model compete with the strongest proprietary systems on a domain-specific reasoning benchmark. It also offers a reusable recipe: use expert-written captions as a trustworthy anchor, then let a graph arbitrate consistency between image, text, and generated question instead of relying on prompt engineering or heavy manual heuristics.

Real-world applications:

• Computational pathology and clinical microscopy: training or adapting smaller, deployable models that can answer structured questions about tissue, cell, subcellular, and atomic-scale images across light, fluorescence, and electron modalities.
• Research-assistant tooling for life scientists: a model that can describe visual phenomena, propose mechanistic hypotheses, and suggest follow-up experiments maps directly onto the EU/HG/EP workflow the dataset is built around.
• Automated educational assessment: the MCQ-with-rationale format plus the reported Bloom's-level difficulty distribution supports exam-style question generation and evaluation for microscopy and biology training.
• Data curation at scale for biomedical ML: the HiCQA-Graph filtering approach transfers to other noisy vision–language corpora where image, caption, and derived text should agree.

Industry relevance: the finding that a 4B model can match GPT-5 on this benchmark matters for groups that cannot run frontier-scale proprietary models, and the reported per-image latency (129.32 ms training end-to-end, 109.82 ms inference end-to-end on a single RTX 3090) indicates the graph stage is not a deployment obstacle. The authors also state that code and dataset will be released after the review process concludes.

Future Directions

• Closing the generalization gap on the MicroVQA++ test set, where SFT on the intentionally low-overlap training split produced only a +0.6 absolute gain for InternVL3.5-4B-Instruct, versus much larger gains on the MicroVQA benchmark.
• Better reinforcement learning for microscopy reasoning. GRPO on MCQ reached 57.4 average versus 59.4 for MCQ SFT, and GRPO on QA data caused severe reward hacking — the paper defers details to the appendix, leaving the question of a stable reward design open.
• Reducing the failure modes identified in error analysis: visual grounding of small and overlapping structures, susceptibility to lexically overlapping distractors, and image-unsupported biomedical tropes such as infection or inflammation.
• Scaling beyond the current selection. Only a microscopy subset of BIOMEDICA (roughly 2.5M image–caption pairs) was available; the paper does not report how far performance would continue to improve with more of that pool, nor does it report the Bloom's-level proportions numerically.

Target Audience

Researchers and engineers working on multimodal LLMs, biomedical and microscopy image analysis, and dataset construction or data-centric machine learning. It is most useful for readers interested in weak supervision, graph-based denoising, and reinforcement learning for vision–language models, as well as practitioners who want a concrete template for turning expert-written literature into large-scale reasoning supervision. Readers need some background in CLIP-style embeddings, graph neural networks, and LLM fine-tuning to follow the methodology, but the dataset-scale and benchmark results are accessible without it.

Authors’ abstract

Multimodal Large Language Models are increasingly applied to biomedical imaging, yet scientific reasoning for microscopy remains limited by the scarcity of large-scale, high-quality training data. We introduce MicroVQA++, a three-stage, large-scale and high-quality microscopy VQA corpus derived from the BIOMEDICA archive. Stage one bootstraps supervision from expert-validated figure-caption pairs sourced from peer-reviewed articles. Stage two applies HiCQA-Graph, a novel heterogeneous graph over images, captions, and QAs that fuses NLI-based textual entailment, CLIP-based vision-language alignment, and agent signals to identify and filter inconsistent samples. Stage three uses a MultiModal Large Language Model (MLLM) agent to generate multiple-choice questions (MCQ) followed by human screening. The resulting release comprises a large training split and a human-checked test split whose Bloom's level hard-sample distribution exceeds the MicroVQA benchmark. Our work delivers (i) a quality-controlled dataset that couples expert literature with graph-based filtering and human refinement; (ii) HiCQA-Graph, the first graph that jointly models (image, caption, QA) for cross-modal consistency filtering; (iii) evidence that careful data construction enables 4B-scale MLLMs to reach competitive microscopy reasoning performance (e.g., GPT-5) and achieve state-of-the-art performance among open-source MLLMs. Code and dataset will be released after the review process concludes.