VisReason: A Large-Scale Dataset for Visual Chain-of-Thought Reasoning

AI summary

Overview

Research area: Computer Vision / Multimodal Large Language Models (MLLMs), specifically datasets and training methods for visual Chain-of-Thought (CoT) reasoning.

Technical level: Intermediate. The paper assumes familiarity with MLLM architectures (visual encoders, projectors, LoRA fine-tuning), Chain-of-Thought prompting, and visual grounding metrics such as IoU.

Scope: This paper introduces VisReason, a 489K-example dataset (plus a 165K curated subset, VisReason-Pro) of multi-round, region-of-interest-grounded visual reasoning traces, and evaluates fine-tuned MLLMs on stepwise visual reasoning, RoI localization, and depth-augmented spatial reasoning.

What This Paper Is About

Chain-of-Thought prompting has worked well for text-only LLMs, but multimodal models are mostly trained to answer visual questions directly, without supervision over intermediate reasoning steps. The authors argue that existing visual-CoT datasets are too small, too narrow in domain, single-step rather than multi-round, and mostly 2D, so they build VisReason and VisReason-Pro to explicitly supervise a global-to-local "zoom-and-verify" reasoning process grounded in bounding boxes and depth cues.

Key Contributions

1. VisReason dataset. A large-scale corpus of 489K annotated examples spanning four domains (text/doc understanding, fine-grained recognition, general VQA, and spatial-aware relational reasoning), each with multi-round, RoI-grounded step-by-step supervision and depth-informed annotations.
2. VisReason-Pro. A 165K-example high-fidelity subset produced under stronger GPT-4.1-series guidance (GPT-4.1-Mini), with richer rationales and depth-augmented spatial annotations derived from monocular depth estimation and semantic segmentation. It includes both multi-round traces and a single-round distilled variant.
3. Evaluation suite. A held-out suite built on VisReason-Pro for measuring fine-grained reasoning, RoI localization, and depth-augmented spatial reasoning through multi-step visual CoT tasks.
4. Unified training and inference pipeline. A formulation in which the model iteratively emits a rationale and a bounding box for the next region of interest, crops the image to that region, and appends the new visual features to the context.

Main Findings

• Overall benchmark performance. VisReason-Pro-7B achieves the highest average score (0.802) on the Visual-CoT benchmark, improving over its Qwen2.5-VL-7B backbone (0.770) and outperforming InternVL-2.5-8B (0.738), LLaVA-NeXT-8B (0.705), VisCoT-7B (0.614), CoF-SFT-7B (0.737), and MiniGPTv2 (0.452).
• Largest gains on fine-grained and spatial tasks. Fine-grained recognition on Birds-200-2011 rises from 0.681 (Qwen2.5-VL-7B) to 0.831, and spatial relation reasoning on Open Images rises from 0.498 to 0.805.
• Doc/Text trade-off is largely a format artifact. Under multi-round CoT inference, text/document scores drop relative to the Qwen2.5-VL-7B base model. With a matched direct-QA prompt, the same VisReason-Pro-7B checkpoint reaches 0.916 on Text/Doc (compared with 0.920 for Qwen2.5-VL-7B direct QA), while multi-round CoT remains stronger on fine-grained (0.831 vs 0.709) and spatial reasoning (0.722 vs 0.599).
• Better RoI localization. On the VisReason-Pro held-out detection suite, VisReason-Pro reaches IoU@0.5 of 0.34 and IoU@0.75 of 0.23, versus 0.29/0.19 for LLaVA-NeXT, 0.27/0.13 for VisReason, 0.14/0.06 for MiniGPTv2, and 0.08/0.03 for InternVL-2.5.
• Stronger ordinal-depth and grounding metrics. VisReason-Pro reports a grounded ratio of 0.276 (vs 0.039 LLaVA-NeXT, 0.011 InternVL-2.5, 0.035 Qwen2.5-VL), BBox IoU of 0.278 (vs 0.207, 0.214, 0.115), and depth absolute difference of 0.266 (vs 0.394, 0.290, 0.294). The paper states this depth error derives from monocular-depth ordinal cues and does not indicate metric 3D reconstruction accuracy.
• Transfer to a second backbone. Fine-tuning InternVL-2.5-8B on VisReason for one epoch improves fine-grained (0.747 to 0.823) and spatial relational (0.621 to 0.669) scores with a similar overall average (0.738 to 0.740), while Doc/Text declines (0.842 to 0.815), suggesting the effect is not specific to Qwen2.5-VL.
• Human-rated quality improves. In a blinded study with 30 raters on 20 sampled items per method, VisReason-Pro-7B scores 4.19 on Answer Accuracy, 4.46 on Grounded Faithfulness, and 4.37 on Stepwise Clarity & Sufficiency (mean 4.34), compared with 2.58 mean for VisCoT-7B and 2.06 for MiniGPTv2.
• Ablations confirm both components matter. Training on VisReason alone raises the average from 0.770 to 0.777; adding VisReason-Pro raises it to 0.789; adding the adaptive zoom-in strategy yields the best overall average of 0.802.
• External benchmark results depend on inference protocol. Under multi-round CoT, VisReason-Pro-7B scores 0.777 on MME and 0.603 on V*; under matched direct-QA it recovers to 0.856 and 0.791, close to the Qwen2.5-VL direct-QA baseline of 0.861 and 0.791.
• Data quality audit. A stratified blind human audit of 2,200 examples (200 per source dataset) reports 99.1% answer consistency, 98.5% target containment, 95.0% RoI tightness, and 86.5% for both rationale necessity and rationale faithfulness.
• RoIs tend to be small. Answer-critical regions average about 13.2% of the image area, and most samples resolve in 2–3 rounds, with harder spatial or depth-augmented cases extending to 4.

Methodology in Plain English

The authors start from the Visual-CoT seed dataset and enrich every image–question–answer triple with process-level supervision. For VisReason, they prompt GPT-4.1-Nano to produce a concise scene description, a normalized region of interest, and a brief rationale. They adjust each predicted RoI so it tightly contains the ground-truth box and iteratively refine it by zooming from global to local, stopping when the RoI area is no more than twice the ground-truth area or when a round budget of at most 3 is reached. When the target object already occupies a large portion of the image (more than 30% of the area), they skip cropping and provide a single detailed reasoning step instead.

For VisReason-Pro, they build primarily on the GQA portion of Visual-CoT and first derive pseudo-depth and segmentation cues (object IDs, categories, pixel boxes, ordinal depths) using monocular depth estimation and semantic segmentation. These structured signals plus the image go to GPT-4.1-Mini, which generates depth-augmented questions combining 2D layout relations (left of, above) with ordinal depth relations (in front of, behind), along with a consistent ground-truth box. The same verify-and-fix routine runs with a round budget of at most 4.

The resulting annotations supervise a model that, given an image and query, generates a sequence of steps. Each step contains a textual rationale and a bounding box for the next region of interest; the image is cropped to that box, new visual features are extracted, and both are appended to the context for the next step, starting from the full image. Training is supervised fine-tuning with LoRA on Qwen2.5-VL-7B: two epochs on VisReason without VisReason-Pro for the VisReason-7B model, then one additional epoch on the full corpus for VisReason-Pro-7B. The learning rate is 2×10⁻⁵ for the LLM backbone and projector, and the ViT encoder is frozen. InternVL-2.5-8B is fine-tuned on VisReason for one epoch as a transfer check.

Evaluation follows the Visual-CoT protocol across 11 source test sets grouped into the four domains, with additional results reported on MME and V*Bench. The automatic judge is GPT-4o-mini, assigning a scalar score in [0, 1] per example, run five times and averaged. Training used four NVIDIA H200 GPUs (144 GB each) with DeepSpeed ZeRO-2, FlashAttention-2, and bf16 precision; evaluations ran on a single NVIDIA A800 GPU (80 GB).

Why This Matters

Impact on research. The paper targets what it identifies as three persistent gaps in visual CoT resources: limited scale and domain diversity, insufficient multi-round supervision, and predominantly 2D annotations with weak depth or occlusion-awareness. By releasing both a large corpus and a curated depth-aware subset with an accompanying evaluation suite, it provides process-level supervision that prior work (such as VisCoT, VPD, Visual CoT, DualFocus, Chain-of-Spot, CogCoM, and VoCoT, as cited) does not combine at the same scale.

Real-world applications. The paper does not enumerate specific deployment scenarios; the capabilities it trains map onto tasks where evidence is small or spatially subtle, such as:

• Document and receipt/invoice understanding (the dataset includes DocVQA, DUDE, SROIE, TextVQA, and TextCaps sources).
• Fine-grained visual recognition where the distinguishing detail occupies a small image region (Birds-200-2011).
• Spatial and relational scene understanding, including ordinal depth relations such as in front of and behind (GQA, VSR, Open Images).
• Assistive or embodied interaction settings where a model must decide where to look and verify local evidence rather than answer from a single global view.

Industry relevance. For teams building multimodal assistants, the paper offers a reusable annotation format and training recipe that can be plugged into an existing MLLM backbone, plus a caveat that matters in deployment: multi-round CoT inference increases latency and can underperform direct-QA prompting on short-answer or multiple-choice protocols, so the choice of inference format should match the evaluation format.

Future Directions

• Self-correcting rollback. The authors identify "cascading localization errors" as a mechanistic vulnerability: because each step is conditioned on previously cropped features, an initially misaligned RoI can trap perception in an irrelevant region, and the current mechanism has no zoom-out or backtracking policy.
• Reinforcement learning over cropping policies. The paper notes that RL could in principle learn where to crop, but doing so from sparse final-answer rewards is difficult and may encourage reward hacking, positioning large-scale SFT datasets like VisReason as trajectory supervision for future multimodal RL or agentic training.
• Improving document reasoning rationales. The human audit found the main weakness in Text/Doc rationales, where local crops can miss global layout cues required for document reasoning.
• Beyond pseudo-depth. Monocular depth provides useful ordinal cues but remains noisy; the paper explicitly frames depth supervision as ordinal rather than metric 3D ground truth, leaving room for stronger spatial supervision.

Target Audience

Researchers and engineers working on multimodal large language models, visual reasoning, and visual grounding, particularly those interested in Chain-of-Thought supervision, dataset construction pipelines, or spatial and depth-aware reasoning. It is also relevant to practitioners who need to evaluate the trade-off between verbose multi-round reasoning and direct question answering in deployed systems.

Authors’ abstract

Chain-of-Thought (CoT) prompting has proven remarkably effective for eliciting complex reasoning in large language models (LLMs). Yet, its potential in multimodal large language models (MLLMs) remains largely untapped, hindered by the absence of large-scale datasets that capture the rich, spatially grounded reasoning intrinsic to visual understanding. Existing visual-CoT resources are typically small, domain-specific, or lack the human-like stepwise structure necessary for compositional visual reasoning. In this paper, we introduce VisReason, a large-scale dataset designed to advance visual Chain-of-Thought reasoning. VisReason comprises 489K annotated examples spanning four diverse domains, each featuring multi-round, human-like rationales that guide MLLMs through interpretable visual reasoning steps. Building upon this, we curate VisReason-Pro, a 165K subset produced with a stronger expert-level GPT annotator, enriched with detailed reasoning traces and 3D spatial grounding via depth-informed annotations. Fine-tuning the state-of-the-art Qwen2.5-VL model on VisReason and VisReason-Pro yields substantial improvements in step-by-step visual reasoning accuracy, interpretability, and cross-benchmark generalization. These results demonstrate that VisReason equips MLLMs with more systematic and generalizable reasoning capabilities. We envision VisReason as a cornerstone for cultivating human-like visual reasoning, paving the way toward the next generation of multimodal intelligence.