PhysVLM-AVR: Active Visual Reasoning for Multimodal Large Language Models in Physical Environments

AI summary

Overview

• Research area: Multimodal large language models (MLLMs), embodied AI, active perception, visual question answering, and robot-oriented reasoning.
• Technical level: Intermediate. The paper assumes familiarity with MLLM architectures, visual question answering benchmarks, and the general idea of a Markov Decision Process, though it explains each of these in context.
• Scope: The paper defines a new task called Active Visual Reasoning (AVR), builds a simulated benchmark (CLEVR-AVR) and a training dataset (AVR-152k) for it, and trains an MLLM (PhysVLM-AVR-3B) that leads on the benchmark while also performing well on embodied and static visual reasoning tasks.

What This Paper Is About

Most visual reasoning work in multimodal LLMs assumes a single, complete image that fully shows everything needed to answer a question. In real physical settings, objects are frequently occluded, contained, or outside the field of view, so a model must move, look, or manipulate things to gather the missing information. The paper formalizes this as Active Visual Reasoning, a closed loop of perception, reasoning, and action, and provides the benchmark, data, and model needed to study it.

Key Contributions

1. Formal definition of the Active Visual Reasoning (AVR) task. AVR extends visual reasoning into partially observable, interactive environments where an agent must actively acquire information through sequential physical actions, integrate observations across multiple steps, and dynamically adjust decisions as new visual feedback arrives.
2. CLEVR-AVR, a simulation benchmark. Built on the Genesis physical simulation platform and extending the classic CLEVR setup into an interactive, embodied domain, it contains 10 occlusion types, 10 stacking types, and 10 composite scenarios, and measures both reasoning correctness and information-gathering efficiency.
3. AVR-152k, a large-scale training dataset with Chain-of-Thought annotations. It has three subsets: AVR-Caption (100k samples), AVR-Embodied Reasoning (50k samples), and AVR-Core (2k samples), where AVR-Core models the task as a higher-order Markov Decision Process and supplies structured CoT supervision for identifying uncertainty, predicting action-conditioned information gain, and selecting information-maximizing actions.
4. PhysVLM-AVR, an MLLM for active visual reasoning. The model reaches state-of-the-art results on CLEVR-AVR and maintains strong results on embodied reasoning (OpenEQA, RoboVQA) and passive/static visual reasoning (GeoMath, Geometry3K).

Main Findings

• Passive visual reasoning models collapse on the AVR benchmark. On CLEVR-AVR averaged across scene types, LLaVA-OV-7B scores 0 on all three metrics (ACC_ISJ, IGR, ACC_FA), Qwen2.5-VL-7B scores 4.9 / 3.7 / 2.6, R1-Onevision-7B scores 4.2 / 3.7 / 1.8, and Reason-RFT-7B scores 0.5 / 0.5 / 0.0. The authors conclude that capabilities learned on static data do not transfer to interactive, partially observable settings.
• Existing embodied MLLMs detect missing information but cannot act on it. Embodied-Reasoner-7B reaches 20.2% ACC_ISJ averaged across scene types but only 1.6% ACC_FA, and RoboBrain-7B reaches 3.6% ACC_ISJ and 1.6% ACC_FA. The paper describes this as a fundamental gap between recognizing incompleteness and strategically acquiring and integrating the missing information.
• The proposed models lead on information sufficiency judgment. PhysVLM-AVR-3B averages 90.5% ACC_ISJ and AVR-Qwen2.5-VL-7B averages 89.3%, both above GPT-4o's 88.4%, and well above Gemini-2.0-flash's 53.2%. The authors attribute this to the CoT annotations in AVR-Core teaching identification of uncertainty.
• Final answer accuracy remains the bottleneck. Per Table 1, PhysVLM-AVR-3B averages 39.7% ACC_FA and AVR-Qwen2.5-VL-7B averages 38.1%, with GPT-4o at 45.7% and Gemini-2.0-flash at 30.2%. The text highlights the large gap between a model's high judgment accuracy and its much lower final accuracy as AVR's central challenge.
• The AVR-Core dataset and its CoT annotations are both critical. In ablation on CLEVR-AVR, the full model scores 90.5 / 29.9 / 39.7 (ACC_ISJ / IGR / ACC_FA). Removing only the CoT annotations drops these to 47.6 / 18.0 / 16.9, while removing the entire AVR-Core dataset collapses them to 16.4 / 11.2 / 2.3.
• Embodied reasoning benchmarks also improve. On OpenEQA (evaluated with the original paper's LLM-score protocol using GPT-4o) and RoboVQA (BLEU1-4 scores), the authors report that PhysVLM-AVR-3B and AVR-Qwen2.5-VL-7B outperform standard multimodal models and dedicated embodied reasoning models across OpenEQA sub-tasks, and that even when trained with only 1/20 of the RoboVQA training set, their models reach BLEU scores close to or surpassing fully supervised baselines.
• Static visual reasoning generalizes too. On GeoMath and Geometry3K, PhysVLM-AVR-3B achieves the highest accuracy among compared models, outperforming GPT-4o and Reason-RFT-7B. The specific accuracy values appear only in Figure 4 and are not reported numerically in the text.

Methodology in Plain English

The authors start by defining the problem: an agent sees only part of a scene, keeps a history of everything it has observed, reasons about whether that history is enough to answer the question, and if not, picks an action (such as picking an object, moving the viewpoint, rotating the viewer, or moving an object) to reveal more. The action selection is framed as maximizing expected information about the true answer.

To study this, they build a simulated benchmark using the Genesis platform. Each question is paired with five to eight candidate responses that mix final answer options with [Action] options, and roughly three to five of those candidates are deliberately designed as distractors or sub-optimal choices that would yield less information. Scoring covers whether the model correctly judges the initial observation as sufficient or not, how often its chosen steps actually gain information, and whether the final answer is right.

For training data, they assemble 152k samples in three tiers. The 100k AVR-Caption samples teach basic perception and spatial understanding with dense captions and bounding boxes over indoor scenes from sources such as ScanNet and RT1. The 50k AVR-Embodied Reasoning samples use multi-image sequences and questions requiring spatiotemporal understanding, with reasoning chains produced by DeepSeek-R1-671B and refined by DeepSeek-V3. The 2k AVR-Core samples were collected with UMI devices across 640 real-world tabletop settings, where human experts first recorded their reasoning live and then structured it into step-by-step CoT annotations, later refined by Gemini; questions there involve an average of 3.2 reasoning-and-action steps.

The model, PhysVLM-AVR, follows a LLaVA-style architecture with Qwen2.5-3B as the language decoder and SigLIP-400M as the visual encoder, plus a max pooling layer after the encoder that cuts visual tokens by a factor of three to handle multiple images efficiently. Training proceeds in stages: connector alignment on LLaVA-Pretrain, full fine-tuning on LLaVA-OneVision-data, broader visual understanding on M4-Instruct-data plus AVR-Caption, and finally a mixture of Reason-RFT-129k, AM-DeepSeek-R1-Distilled-100k, AVR-Embodied Reasoning, and AVR-Core to build general and active reasoning. Training ran on an Ubuntu server with 8 NVIDIA A800 GPUs using PyTorch 2.6.0, transformers 3.72.0, flash attention 2, and DeepSpeed.

Why This Matters

The paper reframes a limitation the field has largely worked around: MLLMs are usually evaluated on what a single image contains, not on what an agent should do to find out what it does not contain. By defining AVR, supplying a benchmark that separates judgment, information gain, and final accuracy, and showing that existing models fail in a specific, diagnosable way, it gives the community a measurable target rather than a vague aspiration toward "embodied" models.

Real-world applications:

• Robotic manipulation and household assistance, where a robot must move or lift objects to answer questions about what is underneath, behind, or inside something.
• Warehouse and inventory inspection, where occluded or stacked items must be uncovered to count or verify them accurately.
• Search and rescue or inspection drones, where viewpoint changes are required to resolve what a fixed camera cannot see.
• Assistive and service robots that need to decide whether they already know enough to answer a person's question or must go look first.

Industry relevance is direct for groups building embodied agents and robot foundation models, since the paper shows a relatively small 3B-parameter model trained on the right data can beat much larger API models on judging when to act, and can match or exceed fully supervised baselines on RoboVQA using only 1/20 of the training data. The finding that detection of missing information does not imply effective action is a useful warning for anyone deploying current embodied MLLMs.

Future Directions

• Improve action-conditioned information gain prediction and optimal action selection. The paper's own conclusion names this as the primary next step, motivated by the gap between high ACC_ISJ and much lower ACC_FA.
• Close the multi-step integration gap. The discrepancy between knowing that more information is needed and actually synthesizing it over time to reach correct final answers remains unresolved.
• Extend AVR to more complex real-world scenarios beyond the simulated CLEVR-AVR setting and the tabletop AVR-Core collection.
• Improve sample efficiency in learning active reasoning skills, since the current approach depends on a 152k-sample mixture and extensive staged training.

Target Audience

Researchers and engineers working on multimodal LLMs, embodied AI, robot learning, and interactive visual question answering will get the most from this paper. It is also useful for benchmark designers interested in metrics that separate judgment quality from answer quality, and for practitioners who need to know where current embodied MLLMs actually break down before committing to them in a product.

Authors’ abstract

Visual reasoning in multimodal large language models (MLLMs) has primarily been studied in static, fully observable settings, limiting their effectiveness in real-world environments where information is often incomplete due to occlusion or limited field of view. Humans, in contrast, actively explore and interact with their environment-moving, examining, and manipulating objects-to gather information through a closed-loop process integrating perception, reasoning, and action. Inspired by this human capability, we introduce the Active Visual Reasoning (AVR) task, extending visual reasoning to partially observable, interactive environments. AVR necessitates agents to: (1) actively acquire information via sequential physical actions, (2) integrate observations across multiple steps for coherent reasoning, and (3) dynamically adjust decisions based on evolving visual feedback. To rigorously evaluate AVR, we introduce CLEVR-AVR, a simulation benchmark featuring multi-round interactive environments designed to assess both reasoning correctness and information-gathering efficiency. We present AVR-152k, a large-scale dataset that offers rich Chain-of-Thought (CoT) annotations detailing iterative reasoning for uncertainty identification, action-conditioned information gain prediction, and information-maximizing action selection, crucial for training agents in a higher-order Markov Decision Process. Building on this, we develop PhysVLM-AVR, an MLLM achieving state-of-the-art performance on CLEVR-AVR, embodied reasoning (OpenEQA, RoboVQA), and passive visual reasoning (GeoMath, Geometry30K). Our analysis also reveals that current embodied MLLMs, despite detecting information incompleteness, struggle to actively acquire and integrate new information through interaction, highlighting a fundamental gap in active reasoning capabilities.