Learning to Think and Explore in 3D World

AI summary

Overview

• Research area: Computer vision and vision-language models (VLMs) for 3D spatial reasoning, combined with reinforcement learning for embodied/agentic exploration.
• Technical level: Advanced. The paper assumes familiarity with POMDPs, GRPO-style RL, supervised fine-tuning of multimodal LLMs, and 3D scene rendering.
• Scope: The paper proposes Act3D, a framework that teaches a VLM to answer questions about a 3D scene by iteratively choosing camera actions, rendering new first-person views, and reasoning over what it sees, trained first by trajectory supervision and then by outcome-only reinforcement learning.

What This Paper Is About

Most 3D vision-language models are given the evidence up front, in the form of point clouds, multi-view images, videos, or RGB-D scans, and are only asked to reason over it. This paper instead asks whether a VLM can learn to acquire the missing evidence itself: starting from a single first-person image of a hidden 3D scene, the model must decide where to move or look next, observe the resulting render, and only then answer. The goal is a model that learns when and how to explore, using nothing but whether its final answer was correct as the training signal for exploration.

Key Contributions

1. An interactive formulation of 3D reasoning. The task is cast as a partially observable Markov decision process where the 3D scene stays external to the model and is accessed only through a deterministic action-observation loop (navigation actions plus a terminal stop(y) answer action).
2. A two-stage training recipe. Trajectory-supervised fine-tuning on curated reason-act-observe demonstrations establishes the interaction protocol, followed by GRPO-based reinforcement learning whose reward depends primarily on final-answer correctness.
3. Outcome-only exploration learning. During RL, no supervision is given over which viewpoints to visit or which actions to take; the exploration policy must emerge from credit assignment on task performance alone.
4. Empirical validation and behavioral analysis. Act3D is evaluated on six 3D benchmarks against both passive-perception and active-interaction baselines, with analysis showing question-conditioned exploration that scales with how much exploration a question actually requires.

Main Findings

• State-of-the-art average accuracy. Act3D trained on mixed data reaches 53.2% average accuracy across six benchmarks, improving on its Qwen3.5-9B base model (31.2% average) by 22.0% and surpassing Gemini 3.6 Flash (52.2%) by 1.0%.
• Beats open-source 3D methods with less input. Act3D outperforms existing open-source 3D methods while using 50–70% fewer input frames and lower-resolution observations.
• RL adds a lot over supervised initialization. In the mixed-data setting, RL lifts average accuracy from 48.6% to 53.2% (+4.6%), improving five of six benchmarks, with the largest gains on ARKit (+14.1%) and SPBench-MV (+8.1%).
• Gains track exploration demand. ScanQA validation questions stratified by a geometry-based estimate of needed actions show improvements rising from 0.6% in the lowest-demand group to 16.3% in the highest, with RL contributing an additional 10.9% over SFT in that hardest group.
• Exploration behavior changes during training. Average interactions per episode rise from roughly 3.3 to 8.3 during mixed-data training, while ScanQA-only validation accuracy rises from 40.2% to 45.5%.
• Cross-domain transfer. The abstract and introduction report an average improvement of 5.9% over the supervised initialization across four out-of-domain benchmarks. The ScanQA-only experiment in Section 4.2 separately reports 47.8% to 50.6% overall (+2.8%), with the other five benchmarks rising from 49.5% to 51.6% (+2.1%), including MP3D (+6.0%) and SQA3D (+3.8%).
• One image is enough to start. A single rendered frame gives 41.0% on ScanQA, close to the 41.4% from a bird's-eye view plus four rendered frames, while a BEV plus one frame drops to 36.7%.
• Rendering beats image retrieval. Pose-aligned rendering outperforms nearest-view retrieval of real ScanNet images on all 13 evaluation checkpoints, with 10.6% higher accuracy on average, showing that action-observation consistency matters more than photorealism.
• Relative actions beat absolute poses. Relative camera movements reach 41.4% on ScanQA versus 36.7% for predicting absolute camera poses.
• SFT accelerates but is not strictly required. RL without SFT improves ScanQA accuracy from 28.9% to 34.8–37.9% after 80 steps at about one interaction per episode, and to 41.4% at 470 steps with 2.0 interactions; SFT-initialized RL goes from 39.5% to 42.7% with interactions rising from 2.7 to 8.7.
• The interaction reward mostly buys exploration. Adding 0.05 per interaction raises average interactions from 5.1–6.3 to 9.4–10.1; it helps held-out benchmarks (SPBench-MV +6.4%) but slightly reduces accuracy on training domains (0.3–2.3%).
• Emergent qualitative behaviors. The policy searches for occluded targets, avoids unnecessary exploration when the answer is already visible, and gathers complementary viewpoints for spatial comparisons, refining hypotheses as new renders arrive.

Methodology in Plain English

The scene is a pre-built 3D reconstruction that the model never sees directly. The model gets one 640×480 first-person render and a question, then chooses from 80 navigation actions (six translations and four rotations, each with a multiplier from 1 to 8) or emits an answer, for up to twelve interaction steps. Translations step 1 m forward/backward and 0.6 m otherwise; rotations step 30 degrees. Actions are relative to the camera ("move left," "turn right"), which the authors find works better than predicting absolute coordinates, because relative moves are scale-invariant and match how the model describes motion in language. Every action produces a fresh render, and the model's own reasoning text becomes part of the history it conditions on.

Training happens in two stages. First, GPT-5.4 generates reason-act-observe demonstrations inside the environment, which are filtered for valid interactions, correct answers, and sufficient visible geometry, yielding 5,712 trajectories, 20,603 rendered observations, and 830 scenes; the model is fine-tuned on these with the observation and prompt tokens masked from the loss. Second, GRPO is run on a much larger set of 51,543 question-answer pairs, sampling eight rollouts per question and rewarding each with a weighted sum: an LLM-judge verdict (0.3), exact match (0.2), a format penalty (0.2), and a small interaction term (0.05, capped at 12 steps). The interaction term pays full credit immediately if the answer was correct and the annotated object was actually visible in some rendered view, so the model is rewarded for obtaining evidence rather than for taking many actions. Crucially, nothing in the reward specifies a viewpoint or trajectory.

Why This Matters

This work shifts 3D question answering from passive perception to active evidence gathering, which is the regime real agents actually operate in: the information needed to answer a question is usually not in the first frame. It shows that a task-level correctness signal alone is sufficient to shape a nontrivial exploration policy, which is a cheaper and more scalable supervision recipe than collecting exploration demonstrations.

Real-world applications:

• Service and mobile robots that must look around a room, check behind furniture, or reposition to answer a user's question before acting.
• AR/VR assistants that guide a user's viewpoint or autonomously select informative views for spatial instructions.
• Accessibility tools that actively explore a scene to describe occluded or off-screen content to a blind or low-vision user.
• Inspection and monitoring (warehouse, construction, infrastructure) where a drone or camera platform must decide where to look next rather than capture everything.

Industry relevance: the recipe uses an off-the-shelf open VLM, standard GRPO, and rendered observations, all of which map onto existing embodied-AI and robotics stacks. The finding that consistent action-observation pairing matters more than image fidelity (10.6% average gain for rendering over retrieval) is a practical design constraint for anyone building simulated training environments.

Future Directions

• From abstract camera control to real embodiment. The paper deliberately abstracts away the body and lets the policy move a camera through a scanned scene, including passing through geometry. Extending this to physical robots with collision, dynamics, and noisy actuation is untested here.
• Reconciling visual fidelity with action-observation consistency. Rendering wins over retrieved real images, but the authors note rendered views have limited fidelity; closing that gap without breaking pose alignment is an open engineering question.
• Better credit assignment over longer horizons. The reward is trajectory-level and the budget is twelve steps; whether the same outcome-only signal scales to long-horizon tasks requiring dozens of interactions is not established.
• Richer action and tool spaces. The paper positions itself against viewpoint-selection and tool-use methods such as Chain-of-View, Think3D, and ViewSuite; combining learned exploration with reconstruction or measurement tools is a natural extension.

Target Audience

Researchers and engineers working on 3D vision-language models, embodied AI, and multimodal reinforcement learning, particularly those interested in agentic VLMs that act rather than only describe. It is also useful for practitioners building simulated environments for robot learning, since the ablation on rendering versus retrieval and on action parameterization has direct design implications. Readers without background in RL or POMDP formalism will find the method sections dense, though the qualitative results and the exploration-demand analysis are accessible.

Authors’ abstract

We study how vision-language models (VLMs) can learn to think and actively explore in 3D space. Instead of providing multi-view images, videos, or 3D representations to the model, we preserve the native visual encoding of VLMs, i.e., a single image, one at a time, just like humans, and treat the 3D representation only as an environment with which the model can interact through actions (e.g., move left) and observations (rendered images). Each rollout begins from an initial first-person view and a visual question. The model can then iteratively reason, navigate, and reorient its viewpoint before committing to a final answer. Following an initial trajectory-supervised finetuning, we apply reinforcement learning (RL) with rewards driven primarily by final-answer correctness. Crucially, RL provides no direct supervision over which viewpoints the model should visit or which exploration actions it should take. Instead, the model must learn an effective exploration policy solely from the extent to which its interactions improve performance. Across six 3D benchmarks, our model ACT3D achieves an average accuracy of 53.2% that improves its base Qwen model by 22.0% and surpasses proprietary Gemini 3.6 Flash. ACT3D also outperforms existing open-source 3D methods while using 50-70% fewer input frames, and generalizes to unseen domains with an average improvement of 5.9% over its supervised initialization. Further analysis reveals adaptive, question-conditioned exploration. Performance gains over the base model increase from 0.6% on questions answerable from the initial view to 16.3% on those requiring extensive exploration. ACT3D searches for occluded targets, avoids unnecessary exploration, and acquires complementary viewpoints for spatial comparisons. These findings demonstrate that outcome-based RL enables VLMs not only to reason over visual evidence, but also to learn when and how to acquire it.