Reasoning Matters for 3D Visual Grounding

AI summary

Overview

Research area: 3D computer vision — specifically 3D visual grounding — combined with large language model (LLM) fine-tuning, synthetic data generation, and chain-of-thought reasoning.

Technical level: Intermediate. The paper assumes familiarity with transformer-based vision-language models, 3D point clouds, bounding-box IoU metrics, and LLM supervised fine-tuning, but its core idea (generate data automatically, teach the model to reason step by step) is explained in accessible terms.

Scope: The paper proposes a fully automatic pipeline that synthesizes 3D scenes, referring queries, and structured reasoning traces, then uses that data to fine-tune an open-source LLM (Reason3DVG-8B) for 3D visual grounding, evaluated on ScanRefer and NR3D.

What This Paper Is About

3D visual grounding asks a model to identify which object in a 3D scene a natural-language query refers to (for example, "the chair closest to the table"). Most existing methods need large amounts of human-annotated 3D data, and a prior LLM-based approach, 3D-GRAND, collected a million-scale dataset yet only achieved moderate improvement that was not proportional to the collection cost. This paper's goal is to test whether a cheap, human-free data pipeline that also supervises the model's reasoning process can beat data scale alone.

Key Contributions

1. A fully automatic 3D visual grounding data pipeline that requires no human annotation, generating 3D scene layouts, referring queries, and corresponding reasoning responses, which substantially lowers data collection cost relative to prior work.
2. Reason3DVG-8B, an LLM fine-tuned on this data for 3D visual grounding, evaluated on the ScanRefer and NR3D benchmarks.
3. A demonstration that structured reasoning supervision, not data scale, drives the gains: the model outperforms 3D-GRAND while using only 1.6% of its training data, where 3D-GRAND was trained on 60× more data.
4. A four-stage structured chain-of-thought format (Related Object Selection, Situation Estimation, Reasoning, Conclusion) that makes the LLM's grounding process interpretable and parseable.

Main Findings

• ScanRefer performance at 1.6% of the data: Reason3DVG-8B reaches Overall Acc@0.25 of 38.7 and Acc@0.5 of 34.4, compared with 3D-GRAND's 38.0 and 27.4. On the "Unique" subset it reaches 76.6 (Acc@0.25) and 69.5 (Acc@0.5), versus 3D-GRAND's 54.4 and 36.4. On "Multiple" it reaches 31.1 and 27.3, versus 3D-GRAND's 26.0 and 20.8. The Figure 1 caption describes this as 25% better grounding accuracy while using only 1.6% of the training data.
• NR3D performance: Without oracle labels, Reason3DVG-8B scores 50.4 Easy, 31.0 Hard, 37.3 View-Dependent, 42.0 View-Independent, and 40.4 Overall, outperforming the zero-shot methods ZSVG3D (39.0 Overall) and SeeGround with InternVL2-8B (34.3) and InternVL2-26B (38.0).
• Fine-tuning helps across every split: On NR3D, fine-tuning raises Easy from 44.6 to 63.0 (+18.4), Hard from 22.7 to 36.7 (+14.0), Dep. from 31.1 to 40.1 (+9.0), Indep. from 34.5 to 54.5 (+20.0), and Overall from 33.3 to 49.3 (+16.0), which the paper describes as an overall accuracy increase of 16%.
• Reasoning supervision is the key ingredient: Training on the same data without the reasoning process yields 42.2 Easy, 25.4 Hard, 30.2 Dep., 35.4 Indep., 33.5 Overall; adding reasoning supervision yields 63.0 (+20.8), 36.7 (+11.3), 40.1 (+9.9), 54.5 (+19.1), 49.3 (+15.8).
• Generalization to unseen queries: Trained only on seven common in-domain spatial relationships, the model still improves on out-of-domain queries by 17.0 points (from 32.1 to 49.1) and on in-domain queries by 15.1 points (from 34.5 to 49.6).
• Small subsets can suffice: Training on a subset of the collected data achieves comparable performance when fine-tuned for more iterations, supporting the claim that Llama-3 already possesses basic reasoning ability and mainly needs the step-by-step format.
• Detection quality is the ceiling: When the oracle object class label is supplied on NR3D, accuracy rises to 63.0 Easy, 36.7 Hard, 40.1 Dep., 54.5 Indep., and 49.3 Overall, bringing the model close to fine-tuned state-of-the-art methods and indicating that object detector quality limits results.

Methodology in Plain English

The team built a program that writes its own training data. First, it creates an empty 3D space and places objects inside it according to one of seven spatial relationships (Closest, Farthest, Next to, Left, Right, Largest, Smallest), used as templates to determine which object a query should point to and to generate extra distractor objects. GPT-4o was prompted to supply 40 common indoor object classes and their typical dimensions in meters, and random size variation was applied for diversity. Each sample contains a scene with over 50 objects, a query, the target's object ID, and a GPT-4o-produced reasoning answer.

Responses were filtered automatically: because the layouts are rule-generated, the ground truth for each query is guaranteed, so any GPT-4o response whose final answer was wrong or badly formatted was discarded. About 10% were removed this way, leaving 3.2K samples (3,167 points) from 3,500 generated scenes (700 per relationship, with per-relationship counts of 460 Closest, 448 Farthest, 458 Next to, 430 Left, 411 Right, 479 Largest, and 481 Smallest). Manual checks on a subset found only minor arithmetic errors (typically after the first decimal place) that did not change the final predictions.

The base model, Llama-3.1-8B, was then fine-tuned on this data using standard next-token prediction with cross-entropy loss, so it learns to emit the four-stage reasoning trace before the final answer. At test time, the Mask3D detector turns a point cloud into object proposals with class labels, these are converted to structured text with unique IDs, and the LLM receives them along with the query and predicts the target object's ID. Training used FSDP, a learning rate of 1×10⁻⁵, 2 epochs, batch size 8, context length 4096, and mixed precision.

Why This Matters

Impact on research. The results challenge the assumption that 3D visual grounding LLMs need huge annotated datasets. A 3.2K-sample synthetic set with reasoning supervision beats a 200K-QA-pair training run in the multi-object setting, and the ablation isolating reasoning supervision provides direct evidence that the form of the supervision matters as much as its volume. It also shows that simple synthetic spatial relationships transfer to complex, unseen real-world queries.

Real-world applications (potential, based on the task studied):

• Robots or assistive agents that follow instructions like "pick up the mug to the left of the laptop" in a scanned room.
• Augmented and virtual reality interfaces where users refer to physical objects by description rather than by pointing.
• Warehouse, home, or retail inventory systems that locate a described item in a 3D scan.
• Annotation cost reduction: teams can generate supervision for new spatial-relationship categories programmatically instead of hiring 3D annotators.

Industry relevance. Because the pipeline is human-free and the fine-tuned model runs without proprietary in-context examples at inference (unlike ZSVG3D, which needs multiple in-context examples and proprietary GPT-4), the approach reduces both data-collection cost and test-time inference cost, which matters for deploying grounding models at scale.

Future Directions

• Better perception upstream: The paper states that accuracy is primarily limited by the quality of object proposals from the 3D detector, and points to better detectors and object captioners with richer semantic information as the most direct improvement path.
• Adding object visual features: The authors note that ZSVG3D and SeeGround use extra visual input and suggest incorporating object visual features could further improve their method.
• Scaling and diversifying the data: The current data covers only seven common spatial relationships; whether more complex, composed relationships can be synthesized and whether they add value beyond the current design is left open.
• Richer supervision for the LLM: The qualitative and quantitative results suggest that supplying more semantic clues supports the reasoning stage, raising the question of how reasoning supervision should be structured as base LLMs improve.

Target Audience

Researchers and engineers working on 3D scene understanding, embodied AI, and vision-language models who are deciding between scaling annotated data and improving supervision quality. It is also useful for practitioners building data-generation pipelines for multimodal LLMs, and for readers interested in how chain-of-thought supervision transfers from language tasks to spatial reasoning tasks.

Authors’ abstract

The recent development of Large Language Models (LLMs) with strong reasoning ability has driven research in various domains such as mathematics, coding, and scientific discovery. Meanwhile, 3D visual grounding, as a fundamental task in 3D understanding, still remains challenging due to the limited reasoning ability of recent 3D visual grounding models. Most of the current methods incorporate a text encoder and visual feature encoder to generate cross-modal fuse features and predict the referring object. These models often require supervised training on extensive 3D annotation data. On the other hand, recent research also focus on scaling synthetic data to train stronger 3D visual grounding LLM, however, the performance gain remains limited and non-proportional to the data collection cost. In this work, we propose a 3D visual grounding data pipeline, which is capable of automatically synthesizing 3D visual grounding data along with corresponding reasoning process. Additionally, we leverage the generated data for LLM fine-tuning and introduce Reason3DVG-8B, a strong 3D visual grounding LLM that outperforms previous LLM-based method 3D-GRAND using only 1.6% of their training data, demonstrating the effectiveness of our data and the importance of reasoning in 3D visual grounding.