P$^3$-VLM: A Point-based Alternative for Grounded 3D Vision-Language Models

AI summary

Overview

Research area: 3D vision-language models (3D VLMs) — specifically grounded 3D scene understanding, where a model must describe or identify a specific object in a 3D scene given a spatial query.

Technical level: Intermediate to Advanced. The paper assumes familiarity with vision-language model architectures, 3D Gaussian Splatting, LoRA fine-tuning, and cross-attention, though the central argument is stated in accessible terms.

Scope: The paper diagnoses a "semantic leakage" shortcut caused by bounding-box grounding prompts in 3D VLMs, proposes a point-based prompting protocol instead, and introduces P³-VLM — a detector-free, scene-centric, tri-pathway 3D VLM built on 3D Gaussian Splat representations — evaluated on Nr3D, ScanRefer, ScanQA, SQA3D, and an out-of-distribution ScanNet++ benchmark.

What This Paper Is About

Current 3D VLMs are almost universally prompted with 3D bounding boxes when asked about a specific object. Because box dimensions, aspect ratio, and volume correlate strongly with object category (a tall narrow box is usually a door; a queen-bed-sized box is a bed), models can guess the answer from box geometry without looking at the actual visual content. The paper quantifies this leakage, replaces box prompts with single 3D point prompts, and builds a new model architecture that works well under the harder point-based protocol.

Key Contributions

1. Identification of bounding-box semantic leakage. The authors show that box geometry acts as a dominant categorical signal in both the textual prompt channel and the architectural channel, and that it lets models neglect visual information entirely.

2. Point-based grounding prompting. A new evaluation and prompting protocol where the query is a single 3D coordinate (x, y, z), which removes information about object extent, shape, and aspect ratio.

3. P³-VLM architecture. A detector-free, scene-centric 3D VLM operating on 3D Gaussian Splats, with three complementary pathways — location-aware, task-aware, and context-aware — producing visual tokens for a Qwen3-VL-4B language backbone.

4. Empirical demonstration that the harder protocol yields better models. P³-VLM trained on points achieves higher class accuracy than box-prompted baselines on grounded captioning, and also improves robustness, out-of-distribution generalization, and ungrounded VQA performance.

Main Findings

• A blind LLM gets 36.4% on Nr3D from box coordinates alone. A LoRA-fine-tuned Qwen3-VL 4B Instruct LLM with no visual encoder of any kind reaches 36.4% class accuracy on Nr3D when given only box center, yaw, and extent.

• Object-centric VLMs barely use their 3D encoder. Replacing LEO's visual features with a 2-layer MLP encoding of box center and extent gives 44.2% on Nr3D, versus 45.1% for the full OD-VLM — a gap of under one point.

• Point prompts cut the shortcut roughly in half. The same blind LLM drops from 36.4% to 18.0% on Nr3D when trained and evaluated on point prompts (a ~50% relative reduction), with no architecture change. On ScanRefer it drops from 38.1% to 18.4%.

• Existing 3D VLMs degrade under point prompting even after fine-tuning. On Nr3D, LL3DA falls from 47.7% (box) to 33.1% (point) and GaussianVLM from 46.2% to 29.3%. On ScanRefer, LL3DA falls from 48.02% to 29.2% and GaussianVLM from 50.4% to 33.2%. The authors read this as a limit of visual capacity, not prompt formatting.

• P³-VLM leads on grounded captioning. On Nr3D class accuracy: P³-VLM (point) 52.1%, P³-VLM (box) 52.4%, versus LL3DA 47.7%, GaussianVLM 46.2%, and a language-matched GaussianVLM^Q 47.1%. On ScanRefer: P³-VLM (point) 56.1%, P³-VLM (box) 58.4%, versus GaussianVLM^Q 51.7% and GaussianVLM 50.4%. Caption similarity stays comparable (P³-VLM (p) 46.0 Sim on Nr3D, 54.2 on ScanRefer), while class accuracy rises.

• Residual point priors are small and concentrated. A geometry-only kNN classifier on Nr3D (38 classes with at least 8 instances) reaches 8.1% from xy position alone, 17.8% from height alone, 16.5% from full xyz, and 33.0% from the full bounding box. Height accounts for nearly all residual signal; the center-based kNN hits at least 30% recall on only 3/38 classes (chair, picture, monitor) versus 15/38 for the box-based one.

• Box-prompted models are fragile to geometric disruption. On Nr3D, in-distribution versus non-tight boxes versus noisy reconstructions: LL3DA 47.7 / 28.6 / 33.4; GaussianVLM 46.2 / 25.3 / 32.2; P³-VLM (box) 52.4 / 36.4 / 39.9.

• Point prompting degrades gracefully under query perturbation. Trained on object centers, P³-VLM (point) falls only 5.5 accuracy points on Nr3D and 3.3 on ScanRefer at the most extreme offset (0.9 × box extent), reaching 46.6% and 52.8%. Sampling query points uniformly from the object surface gives 50.7% on Nr3D and 56.9% on ScanRefer.

• Ungrounded scene understanding also improves. ScanQA EM-R: P³-VLM (point) 31.9 versus GaussianVLM 30.7, GaussianVLM^Q 30.9, and LL3DA 30.3 (though on METEOR, GaussianVLM's 22.9 edges out P³-VLM (point)'s 22.6). SQA3D EM-R: P³-VLM (point) 51.8, P³-VLM (box) 51.9, GaussianVLM^Q 51.9, GaussianVLM 51.4; P³-VLM (point) has the highest BLEU-4 at 30.6.

• Out-of-distribution generalization improves substantially. On the GaussianVLM-OOD counting benchmark over unseen ScanNet++ scenes (exact match): P³-VLM (point) 17.31, P³-VLM (box) 15.42, GaussianVLM 9.3, LEO 7.9, LL3DA 1.5. The +1.89 EM advantage of point over box training supports the shortcut-reduction hypothesis.

• Every pathway matters, and the design beats a matched baseline. Removing the location, task, or context pathway drops Nr3D accuracy to 49.3, 50.2, and 50.2 respectively versus 52.1 for the full model. A matched baseline sharing the same 3D encoder and LLM but using only nearest-voxel features plus context tokens reaches 39.8% on Nr3D and 44.2% on ScanRefer, versus 52.1% and 56.1%.

• n-gram metrics hide failures. On Nr3D, LL3DA scores 48.1 Sim but only 5.8 METEOR and 9.9 ROUGE, while the blind LLM reaches 36.4% class accuracy despite no visual input — evidence that surface-overlap metrics overstate spatial understanding.

Methodology in Plain English

The authors start with a diagnostic experiment rather than a model. They take a standard language model and fine-tune it to guess an object's class from nothing but the coordinates, size, and orientation of its bounding box. If that model does well, box prompts must be carrying category information on their own. It does — 36.4% on Nr3D — and a parallel experiment swapping a real 3D encoder for a tiny box encoder inside an existing model (LEO) reproduces the same effect.

They then change the prompt format. Instead of a box, the model receives a single point in space. The same diagnostic models are retrained under this format and their accuracy collapses, confirming the box was doing the heavy lifting.

To address the harder setting, the authors build P³-VLM. A 3D Gaussian Splat scene is voxelized by a sparse 3D encoder (initialized from SceneSplat and kept unfrozen), producing features anchored to specific spatial locations. These dense features (potentially over 10k tokens) are downsampled to at most 2048 tokens via average pooling, then passed through self-attention layers where both queries and keys carry 3D rotary position embeddings, so attention scores encode relative 3D distance directly.

Three pathways then read from these features. The location pathway takes the single voxel nearest the query point, concatenates its feature with the normalized query coordinate, and pushes it through a 4-layer MLP into one LLM-space token. The context pathway uses 8 learnable query tokens that cross-attend over the scene through two cross-attention layers to produce a prompt-independent scene summary. The task pathway refines the language query through two self-attention layers, adds 3D RoPE to the voxel tokens, and lets the voxels cross-attend to the refined text — with a zero-initialized residual output projection for stable training — before compressing the result into 128 learnable tokens. All tokens go to a Qwen3-VL-4B LLM fine-tuned with LoRA at rank 16.

Training runs in two stages: first only the location pathway is active (5 epochs on grounded captioning), so the voxel-to-language mapping stabilizes; then all three pathways train jointly on the full task mixture (12 epochs). Total training is 10 hours on 2 NVIDIA H200 GPUs with batch size 16, AdamW, learning rate 3e-5, weight decay 0.05, and cosine annealing. Two variants are trained, differing only in prompt format: box [center, yaw, extent] and point (object centers from ground-truth segmentation). Because baselines do not report object class accuracy, the authors extract predicted classes from all models' captions with Gemini 2.5 Flash as an LLM-based evaluator, with human quality control on 5% of validation predictions finding no extraction errors.

Why This Matters

Impact on research. The paper argues that a widely used evaluation convention — box prompts — has been inflating apparent progress in 3D grounding, and it supplies a stricter protocol plus a diagnostic tool (the blind LLM) that any future benchmark can apply. It also shifts attention from object-centric, detector-dependent pipelines toward detector-free, scene-centric ones, and shows that removing a shortcut improves performance on tasks that were never about boxes.

Real-world applications:

• Robotics and navigation, where a user or planner indicates a target by pointing at a location rather than supplying an accurate 3D bounding box, and where detector outputs are unreliable.
• Visual assistance systems that describe a scene region a user indicates in a headset or on a phone screen.
• AR/VR and digital-twin interfaces built on Gaussian Splat reconstructions, where a tap in space is the natural input.
• Scene editing and inspection pipelines that need to identify what is at a given coordinate in a noisy or partially reconstructed environment.

Industry relevance. Bounding-box-free interaction lowers the annotation and detector-accuracy burden for deployment: a point is trivially obtainable from a click, a gaze ray, or a depth sensor, whereas a tight box requires a reliable detector. The paper's robustness results (non-tight boxes and noisy reconstructions costing box-based models 14–21 accuracy points on Nr3D) speak directly to the gap between benchmark conditions and deployed ones. The use of 3D Gaussian Splats also aligns with a representation already widespread in industrial reconstruction pipelines.

Future Directions

• Reducing the reconstruction dependency. P³-VLM requires 3D Gaussian Splat input, adding a reconstruction step; the authors note this has been done at scale but do not test alternative representations.
• Generalizing beyond the SceneSplat encoder. The method relies on voxelized features from a specific 3D encoder, and how well the tri-pathway design transfers to other encoders is explicitly left open.
• Extending to language-intensive tasks. Evaluation is concentrated on perception-oriented benchmarks; dialogue-driven and complex multimodal reasoning settings are named as future work.
• Applying the leakage diagnostic more broadly. The blind-LLM probe and point-based protocol could be extended to output-side grounding, scene-level spatial reasoning benchmarks, and other tasks where a prompt may encode the answer — the authors frame the work as a step toward drawing attention to information leakage in 3D grounding generally.

Target Audience

Researchers and engineers working on 3D vision-language models, embodied AI, and 3D scene understanding who design or evaluate grounding benchmarks; practitioners building point-and-ask interfaces on Gaussian Splat or other 3D reconstructions; and anyone evaluating whether a benchmark number reflects genuine visual reasoning or exploitation of a prompt artifact. Readers without a vision-language background will still follow the leakage argument and results tables, but the architecture section assumes comfort with attention mechanisms and LoRA.

Authors’ abstract

Examining existing benchmarks for grounding in 3D VLMs reveals a pervasive "semantic leakage" in the evaluation and prompting protocols: a heavy reliance on bounding boxes as spatial prompts or representation primitives. Because bounding boxes are highly informative of object categories, they introduce a shortcut for the models to neglect visual information during 3D VLM training. We demonstrate this with a "blind" LLM that, prompted only with the box coordinates, can perform surprisingly well. To further analyze this shortcut, we propose point-based grounding prompting - querying models with a single 3D point instead of a box. Under this protocol, the accuracy of the blind LLM drops from 36.4% to 18.0% on ScanNet's Nr3D, and similarly degrades for 3D VLMs (e.g. LL3DA). Building on this insight, we introduce P$^3$-VLM, a novel tri-pathway 3D VLM that supports point-based prompts and achieves 52.1% state-of-the-art accuracy on location-grounded captioning under this setting. To achieve this, we redesign the architecture while keeping it detector-free, scene-centric, and suitable for 3D GS inputs. P$^3$-VLM utilizes three complementary pathways (location-, task-, and context-aware), enabling strong reasoning about scenes without any bounding box dependencies. The key benefit of point-based learning is a consistent improvement across settings: P$^3$-VLM achieves state-of-the-art performance under point-based grounding while also improving robustness, OOD generalization, and performance on ungrounded scene-centric 3D VQA benchmarks, reflecting its reduced reliance on bounding-box shortcuts and resulting stronger visual-spatial reasoning.