LLaVA$^3$: Representing 3D Scenes like a Cubist Painter to Boost 3D Scene Understanding of VLMs

AI summary

Overview

• Research area: 3D scene understanding with vision-language models (VLMs), specifically 3D visual question answering (3D VQA), 3D language grounding, and 3D semantic segmentation from multi-view images.
• Technical level: Advanced. The method combines grid-based NeRF reconstruction, multi-view feature-field distillation, hierarchical segmentation (SAM/CLIP), and frozen LLaVA-OV inference.
• Scope: The paper introduces LLaVA³ (pronounced "LLaVA Cube"), an inference-time method that re-represents a 3D scene as per-object omnidirectional visual descriptions so a frozen VLM can reason about 3D content without any fine-tuning.

What This Paper Is About

Building a multimodal model that understands 3D scenes is hard because 3D training data is scarce compared to the abundant 2D data used to train vision-language models. LLaVA³ sidesteps this by leaving the VLM untouched: it first reconstructs the scene in 3D from multi-view images, then rewrites each object in the scene as a single visual description that shows the object from many viewpoints at once, much like a Cubist painting shows several sides of a subject in one picture. The goal is to give an existing 2D VLM a better visual input for 3D tasks rather than retraining it.

Key Contributions

1. Object-centric omnidirectional scene representation. The scene is described to the VLM as a collection of omnidirectional visual descriptions, one per 3D object, instead of a flat set of multi-view images or unstructured sampled tokens.
2. First object-hierarchy decomposition of a 3D neural field scene. HDBScan clustering plus three scale-specific decoders (trained with the LangSplat SAM discretization trick) produce a discrete object/part/sub-part tree, rather than Garfield's continuous volume-based decomposition.
3. Joint view-independent and view-dependent LLaVA token modeling. Two complementary feature fields are learned, one view-invariant (f_VI) and one view-dependent (f_VD = f_VI + δ_VD(X, d)), so object semantics and view-specific spatial relationships are captured together.
4. State-of-the-art results among VLM-based solutions on 3D VQA and 3D grounding, with no VLM fine-tuning, while also enabling 3D semantic segmentation, open-vocabulary segmentation and instance segmentation through a separate SAM-CLIP field.

Main Findings

• ScanQA validation set: LLaVA³ reaches CIDEr 77.69, METEOR 15.83, Rouge 39.69, EM@1 26.00 and EM@1-R 38.43. This beats the zero-shot hybrid baselines SplatTalk (61.7 CIDEr) and Ray-NeRF (60.56 CIDEr), and the 2D VLM baselines LLaVA-OV (55.38 CIDEr) and an object-based LLaVA baseline (51.78 CIDEr). Fine-tuned SplatTalk reaches 77.5 CIDEr, 15.6 METEOR, 38.5 Rouge, 22.4 EM@1 and 38.3 EM@1-R. LEO, which is trained on ScanQA-style data, reaches 101.4 CIDEr.
• Gap to trained 3D LMMs: against most 3D LMMs, LLaVA³'s metrics range between −4.6% and +13.2% of those baselines, with an average of +2% across all metrics, despite using no 3D-specific training information and no ScanNet training.
• Source of the gain on ScanQA: using a NeRF to spread visual tokens across the scene accounts for +5.18 CIDEr over LLaVA-OV, while adding the object-centric representation accounts for a further +22.31 CIDEr. The 2D object-based baseline scores −3.60 against LLaVA-OV, indicating object decomposition only helps when built on the 3D representation.
• MSR3D test set (overall correctness score): LLaVA³ scores 44.89 against Ray-NeRF 30.54, SplatTalk 41.8, LLaVA-OV 27.87, an object-based LLaVA-OV 25.02, and LEO 7.8.
• Ablation contributions on MSR3D (overall): object-based decomposition +6.85, multi-scale hierarchical sampling +2.5, filtering +0.55, radar sweeping +2.09. Object-based decomposition contributes most (+14.32 on existence questions), multi-scale sampling most helps appearance questions (+3.04 existence, +3.08 attributes), and radar sweeping contributes +2.05 on counting and +3.28 on existence but −0.11 on attributes.
• View dependency matters: using 100% view-independent features gives an overall score of 42.53 and 100% view-dependent gives 42.60, an even 50/50 split gives 43.84, and the adaptive VI/VD distribution used by LLaVA³ gives 44.89. The paper reports that VI features help descriptive queries (counting, existence, attributes) while VD features help spatial and navigation questions.
• 3D grounding: on Sr3D+ LLaVA³ reaches Acc@0.1 14.41 and Acc@0.25 6.54; on Nr3D it reaches 16.17 and 7.81. Baselines are LeRF (6.88/1.97 on Sr3D+, 5.53/1.43 on Nr3D), OpenNeRF (9.07/4.03 and 9.70/5.11) and ConceptGraph (13.3/6.2 and 16.0/7.2). The paper reports gains of +245% and +60% in average over LeRF and OpenNeRF respectively.
• Semantic segmentation: evaluated on Replica and ScanNet scenes, LLaVA³ reports up to +146% mIoU / +91% mAcc over LeRF, +21% / +7% over OpenNeRF, +15% / +14% over DiSCO-3D, +36% mIoU and +20% mAcc over ConceptGraph, and +4% / −3% relative to HOV-SG.
• Auxiliary finding on feature supervision: supervising the LLaVA field with a Relative MSE loss persistently degraded performance and produced severe hallucinations in the VLM, so standard MSE was retained.

Methodology in Plain English

The pipeline has four stages and never updates the language model.

1. Reconstruct the scene in 3D. Using the Nerfacto grid-based NeRF in the Nerfstudio framework, with Mip-NeRF-360 style improvements, the method fits a 3D model of the scene from posed multi-view images. Alongside geometry, it distills LLaVA's post-projector visual tokens into a 3D feature field. Tokens are high-dimensional (3584D), sparse and unnormalized, so an auto-decoder maps a normalized lower-dimensional feature into the full token dimension. Working with a NeRF avoids the memory limits and compression losses that Gaussian-Splatting methods such as SplatTalk accept.
2. Model both semantics and viewpoint. Each 3D point gets a view-invariant feature plus a direction-dependent deviation from it. Both are decoded into full LLaVA tokens and supervised with MSE against the same ground-truth tokens, so the field keeps object identity while retaining cues that only some viewpoints reveal (for example, how two objects relate from one side of the room).
3. Break the scene into objects. A second feature field is trained with Garfield-style contrastive SAM-mask losses, but with three separate decoders for object, part and sub-part levels; CLIP features are also decoded per scale. HDBScan clusters sampled ray embeddings independently at each scale, then heuristics discard noisy clusters, split under-segmented regions using the finer scale, and merge over-segmented ones when CLIP or SAM centroids nearly match. Parents are assigned by co-occurrence statistics, yielding a discrete scene tree.
4. Describe each object to the VLM. Rays are cast through the supervision images and their LLaVA features are assigned to objects, with a budget of N_f = W/O features per object (W is the VLM's maximum context window, O the number of objects) split equally across parts and sub-parts, and equally between view-independent and view-dependent features. For observer-relative questions, view-dependent features are drawn only from rays near each object's canonical viewing direction (the most frequently observed direction, computed by spherical binning). Objects are ordered by a radar-inspired sweep: sorted by polar angle around the vertical axis with radial distance as a tie-breaker. Each object is presented as a virtual image with a unique ID, so the VLM can answer a grounding query by emitting an ID that maps back to a NeRF segment.

Implementation details: ScanNet NeRFs use 80% of images (selected by estimated blurriness, with 20% of the most blurred frames discarded by a Laplacian variance filter) for geometry and 400 selected images for the two feature fields, with greedy frame selection to maximize 3D camera coverage. All experiments ran on one A100 GPU. The LLaVA-OV backbone couples a SigLIP encoder to an LLM via an MLP projector.

Why This Matters

The work shows that a substantial part of the 3D gap for VLMs is a representation problem, not a training-data problem: changing how the scene is presented to a frozen model produced large gains. It also argues against unstructured token sampling from neural fields and against mixing view-dependent information away through naive multi-view averaging.

• Robotics: the paper motivates 3D scene understanding by applications in robotics, where a robot must reason about objects and their spatial relations.
• Autonomous navigation: the paper cites autonomous navigation as a driving application area.
• 3D VQA and 3D language grounding: both are directly evaluated, and grounding is performed by predicting an object instance ID that is mapped back to a segmented point cloud.
• 3D semantic, open-vocabulary and instance segmentation: supported hierarchically through the SAM-CLIP field, with the hierarchy also enabling sub-scale grounding.

Industry relevance: the method requires no VLM fine-tuning, keeps the two feature fields independent so segmentation tasks can run without the language model (reducing memory footprint and training complexity), and avoids the per-scene checkpoints larger than 100 Gb that a mask-level LLaVA supervision scheme would imply. Those properties matter for deployment where 3D data collection and retraining are expensive.

Future Directions

• Entangling the two feature fields. The supplementary material explicitly discusses interlacing LLaVA and segmentation training, for example by computing per-mask LLaVA features, and explains why the authors rejected it: over 100 SAM masks per image would imply feature maps of size 400 × 100 × 27 × 27 × 3584 (≈ 105×10⁹) and more than 100 Gb per-scene checkpoints, plus loss of surrounding spatial context. Finding a tractable version of this idea is an open direction.
• Closing the gap with trained 3D LMMs. LEO still leads on ScanQA (101.4 CIDEr versus 77.69), and the paper notes that 3D LMMs train on ScanNet and other indoor scenes. Whether LLaVA³-style representations can match trained models without that data, or generalize beyond indoor scans, is untested here.
• Scene ordering and spatial reasoning. The radar-inspired ordering yields a −0.11 change on MSR3D attribute questions while helping other categories, suggesting ordering schemes that encode layout more faithfully remain an open design question.
• Extending hierarchical decomposition. The paper does not report a formal future-work section, so further questions such as generalizing the object hierarchy beyond the object/part/sub-part levels, or evaluating the sub-scale grounding capability more broadly, are left open.

Target Audience

Researchers and practitioners working on 3D vision-language models, neural-field feature distillation, and embodied or robotic perception, plus engineers who need 3D scene understanding from multi-view image captures without the cost of collecting 3D training data or fine-tuning a large model. Readers should already be comfortable with NeRF-style reconstruction, CLIP/SAM-style features, and transformer-based VLMs.

Authors’ abstract

Developing a multi-modal language model capable of understanding 3D scenes remains challenging due to the limited availability of 3D training data, in contrast to the abundance of 2D datasets used for vision-language models (VLM). As an alternative, we introduce LLaVA$^3$ (pronounced LLaVA-Cube), a novel method that improves the 3D scene understanding capabilities of VLM using only multi-view 2D images and without any fine-tuning. Inspired by Cubist painters, who represented multiple viewpoints of a 3D object within a single picture, we propose to describe the 3D scene for the VLM through omnidirectional visual representations of each object. These representations are derived from an intermediate multi-view 3D reconstruction of the scene. Extensive experiments on 3D VQA and 3D language grounding show that our approach outperforms previous 2D-based VLM solutions.