VLSplat: Vision-Language Guided Object-Centric 3D Gaussian Splatting via Scene Graph

AI summary

Overview

Research area: Computer Vision — semantic 3D scene understanding with 3D Gaussian Splatting (3DGS), combining vision-language models (CLIP, SAM2, SAM3, a VLM) with scene-graph reasoning.

Technical level: Advanced. The paper assumes familiarity with 3DGS optimization, codebook/vector-quantization style feature grouping, CLIP similarity objectives, and differentiable rendering.

Scope: A framework (VLSplat) that lifts 2D masks into object-level semantic Gaussian groups and then refines those groups using vision-language priors organized as a 3D scene graph, evaluated on render-level and point-level indoor segmentation benchmarks.

What This Paper Is About

Most methods that give 3D Gaussian Splatting scenes semantic meaning simply copy 2D segmentation masks onto 3D points. Because those masks come from tools like SAM that are inconsistent across viewpoints, the same object can get split into pieces or merged with a neighbor, and the resulting 3D groups follow local mask evidence rather than whole-object structure. VLSplat's goal is to push this "mask-based lifting" from category-agnostic instance clustering toward category-aware object representation, by attaching object-level language priors to Gaussian groups and using a scene graph to clean up and complete them.

Key Contributions

1. A vision-language guided semantic lifting framework that extends codebook-based Gaussian grouping (built on Unified-Lift) from category-agnostic instance clustering to object-centric representation learning.
2. An object-level scene graph in which each codebook group becomes a node enriched with a semantic category, a VLM-generated object description, a codebook prototype, a tight 3D ROI, an expanded ROI, and SAM3-based target-object evidence; edges encode spatial adjacency between expanded ROIs.
3. Language-guided Gaussian gating on synthetic object views, enabling two refinement operations — intra-node refinement (suppressing noisy primitives inside an object node) and inter-node acquisition (recovering missing object parts from neighboring non-target/background regions) — without changing Gaussian geometry.
4. Hybrid 2D supervision that combines text-prompted SAM3 masks for major indoor categories with SAM2 automatic masks to fill uncovered regions, giving semantic object cues plus dense coverage.

Main Findings

• Render-level segmentation gains on Thing categories: on ScanNet++, VLSplat reaches 70.31 Thing mAcc and 69.81 Thing mIoU versus 65.69 / 61.32 for LangSplatV2, 19.09 / 18.33 for OpenGaussian, and 64.77 / 64.42 for "Unified-Lift + Ours." On Replica it reaches 68.69 Thing mAcc and 68.59 Thing mIoU versus 66.88 / 59.64 for LangSplatV2 and 12.89 / 12.03 for OpenGaussian; LegGaussian is marked out-of-memory on Replica.
• Stuff categories improve less: VLSplat gets 74.09 Stuff mAcc and 67.79 Stuff mIoU on ScanNet++ and 68.11 / 67.75 on Replica, which the authors attribute to the method deliberately targeting foreground object nodes rather than large amorphous layout regions. On ScanNet++ "Unified-Lift + Ours" slightly exceeds VLSplat on Stuff mAcc (74.99 vs. 74.09).
• Point-level results are mixed but favorable in object-centric terms: on ScanNetV2 VLSplat scores 40.49 mIoU / 59.45 mAcc full-scene, below PanoGS (54.50 / 74.98), but within scene-graph ROIs ("Ours †") it reaches 61.89 mIoU / 79.12 mAcc, 46.18 PRQ(T) and 27.80 PRQ(S). On Replica VLSplat beats PanoGS full-scene on mIoU and mAcc (64.03 / 74.34 vs. 56.91 / 68.27), and reaches 72.20 / 80.37 with ROI restriction.
• Qualitative object-level advantage: the authors report that PanoGS tends to merge objects on a desk into the desk and label chair/desk legs as floor, whereas VLSplat keeps them as separate, complete segments.
• Scene-graph construction ablation (ScanNetV2, mIoU / mAcc): no ROI 40.49 / 59.45; Euclidean distance 42.70 / 63.64; anisotropic covariance 43.88 / 64.91; feature similarity 58.51 / 76.11; feature similarity combined with anisotropic connectivity 61.89 / 79.12. Feature similarity is the largest single factor.
• Refinement ablation (ScanNet++, Thing, mAcc / mIoU): baseline 61.58 / 61.25; inter-node acquisition only 69.85 / 69.45; intra-node refinement only 69.74 / 69.31; both 70.31 / 69.81. Each module alone helps substantially, and combining them gives the best result with a modest additional gain.
• Training configuration: Unified-Lift is trained for 15k iterations, then from 15k to 30k iterations the scene graph is updated every 500 iterations, with gating optimized for 40 iterations per update using mini-batches of four rendered views and the CLIP ViT-L model, all on a single NVIDIA RTX 3090 GPU (24GB).

Methodology in Plain English

Step 1 — Group the points. Each Gaussian carries a learnable feature vector. A shared "codebook" of K object prototypes is learned alongside, and every Gaussian is assigned to whichever prototype its feature is most similar to (highest dot product). Supervision comes from 2D masks: SAM3 masks generated from category text prompts for indoor classes (wall, floor, ceiling, bed, cabinet, chair, curtain, desk, door, monitor, picture, sofa, window), with SAM2 automatic masks filling regions SAM3 did not cover. SAM2 masks smaller than 200 pixels or overlapping SAM3 by more than 0.3 are discarded; SAM3 masks below 500 pixels are removed and duplicates across related classes (e.g., desk/table) are suppressed.

Step 2 — Build a scene graph. Each codebook group becomes a node. Only high-confidence Gaussians (above a codebook threshold) are kept, and spatial coherence is enforced with a geometry-aware edge test based on center distance relative to each Gaussian's anisotropic covariance along the connecting direction. Isolated components are dropped; the rest are boxed into a tight 3D ROI and an expanded ROI. The group is projected into training views to compare against the SAM3 semantic maps, and a VLM writes an object-specific description (shape, parts, material, context) used as the text prompt for CLIP. Two nodes are connected if their expanded ROIs overlap in 3D volume.

Step 3 — Refine with rendered "object portraits." For each target node, the method renders the object from synthetic camera viewpoints placed around it, in a normalized local frame. One image uses only the reliable high-confidence core; a second image adds candidate Gaussians, each with a learnable gate in [0,1] that controls whether it appears. Gates are optimized by CLIP similarity, but the objective is the improvement of the gated image over the base image, not the absolute score — so gates learn which additions actually add semantic support. High-gate candidates then have their feature direction pulled toward the target node's prototype, low-gate candidates pushed away, with feature magnitude preserved and geometry untouched.

Step 4 — Two refinement passes. Intra-node refinement handles Gaussians already assigned to the node but with weak confidence (suppress or reinforce). Inter-node acquisition looks at all Gaussians inside the node's ROI that belong to other or background groups (excluding the target and its neighboring foreground nodes) to recover missing parts like chair legs.

Evaluation. Two protocols: (a) render-level, using standard 3DGS optimization with densification/pruning enabled, scored with mAcc and mIoU on 10 ScanNet++ scenes and 8 Replica scenes; and (b) point-level, freezing the primitives and disabling pruning, densification and position updates, scored with mIoU, mAcc, and panoptic quality PRQ(T)/PRQ(S) at IoU 0.25 on 7 ScanNetV2 scenes and 8 Replica scenes.

Why This Matters

The paper argues that semantics in 3DGS should not just be pasted-on CLIP features for retrieval, but an object-level structure that is itself optimized. Treating codebook groups as graph nodes turns the scene graph from a passive output abstraction into an optimization scaffold — a shift that could make 3D semantic maps more structurally faithful rather than merely locally mask-consistent.

Real-world applications the paper points to:

• Robotics — object-level 3D understanding for manipulation and navigation.
• AR/VR and scene editing — selecting, moving, or restyling discrete objects inside a reconstructed scene.
• Semantic mesh extraction and semantic mapping — producing labeled 3D geometry for built environments.
• Progress monitoring in built environments — the broader-impact section names object-level assessment of 3D scenes for construction-style monitoring.

Industry relevance: the pipeline depends entirely on off-the-shelf foundation models (SAM2, SAM3, CLIP ViT-L, a VLM) and runs on a single 24GB consumer-grade GPU, making it reproducible and easy to slot into existing 3DGS reconstruction stacks. The explicit caution is that incorrect lifting could cause unreliable object localization in safety-critical uses.

Future Directions

• Handling weakly observed objects. The method still needs a reasonable coarse group to exist from initial lifting; rarely observed or rarely detected objects yield no reliable node, and synthetic rendering gives CLIP too little evidence. Stronger 3D or generative priors (SAM3D, video diffusion) are suggested, along with the open problem of keeping those priors multi-view consistent and aligned to the reconstructed coordinate frame.
• Systematic sensitivity analysis of the foundation models. The authors state outright that dependence on SAM3, the VLM, and CLIP "has not been systematically studied" — prompt choice, VLM description quality, and CLIP backbone are all open variables (the VLM-prompt study appears only qualitatively in the supplementary).
• Beyond indoor thing categories. Evaluation is limited to thing classes (bed, cabinet, chair, desk, monitor, sofa) in indoor scenes; stuff performance is weaker and extension to layout regions or outdoor scenes is untested.
• Reducing refinement cost. The refinement adds offline optimization whose cost grows with the number of target nodes, which matters for scaling to large scenes.

Target Audience

Graduate students and researchers in 3D vision and scene understanding who already know 3DGS and want to see how vision-language priors can be used as an optimization signal rather than just an annotation; robotics and AR/VR engineers building object-level semantic maps from posed RGB; and practitioners evaluating whether scene-graph structure pays off over plain mask lifting, who will find the Unified-Lift vs. "Unified-Lift + Ours" and intra/inter ablations the most directly useful comparisons.

Authors’ abstract

Learning semantic representations for 3D Gaussian Splatting has recently emerged as a promising direction for 3D scene understanding. However, existing 2D-to-3D semantic lifting approaches rely primarily on 2D segmentation supervision and lack object-level priors, causing the learned representation to be driven by local mask evidence rather than holistic object structure. In this paper, we present VLSplat, a vision-language guided approach for object-centric semantic lifting in 3D Gaussian Splatting. Our method augments mask-based lifting with object-level semantic priors derived from vision-language models and refines Gaussian representations using an object-group scene graph. The refinement is performed through two language-guided modules: intra-group refinement suppresses semantic noise within object groups, while inter-group acquisition expands object support regions by incorporating structurally relevant candidates from neighboring groups. Extensive experiments on indoor scene datasets demonstrate that VLSplat improves the semantic consistency and structural completeness of object-level categories. Our code and models are available at https://vlsplat.github.io/.