Lang3D-XL: Language Embedded 3D Gaussians for Large-scale Scenes

AI summary

Overview

Research area: Computer vision / 3D scene understanding — specifically open-vocabulary semantic feature distillation into 3D Gaussian Splatting representations of large-scale, in-the-wild scenes.

Technical level: Advanced. The paper assumes familiarity with 3D Gaussian Splatting, CLIP/DINOv2 feature distillation (LERF, LangSplat, Feature3DGS), multiresolution hash grids (Instant-NGP), and open-vocabulary segmentation evaluation.

Scope: The paper introduces Lang3D-XL, a method that embeds a compact learnable semantic bottleneck into a 3D Gaussian Splatting model so that language features can be distilled efficiently and accurately across massive Internet image collections of landmarks, evaluated on the HolyScenes dataset.

What This Paper Is About

Embedding a language field into a 3D representation lets users query, localize, and explore scenes using natural text. The authors ask whether existing 2D-to-3D feature distillation methods can actually handle large-scale Internet scene collections, and find that they cannot: GPU memory limits block training at scale, and directly supervising rendered features with pixel-aligned 2D feature pyramids introduces semantic misalignments where neighboring regions get contaminated with false semantics. Lang3D-XL addresses both problems with an extremely low-dimensional semantic bottleneck rendered through a multiresolution hash encoder, plus a downsampler and regularizations that bridge the misalignment.

Key Contributions

1. An approach for efficiently augmenting large-scale scenes represented with 3D Gaussians with language features, using a learnable low-dimensional semantic bottleneck that is rendered and decoded to high-dimensional CLIP and DINOv2 features.
2. Novel mechanisms for bridging semantic misalignments in image-level feature distillation pipelines — an Attenuated Downsampler module, a DINO-based regularization, and a SAM-based variance regularization — that boost downstream segmentation performance.
3. A multiresolution hash encoder that operates in 2D rendered feature space (rather than 3D Gaussian space, as in FMGS and Fast-Splat) and hashes with respect to similar features rather than similar locations, reducing 3D memory and rendering cost.
4. Strong performance on challenging Internet scenes with interactive inference times, reported as orders of magnitude faster than prior work targeting large-scale scenes.

Main Findings

• Dominates feature-based baselines on HolyScenes segmentation: Lang3D-XL reaches mAP 0.59, versus FMGS 0.51, LangSplat 0.19, LERF 0.14, DFF 0.11, and Feature3DGS 0.05.
• Comparable to HaLo-NeRF while vastly faster: HaLo-NeRF attains mAP 0.68 but requires roughly two hours of optimization per text prompt, whereas Lang3D-XL reports >0.1 seconds per text prompt. The authors describe the overall mAP as "slightly lower" but note their method wins on specific categories.
• Strong on specialized architectural vocabulary: The authors achieve better performance than HaLo-NeRF on Minarets (0.84 vs. 0.80) and Spires (0.67 vs. 0.61). Per-category scores for Lang3D-XL are Window 0.51, Minaret 0.84, Dome 0.70, Tower 0.46, Spire 0.67, Portal 0.38.
• Generalizes to novel views: On a small held-out validation set, the method achieves mAP 0.60 on novel views versus 0.59 on the standard evaluation views.
• Extremely compact bottleneck suffices: A feature dimensionality as small as d' = 3 is reported as sufficient to produce high-quality output.
• Bottleneck ablation favors the proposed design: Against Feature3DGS's Speedup CNN encoder and FMGS's hash encoder, the proposed bottleneck achieves L1 0.146 vs. 0.149 and 0.180, PSNR 13.9 vs. 13.3 and 11.6, training time 0.10 s vs. 0.21 s and 0.49 s per image, inference time 0.026 s vs. 0.04 s and 0.07 s per image, and training memory ~10GB vs. ~15GB for both alternatives.
• Every distillation component matters: Ablating the Attenuated Downsampler (low resolution) drops mAP from 0.69 to 0.53; high resolution drops it to 0.46; removing DINO regularization drops it to 0.30 — the largest single degradation; removing SAM regularization drops it to 0.60; removing physical scale pyramids drops it to 0.64.
• 2D segmentation baselines are strong but 3D-inconsistent: CLIPSeg_FT is the best 2D method at mAP 0.66, followed by CLIPSeg at 0.56, LangSAM at 0.25, ToB at 0.23, and LSeg at 0.09. The authors attribute CLIPSeg_FT's strength to domain adaptation and note LangSAM produces inconsistent results across viewpoints.
• Feature3DGS fails for a specific reason: Even when given the authors' Semantic Hash Encoder, Feature3DGS yields mAP 0.05, which the authors attribute to 2D features lacking sufficient generalization (LSeg features being insufficiently discriminative).
• FMGS is the closest competitor: As the only other hash-based method, it narrows the gap (0.51 vs. 0.59), but its embedding module operates before rendering, requiring full feature dimensionality to be rendered, leading to longer and more memory-intensive training.
• In-the-wild adaptations are necessary: The pipeline integrates SWAG for appearance/transient-object robustness in the photometric loss, uses an LLM to generate more common prompt synonyms, and replaces standard image-resolution-dependent CLIP pyramid scales with physical scales computed from COLMAP metadata, combined with the HaLo-NeRF finetuned CLIP model adapted for the architectural domain.
• Dataset scale: HolyScenes contains scenes with over 2,000 images and large physical dimensions, with the Badshahi Mosque reported as spanning 25,600 square meters. Results on Lerf-OVS and DL3DV-1K indoor scenes appear in supplementary material.

Methodology in Plain English

The authors build on 3D Gaussian Splatting, which models a scene as a collection of 3D anisotropic Gaussians each carrying position, covariance, opacity, and spherical-harmonic color coefficients. To this standard tuple they add one small extra thing per Gaussian: a learnable feature vector with a very low dimensionality, which they call the semantic bottleneck.

Training works in stages. The Gaussian attributes and the low-dimensional features are optimized jointly using the differentiable rasterizer, which produces a 2D feature map from any camera view — cheap, because each pixel carries only a handful of numbers. That small feature map is then fed into a multiresolution hash grid inspired by Instant-NGP. Here lies the key twist: the hash lookup is indexed by feature similarity rather than spatial location, so two pixels with similar low-dimensional features land near the same entries in the hash table regardless of where they appear in the image. Linear interpolation between neighboring table entries produces embeddings that are concatenated across levels and passed through a small MLP, which predicts high-dimensional CLIP and DINOv2 features.

To supervise these predictions, the authors compare them against CLIP feature pyramids built from image crops at multiple scales. Because that comparison is noisy, they add three fixes. First, they render features at higher resolution than the target and use a small three-layer convolutional network to learn per-feature attention weights, so a weighted average (rather than uniform averaging) produces the downsampled map — this is the Attenuated Downsampler. Second, they force a single shared MLP to predict both CLIP and DINOv2 features and apply weight decay, coupling the two feature maps. Third, instead of constraining features to lie inside SAM masks as prior work does, they penalize the variance of CLIP features within each SAM mask, encouraging intra-object consistency without trusting mask boundaries that are entangled in complex in-the-wild scenes.

For the messy realities of Internet photo collections, they integrate the SWAG approach into the photometric reconstruction loss to handle varying illumination, weather, and transient objects, ask an LLM to rewrite query prompts into more common synonyms, and compute pyramid scales in physical units from COLMAP metadata so that scale is consistent across images with different resolutions and camera parameters. The full objective combines RGB reconstruction with the CLIP supervision loss and the DINO and SAM regularization terms, weighted by hyperparameters, and everything is optimized jointly from scratch without a pre-existing Gaussian model.

Why This Matters

Impact on research. The paper challenges an implicit assumption in the feature-distillation literature — that methods validated on small, controlled object-centric datasets will scale to massive in-the-wild landmark collections. It provides evidence that they do not, identifies semantic misalignment from 2D feature pyramid supervision as a contributing cause, and offers a concrete architectural remedy. The finding that hashing on feature similarity rather than spatial location improves results while cutting memory and runtime is a transferable design idea.

Real-world applications:

• Virtual tourism and digital preservation, letting users query landmarks like the Milano Cathedral or St. Paul Cathedral for architectural elements ("spires," "lintels," "windows") and zoom to them interactively.
• Text-guided navigation and visual question answering over reconstructed cultural heritage sites, which the authors explicitly name as applications hindered by prior per-prompt optimization times.
• Scene retrieval and multimodal reasoning over large spatial environments where semantic grounding on 3D geometry is required.
• Educational and accessibility experiences built on explorable models of historically significant sites reconstructed from unconstrained photo collections.

Industry relevance. Rendering and graphics pipelines (the paper's own CCS categories) depend on the cost of representing and querying scene semantics. A representation that stores only three extra numbers per Gaussian and trains in ~10GB of memory versus ~15GB for alternatives lowers the hardware barrier for building interactive 3D products. Reducing prompt query time from roughly two hours to fractions of a second is the difference between a batch research workflow and an interactive user-facing tool.

Future Directions

• More spatially precise 2D features. The authors state their reliance on multi-scale CLIP features introduces spatial ambiguities that are hard to fully resolve, and suggest advances in 2D localization techniques could improve results.
• Better scale fusion. Because architectural elements vary enormously in size, large-scale features can dominate and obscure smaller details, and the current simple averaging may discard fine-grained semantics. The authors propose attention-based mechanisms or scale-specific decoders as candidate solutions.
• Smarter relevancy scoring. The LERF-style relevancy computation struggles to distinguish semantically related regions — the paper's example is decorative openings being scored above actual windows on Notre Dame Cathedral. Integrating context-dependent negative text-prompt selection instead of generic negatives like "Objects" or "Things" is one suggested route.
• Extending evaluation beyond landmark pyramids. The paper only previews indoor results on Lerf-OVS and DL3DV-1K in supplementary material, leaving broader validation of the large-scene claims as open ground.

Target Audience

Researchers and practitioners in 3D computer vision and graphics working on neural scene representations, open-vocabulary 3D segmentation, and 3D Gaussian Splatting. It is most valuable to readers already familiar with LERF, LangSplat, and Feature3DGS who want to understand the specific failure modes of feature distillation at Internet scale, and to engineers building interactive or memory-constrained 3D reconstruction products where per-prompt latency and GPU memory are hard constraints. Readers without a background in radiance fields, hash encoders, or vision-language embeddings will find the methodological density high.

Authors’ abstract

Embedding a language field in a 3D representation enables richer semantic understanding of spatial environments by linking geometry with descriptive meaning. This allows for a more intuitive human-computer interaction, enabling querying or editing scenes using natural language, and could potentially improve tasks like scene retrieval, navigation, and multimodal reasoning. While such capabilities could be transformative, in particular for large-scale scenes, we find that recent feature distillation approaches cannot effectively learn over massive Internet data due to challenges in semantic feature misalignment and inefficiency in memory and runtime. To this end, we propose a novel approach to address these challenges. First, we introduce extremely low-dimensional semantic bottleneck features as part of the underlying 3D Gaussian representation. These are processed by rendering and passing them through a multi-resolution, feature-based, hash encoder. This significantly improves efficiency both in runtime and GPU memory. Second, we introduce an Attenuated Downsampler module and propose several regularizations addressing the semantic misalignment of ground truth 2D features. We evaluate our method on the in-the-wild HolyScenes dataset and demonstrate that it surpasses existing approaches in both performance and efficiency.