AOV-GS: Active Open-Vocabulary Semantic Mapping with 3D Gaussian Splatting

AI summary

Overview

Research area: Computer vision and robotics — specifically active (next-best-view) mapping with 3D Gaussian Splatting combined with open-vocabulary semantic segmentation using frozen foundation models (SAM, CLIP).

Technical level: Intermediate. The paper assumes familiarity with 3D Gaussian Splatting, information-gain next-best-view planning, instance segmentation, and vision-language embedding spaces, but the core idea is explained clearly enough for a reader with general machine learning background.

Scope: The paper proposes AOV-GS, a mapping system that steers a robot's exploration toward semantically novel viewpoints using frozen SAM and CLIP, then fits an offline language field on the collected embeddings so the resulting map can answer free-form text queries, evaluated on Replica (eight scenes) and Matterport3D (five scenes).

What This Paper Is About

Robots working in human environments need maps that do two things at once: support geometric navigation and answer open-ended questions like "find the chair" or "locate the table near the window." Existing geometry-driven active mapping systems maximize coverage but treat semantics as a passive by-product of whatever trajectory happens to be flown, while existing active semantic mapping systems typically depend on supervised segmentation with a fixed label set (the paper cites ActiveSGM's per-scene OneFormer fine-tuning as the example). The goal of AOV-GS is to close that gap: explore an unseen scene with frozen foundation models only, choose viewpoints that are semantically informative as well as geometrically useful, and still preserve competitive RGB-D reconstruction quality.

Key Contributions

1. A semantic re-ranking criterion for coarse exploration. The top-K candidates (K = 8 by default) produced by the geometric planner are re-ranked by CLIP mask-embedding novelty computed from SAM instance masks, biasing exploration toward semantically novel object instances. The criterion applies only in stage 0; from stage 1 onward planning reverts to geometry-only behavior as in ActiveGS.

2. An active open-vocabulary semantic mapping pipeline. Online semantic exploration is combined with offline LangSplatV2 language-field construction, trained directly on collected CLIP mask embeddings rather than manually annotated masks, with no per-scene segmenter fine-tuning.

3. A two-benchmark evaluation against matched baselines. Under the same SAM–CLIP perception stack, AOV-GS reaches 34.6% average language-field mIoU on Replica (+1.3 pp over geometry-only) and 23.1% on Matterport3D (+1.8 pp), with per-scene Replica results explicitly analyzed including regressions.

4. A trajectory-transfer experiment. Replaying ActiveSGM's recorded pose sequence through the same SAM–CLIP and LangSplatV2 pipeline yields 28.2% on Replica, below both the geometry-only baseline (33.3%) and AOV-GS (34.6%), arguing that viewpoint selection must be matched to the semantic representation used downstream.

Main Findings

• Modest but consistent average semantic gains on Replica: AOV-GS averages 34.6% open-vocabulary language-field mIoU over eight scenes, versus 33.3% for the geometry-only planner (+1.3 pp) and 32.1% for the passive fixed-trajectory baseline (+2.5 pp). With CorrCLIP, AOV-GS reaches 35.6% versus 33.6% for Geometry-based + CorrCLIP, a 2.0 pp gap.

• Per-scene Replica results are heterogeneous: AOV-GS is strongest on office3 (39.3%), office4 (38.7%), room1 (42.8%), and office0 (34.4%, +7.1 pp over Geometry-based). It regresses on office1 (30.1% vs. 22.8% for Passive but 1.2 pp below Geometry-based), office2 (28.7%, 3.9 pp below Geometry-based), room0 (34.7% vs. Passive's 35.3%), and room2 (27.9% vs. Passive's 30.9%).

• Semantic feature quality, not scene size, drives the variance: The authors report no consistent correlation with scene area or object count. Large, well-segmented objects produced stable SAM masks and CLIP embeddings, while low illumination, overexposure, and cluttered scenes with many small objects reduced feature quality.

• CorrCLIP helps most where masks fragment: On office2, AOV-GS is 3.9 pp below Geometry-based, but adding CorrCLIP recovers 8.6 pp. Because CorrCLIP is applied both to candidate masks used for re-ranking and to accepted keyframe masks populating the embedding bank, the authors note this combines planner and representation effects.

• Failure signatures differ by scene: office1 is described as the darkest Replica scene in the image-statistics analysis (mean RGB intensity 44/255; 5th percentile 4/255). On room0, AOV-GS improves floor IoU from 17.4% to 57.2% and chair from 75.2% to 82.8% versus Passive, yet the full-category mean drops from 35.3% to 34.7%. On room2, chair IoU falls from 76.3% to 32.8% while floor rises from 7.0% to 57.6%, which the authors attribute to which object views were collected rather than to reconstruction quality.

• Gains on Matterport3D: AOV-GS averages 23.1% versus 21.3% for Geometry-based (+1.8 pp), with the largest absolute gains on GdvgFV5R1Z5 (+3.5 pp), gZ6f7yhEvPG (+2.7 pp), and pLe4wQe7qrG (+1.9 pp). With CorrCLIP, AOV-GS reaches 24.0% versus 21.8% for the matched CorrCLIP baseline (+2.2 pp).

• Trajectories do not transfer across semantic stacks: ActiveSGM's trajectory replayed with the SAM–CLIP + LangSplatV2 pipeline averages 28.2% on Replica, 6.4 pp below AOV-GS, with the largest degradation on office4 (8.9%) and a still-high 40.2% on room1.

• Reconstruction quality is preserved rather than improved on Replica: A single run gives AOV-GS PSNR 31.19 dB, SSIM 0.964, LPIPS 0.123, and depth L1 1.18 cm, versus 31.47 dB, 0.964, 0.131, and 1.26 cm for Geometry-based. The paper describes PSNR, SSIM, and LPIPS as matched while depth L1 improves.

• Five-seed stability on Replica: PSNR 31.53 ± 2.44 versus 31.62 ± 2.63 for Geometry-based; SSIM 0.965 ± 0.014 for both; LPIPS 0.126 ± 0.027 versus 0.128 ± 0.033. On office0, semantic mIoU is 33.6 ± 1.1% for AOV-GS versus 28.8 ± 1.4% for Geometry-based.

• Matterport3D rendering improves: AOV-GS gives 26.25 dB PSNR, 0.931 SSIM, 0.232 LPIPS, and 2.83 cm depth L1 versus 26.19 dB, 0.928, 0.233, and 2.80 cm for Geometry-based — depth L1 is the only metric slightly above the geometry baseline.

• Runtime and memory costs: Semantic scoring costs roughly 8 s per view on office0 with an NVIDIA RTX 4070 Ti SUPER (16 GB). A full office0 run takes 52:32, 61:23, and 70:41 for K = 4, 8, and 16, with peak GPU memory of 12.59, 12.92, and 13.80 GB. Offline language-field training peaks at 12.6 GB at the default 0.5 render downscale and ranges from 9.9 to 29.1 GB across evaluated downscales; multi-scale query decoding takes 1.41 s per query–view pair at 4.2 GB.

• ActiveSGM is far more memory-hungry: On Matterport3D both open-vocabulary planners fit in 16 GB, whereas ActiveSGM in the authors' setup required approximately 36–48 GB because of the OneFormer module.

• Ablations on office0 (23 categories, 2477 query pairs): K = 4 gives 33.2% mIoU, K = 8 gives 34.4%, and K = 16 also gives 34.4% while costing more time. Render downscale 1.0 overfits CLIP feature noise (26.2%), 0.25 gives 32.0% at 9.9 GB, and 0.5 gives the best 34.4%. Negative prompts matter heavily: no negatives gives 25.1%, one-class-versus-all-others gives 17.7%, and one-class-versus-"other" gives 34.4%. Thresholds of 0.50, 0.55, and 0.60 give 25.0%, 34.4%, and 33.9%.

• Closed-set context numbers: ActiveSGM with per-scene OneFormer reaches 78.8% closed-set mIoU on Replica and 61.5% on Matterport3D in the authors' runs; these are listed as supervised references outside the open-vocabulary protocol, not as baselines.

Methodology in Plain English

Map representation and simulator. The system builds the map with SplaTAM, an isotropic 3D Gaussian Splatting reconstruction backbone, wrapped in ActiveGS-style active planning. Habitat supplies synchronized RGB-D frames, and a voxel-based exploration map plus an RRT-based local planner handles collision-free candidate sampling and view execution. All experiments use Habitat ground-truth camera poses as oracle SLAM input — no estimated poses or localization noise.

Scoring a candidate view in two ways. For each candidate camera pose, the current Gaussian map is rendered and the number of unseen silhouette pixels is counted; this geometric information gain is normalized across candidates and down-weighted by distance from the robot. In stage 0 only, the top-K candidates by this geometric score are then rendered as RGB images, segmented with SAM, and each masked region is encoded with CLIP into a 512-dimensional embedding.

Measuring semantic novelty. Each candidate embedding is compared by cosine similarity against a growing bank of embeddings extracted from keyframes the robot has actually visited. Novelty is one minus the best match, averaged over masks. Candidates are only used transiently for scoring and are never inserted into the bank. To avoid an empty reference set, semantic scoring is skipped until the bank contains at least one embedding (the configured threshold is 1). The final score is the geometric score multiplied by a softmax-normalized semantic novelty, and the argmax is executed.

Switching back to pure geometry. From stage 1 onward, planning reverts to geometry-only behavior identical to ActiveGS. Semantic novelty therefore shapes the coarse trajectory, not the refinement.

Perception details. SAM ViT-H produces instance masks on accepted keyframes; CLIP ViT-B/16 encodes each region into a 512-D vector. Both models are frozen, with no per-scene fine-tuning. Extraction runs in a background thread so it does not block the planner. CorrCLIP-style mask merging and inter-class suppression are optionally applied, and when enabled are used both for candidate re-ranking and for the keyframe embedding bank.

Building the queryable language field offline. After exploration, the stored CLIP mask embeddings train a LangSplatV2 sparse language field on the Gaussians: k-means over mask embeddings initializes a global codebook, each Gaussian receives sparse coefficients over codebook entries, and text prompts are matched against multi-scale (s/m/l) rendered cosine-similarity maps, thresholded at τ_q = 0.55 with LangSplatV2 "other" negatives and open-dictionary negatives. Language-field training uses render downscale 0.5 and is decoupled from geometric refinement.

Evaluation protocol. The primary metric is open-vocabulary language-field mIoU on held-out novel views: each dataset category name is used as the text query, the dataset annotation supplies the binary ground truth, and IoU is averaged over category–view pairs and then over scenes. Secondary metrics are PSNR, SSIM, LPIPS, and depth L1 after refinement. Replica uses eight scenes (office0–office4, room0–room2) with a 2000-step exploration limit and 680×1200 RGB-D; Matterport3D uses five scenes (GdvgFV5R1Z5, gZ6f7yhEvPG, HxpKQynjfin, pLe4wQe7qrG, YmJkqBEsHnH) with a 5000-step budget. No passive semantic baseline is reported for Matterport3D because none exists for that protocol.

Why This Matters

Impact on research. The paper reframes semantics as something that can actively drive data acquisition rather than something computed after the fact, and it does so without closed-set supervision. Its most pointed result is negative-leaning: an ActiveSGM trajectory replayed through a SAM–CLIP language field drops to 28.2% on Replica, 6.4 pp below AOV-GS. That suggests benchmark trajectories and semantic representations must be co-designed, which is a useful corrective for a subfield that often swaps perception modules while holding trajectories fixed. The explicit reporting of four Replica regressions, with per-scene diagnosis (office1's 44/255 mean intensity, office2's mask fragmentation, room2's chair collapse from 76.3% to 32.8%), is also a contribution in itself.

Real-world applications:

• Service and assistive robots that must respond to ad-hoc natural-language requests ("bring me the cup near the sink") in homes and offices without a predefined object taxonomy.
• Warehouse and logistics inspection, where a robot could prioritize unexplored regions containing object categories it has not yet observed.
• Construction and facility digitization, producing navigable 3D reconstructions that are also text-searchable after the scan.
• Search-and-rescue or inspection in unknown indoor spaces, where biasing coverage toward semantically novel objects could reduce time to first detection.

Industry relevance. The memory profile is the practical headline: both open-vocabulary planners fit in 16 GB on an RTX 4070 Ti SUPER, whereas ActiveSGM in the same setup needed approximately 36–48 GB due to OneFormer. Removing per-scene fine-tuning also removes a per-deployment training step. The costs are explicit — roughly 8 s per view for semantic scoring, and total office0 runs of 52:32 to 70:41 depending on K. The paper does not report real-robot deployment or performance under SLAM drift, so industrial adoption would depend on closing that gap.

Future Directions

• Robustness to pose error and real sensors. All experiments use Habitat ground-truth camera poses as oracle SLAM input; the authors state that robustness to SLAM drift and real sensor noise is outside the scope of this simulator study. Evaluating AOV-GS with estimated poses or on a physical platform is the obvious next test.

• Making semantic scoring cheaper. Candidate views are scored sequentially at roughly 8 s per view, and runtime grows much faster than memory as K increases (52:32 at K = 4 to 70:41 at K = 16). Batched or cached scoring could make larger K practical.

• Fixing the perception bottleneck. The office2 result — 3.9 pp below geometry-only, recovered by 8.6 pp with CorrCLIP — and the office1 lighting analysis both point at SAM/CLIP feature quality under poor illumination and mask fragmentation. The authors note that CorrCLIP is not uniformly beneficial, since merging adjacent regions or suppressing small masks can remove useful evidence.

• Extending semantic guidance beyond stage 0. Semantic re-ranking is deliberately confined to coarse exploration and disabled from stage 1 onward, and the paper notes that MP3D's longer trajectories reduce the relative influence of the semantic planner. Whether semantic novelty also helps during refinement, or on larger layouts where it currently contributes less, is left open.

Target Audience

Robotics and computer vision researchers working on active perception, neural mapping (3DGS or NeRF SLAM), and open-vocabulary 3D scene understanding. It is particularly relevant to readers comparing active semantic mapping approaches against ActiveGS, ActiveSplat, ActiveGAMER, ActiveSGM, NARUTO, and LangSplat-style language fields, and to engineers evaluating whether a frozen SAM–CLIP stack is a viable substitute for per-scene supervised segmentation in a deployed mapping system. Readers interested in an honest account of failure modes — including negative per-scene results and the trajectory-transfer experiment — will find the paper's per-scene diagnostics more useful than its headline averages.

Authors’ abstract

Robots deployed in human-centered environments need maps that support both geometric navigation and open-vocabulary semantic queries beyond a fixed label set. A key challenge is enriching such maps with semantic information without relying on manually annotated segmentation masks or closed-set semantic categories. Existing geometry-driven active mapping maximizes coverage but treats semantics as a passive by-product of the collected trajectory, whereas active semantic mapping typically depends on supervised semantic segmentation. We present AOV-GS, an active 3D Gaussian mapping system that guides exploration using frozen foundation models. During coarse exploration, AOV-GS re-ranks geometrically promising candidate views according to CLIP mask-embedding novelty computed from SAM instance segmentation, biasing exploration toward semantically informative regions. Frozen SAM and CLIP provide instance masks and language-aligned embeddings throughout mapping, while a lightweight scene-specific language field is fitted offline directly to the collected CLIP embeddings rather than manually annotated masks, enabling open-vocabulary text-query segmentation on novel views. In a controlled comparison using the same SAM-CLIP perception stack, AOV-GS improves average language-field mIoU over geometry-only active planning by 1.3 percentage points on Replica (34.6% on eight scenes) and by 1.8 points on Matterport3D (23.1% on five scenes). With CorrCLIP, AOV-GS reaches 35.6% on Replica, 2.0 points above the matched Geometry-based + CorrCLIP baseline. Novel-view PSNR remains comparable to the geometry-only baseline on both datasets while depth L1 improves on Replica. Code is available at https://github.com/OptimusFaber/AOV-GS.