LatentAM: Real-Time, Large-Scale Latent Gaussian Attention Mapping via Online Dictionary Learning

AI summary

Overview

Research area: Robotics / 3D computer vision — specifically online dense visual–language mapping (SLAM) and open-vocabulary scene understanding.

Technical level: Advanced. The paper assumes familiarity with 3D Gaussian Splatting, SLAM pipelines, vision–language models, and dictionary learning.

Scope: The paper introduces LatentAM, an online 3D Gaussian Splatting mapping framework that learns a compact, adaptive dictionary of vision–language embeddings so that a robot can build large-scale, open-vocabulary 3D feature maps from streaming RGB-D data at near-real-time speed.

What This Paper Is About

Robots need maps that carry semantic meaning, not just geometry — so that a user can ask for "the chair" and have the robot find it in 3D. Modern vision–language models (VLMs) provide rich semantic features, but fusing them into a 3D map usually requires either expensive per-model decoders trained offline, or a fixed codebook that cannot adapt as the robot explores new places.

LatentAM's goal is to build latent (feature-carrying) 3D Gaussian maps online from streaming observations, in a way that is model-agnostic and requires no pretraining, while remaining fast enough (12–35 FPS) and memory-efficient enough to cover large multi-floor environments.

Key Contributions

1. Online dictionary learning for feature mapping. Instead of distilling VLM embeddings through a model-specific decoder, each Gaussian primitive stores a compact query vector (32 dimensions by default) that is converted into an approximate VLM embedding via an attention mechanism over a learnable dictionary. The dictionary is initialized from streaming observations and optimized online.

2. Trust-region regularized streaming optimization. A "soft" trust-region penalty keeps dictionary atoms near their initial values, preventing overfitting and catastrophic forgetting as scene semantics evolve. This is combined with a historical learning strategy that samples past keyframes from a buffer.

3. Streaming K-means dictionary initialization. Incoming embeddings are summarized into micro-factors (5 by default) via local K-means and merged/re-clustered with global centers, giving explicit control over dictionary size (K = 2000 by default) — unlike pairwise-similarity methods that lack size control or SVD-based approaches that are computationally expensive.

4. Large-scale voxel-hashing map management. The global Gaussian map lives in CPU memory indexed by a voxel hash (0.01 m voxels), while only an active local map is transferred to the GPU and optimized, keeping GPU memory bounded on long trajectories.

Main Findings

• Feature reconstruction fidelity: LatentAM reports a 39.2% improvement in feature reconstruction accuracy and 28.8× speedups compared to the state-of-the-art method (M3). On TUM, its cosine loss is 0.110 / 0.096 / 0.094 on TUM1 / TUM2 / TUM3, versus 0.136 / 0.116 / 0.133 for M3 and 0.130 / 0.078 / 0.100 for Feature-3DGS.

• Open-vocabulary segmentation: On TUM, LatentAM reaches mIoU of 0.792 / 0.960 / 0.885 (TUM1/2/3), compared with 0.674 / 0.957 / 0.910 for M3 and 0.576 / 0.949 / 0.797 for Feature-3DGS. It also leads on the Replica scenes (Room0–Office4, Lab) reported in Table I.

• Large-scale scalability: On the FastCaMo-Large dataset, LatentAM reports cosine losses of 0.079 (Floor1), 0.074 (Floor2), 0.043 (Stair1), 0.042 (Stair2), while M3 runs out of memory on Floor1 and Floor2 due to unbounded dictionary growth. LatentAM's streaming K-means keeps dictionary size fixed.

• Runtime: LatentAM runs at 17.89–20.24 FPS on TUM and 13.64–28.62 FPS on FastCaMo-Large scenes, versus 0.43–1.57 FPS for Online Language Splatting and 6.33–9.10 FPS for OmniMap. Per-keyframe mapping runtime stays under 450 ms including VLM feature extraction. The paper notes 12 FPS with short keyframe intervals (every 4 frames on average) and over 30 FPS with longer intervals (every 10 frames on average).

• Model-agnostic operation: The same framework works plug-and-play with CLIP-SAM, DINOv3-SAM, and LSeg on TUM, with LatentAM outperforming the offline baselines on TUM1 and TUM3 across all three VLM settings. In TUM2 (a highly repetitive scene with minimal camera motion), Feature-3DGS is slightly better.

• Memory efficiency: Peak GPU usage with local mapping is 5180.16 MB / 7388.47 MB / 8538.66 MB on TUM1/2/3, compared to 6428.77 / 14133.2 / 13845.0 MB without local mapping and 5188.40 / 12852.8 / 10767.2 MB for M3.

• Ablation — dictionary learning matters: Full method (streaming K-means + dictionary learning) achieves 0.110 cos-loss / 0.792 mIoU on TUM1, versus 0.146 / 0.506 with neither component. The paper states refining the dictionary via the proposed optimization yields up to 59.4% improvement in mIoU over a static dictionary.

• Photometric trade-off: PSNR is lower than Online Language Splatting's, which the authors attribute to iteration counts — LatentAM uses 1 Stage I and 5 Stage II iterations, whereas Online Language Splatting uses 150 iterations by default.

• Trust-region regularization: Shown qualitatively on the large-scale scene, where without regularization the model misdetects monitors on the wall.

• Custom dataset: A multi-floor campus dataset captured with a Gemini 335Le RGB-D camera, spanning approximately 530 m of trajectory over 14.6 minutes, across two floors connected by a staircase. The full quantitative results of this section are not reported in the provided (truncated) text.

Methodology in Plain English

Standard 3D Gaussian Splatting represents a scene as many 3D blobs, each with position, orientation, color, scale, and opacity. LatentAM adds a sixth attribute to each blob: a small query vector (32 numbers). Attached to this is a shared "dictionary" — a bank of representative VLM embeddings — plus a small projection matrix that maps queries into the VLM feature space.

To recover what a Gaussian "means" semantically, the system projects its query into feature space, compares it against every dictionary entry using cosine similarity, and takes a softmax-weighted average of the dictionary entries. This is essentially an attention read-out over a learned vocabulary of semantic concepts. The whole thing is trained with standard photometric and depth losses plus a feature reconstruction loss.

The key twist versus prior work is that this dictionary is not fixed or built offline. Whenever a new keyframe arrives, newly observed embeddings are clustered with the existing dictionary centers (streaming K-means), producing a fresh initialization. Then a two-stage optimization runs: stage one jointly nudges the Gaussians, projection matrix, and dictionary (1 iteration by default); stage two refines only the projection matrix and dictionary (5 iterations by default) with the Gaussians frozen, which is cheaper. A trust-region penalty keeps dictionary atoms from drifting too far from their initialization, and a small buffer of past keyframes (B = 5) is replayed to prevent forgetting.

For scale, the system separates the map into a global store in CPU memory indexed by a voxel hash and a local active map on the GPU. Gaussians are added and re-synced to the global map when the robot has traveled far enough, so GPU memory stays bounded regardless of trajectory length.

Why This Matters

Research impact: The paper reframes online semantic mapping as a dictionary learning problem rather than a distillation problem. It shows that a compact, continuously updated dictionary can beat offline decoders in feature fidelity, challenging the widespread assumption that model-specific pretrained decoders are required. It also demonstrates that a latent map can scale to environments where a growing dictionary would otherwise exhaust memory.

Real-world applications:

• Home and service robots that need to answer natural-language queries like "bring me the cup on the desk" using maps built during navigation.
• Warehouse and logistics robots that must locate arbitrary, unlabeled objects by description rather than by a fixed class list.
• Augmented and virtual reality where a headset builds a semantic 3D map of a room in real time for content placement or interaction.
• Search-and-rescue or inspection robots operating in unknown multi-floor buildings where a human operator queries the map for hazards or specific objects after the fact.

Industry relevance: The combination of model-agnosticism, no pretraining requirement, and bounded GPU memory is attractive for deployment. A company can swap in a newer VLM at test time without retraining the mapping stack, and the CPU/GPU split lets the system run on hardware with limited GPU memory while covering buildings rather than single rooms. The reported 28.8× speedup over M3 is directly relevant to onboard operation.

Future Directions

1. Tighter integration with tracking. LatentAM currently relies on an external real-time tracker (GS-ICP or MonoGS) and does not jointly optimize camera poses with the map. Coupling tracking and feature mapping could improve robustness where geometric-only trackers drift under sparse keyframes.

2. Adaptive dictionary sizing. The dictionary size K is set to a fixed 2000 and the paper observes that K has little effect on performance on the evaluated data. Whether this holds in environments with far richer or more diverse semantics remains an open question.

3. Closing the loop with the VLM. The system distills embeddings into a map but does not query the VLM during mapping. Using downstream task feedback to shape the dictionary online could improve efficiency further.

4. Dynamic and non-stationary scenes. The trust-region regularization is designed for non-stationary feature distributions over long trajectories, but the paper's evaluation focuses on static reconstruction benchmarks. Behavior in scenes with moving objects, changing lighting, or seasonal appearance changes is not established.

5. Extending beyond CLIP/DINOv3/LSeg. The framework is presented as model-agnostic, but only three VLMs are evaluated and only on TUM; broader validation across model families and datasets would strengthen the claim.

Target Audience

Robotics and SLAM researchers working on semantic or open-vocabulary mapping; 3D vision researchers interested in Gaussian Splatting and neural field representations; practitioners building embodied AI or autonomous systems that need real-time, large-scale, language-queryable 3D maps; and engineers evaluating the deployment trade-offs of VLM-based perception under memory and latency constraints.

Authors’ abstract

We present LatentAM, an online 3D Gaussian Splatting (3DGS) mapping framework that builds scalable latent feature maps from streaming RGB-D observations for open-vocabulary robotic perception. Instead of distilling high-dimensional Vision-Language Model (VLM) embeddings using model-specific decoders, LatentAM proposes an online dictionary learning approach that is both model-agnostic and pretraining-free, enabling plug-and-play integration with different VLMs at test time. Specifically, our approach associates each Gaussian primitive with a compact query vector that can be converted into approximate VLM embeddings using an attention mechanism with a learnable dictionary. The dictionary is initialized efficiently from streaming observations and optimized online to adapt to evolving scene semantics under trust-region regularization. To scale to long trajectories and large environments, we further propose an efficient map management strategy based on voxel hashing, where optimization is restricted to an active local map on the GPU, while the global map is stored and indexed on the CPU to maintain bounded GPU memory usage. Experiments on public benchmarks and a large-scale custom dataset demonstrate that LatentAM attains significantly better feature reconstruction fidelity compared to state-of-the-art methods, while achieving near-real-time speed (12-35 FPS) on the evaluated datasets. Our project page, including the code, is available at: https://junwoonlee.github.io/projects/LatentAM