SuperMap: A Spatio-Temporal SLAM System for Visual-Language Navigation

AI summary

Overview

Research area: Robotics — simultaneous localization and mapping (SLAM), open-vocabulary semantic perception, 3D scene graphs, and vision-language navigation (VLN).

Technical level: Advanced. The paper assumes familiarity with SLAM state estimation, Kalman filtering, log-odds occupancy updates, Bayesian label fusion, and 4D scene graphs.

Scope: The paper presents SuperMap, an onboard, real-time system that builds and maintains an open-vocabulary, instance-level 4D spatio-temporal scene graph so a robot can answer language queries about objects, their relations, and their history.

What This Paper Is About

Robots asked to follow instructions like "go to the monitor next to the whiteboard" need a map that remembers which object is which over time, even as objects are occluded, moved, removed, or newly introduced. Foundation models detect and label objects well in single frames, but their outputs are intermittent and view-dependent, so naively dropping them into a mapping pipeline causes object identities to fragment and semantics to go stale. SuperMap's goal is to fuse high-frequency geometric SLAM with asynchronous open-vocabulary perception into a persistent, queryable 4D scene memory that survives occlusions and scene change.

Key Contributions

1. Open-vocabulary, spatio-temporal semantic SLAM: An online robotic system that builds a persistent, queryable open-vocabulary 4D scene memory suitable for downstream language-conditioned tasks.
2. Spatio-temporal object tracking: An online pipeline integrating 2D–3D association, validation, and change-aware updates to maintain instance consistency under occlusions, partial observations, label variability, and scene change.
3. Instance-level scene graph: A 4D graph that incorporates spatial and temporal information for each object, giving robots instance-level spatio-temporal reasoning (locating moved objects, recalling past scenes).
4. Open-source framework: The authors state they will release their change-detection benchmark, ablations and runtime profiling, and the real-robot visual-language navigation pipeline.

Main Findings

• Class-level segmentation on ScanNet: SuperMap reaches 55.48% accuracy without background, outperforming object-centric baselines such as ConceptGraphs and ConceptFusion. The paper states this is slightly lower (1.32%) than the point-feature fusion method RayFronts, while SuperMap maintains a significantly lower computational footprint. SuperMap's other class-level numbers are 27.42 mIoU / 43.50 f-mIoU / 55.48 Acc without background and 22.61 mIoU / 29.10 f-mIoU / 33.00 Acc with background.
• Instance-level segmentation on ScanNet: SuperMap reports 63.76 mAP50 / 74.72 mAP25 on Chair, 42.20 / 67.92 on Window, 62.50 / 62.50 on Refrigerator, 33.35 / 83.35 on Sofa, and 10.00 / 25.00 on Door, substantially above HOV-SG and ConceptGraphs.
• Memory and runtime (supplementary Table I): SuperMap uses 2030.09 MB average and 2184.86 MB peak memory with 0.3604 s mapping time per frame, versus HOV-SG's 8755.86 MB average / 10226.66 MB peak and 8.623 s, and ConceptGraphs' 859.86 MB average / 886.18 MB peak and 0.092 s.
• Long-horizon consistency in a real scene: In a 10-minute experiment in a 30m × 20m indoor area, three objects were removed (a plant, a trash can, a chair) and three added (a bucket, a cart, a safety sign). Newly introduced objects received new IDs while nearby objects retained their original IDs, and disappeared objects kept the same instance ID across time.
• Change-detection recall: SuperMap's object detection recall was 1.000 (bucket), 0.262 (cart), 0.583 (sign), 0.755 (plant), 0.434 (trash), 1.000 (chair); change-detection recall was 1.000, 0.622, 0.790, 0.865, 0.679, and 1.000 respectively. DualMap scored near zero on object detection and roughly 0.5 on change detection, which the authors attribute to a mathematical artifact of failing to detect objects initially. Khronos produced no consistent results in the table.
• Ablation over 41 annotated objects: The full system achieved precision 0.8677, recall 0.4955, F1 0.6308, beating all three ablated variants — without 2D tracker (0.7787 / 0.4595 / 0.5780), without semantic fusion (0.7929 / 0.3870 / 0.5201), and without geometric consistency update (0.8189 / 0.4448 / 0.5764).
• Grounded VLM reasoning: Using Gemini 2.0 Flash, structured scene-graph input yielded more reliable spatial reasoning (disambiguating a fire extinguisher relative to a cone and a plant), clearer temporal inference (tracing a bag's past trajectory via temporal edges), and more accurate instance-level retrieval than raw video input, which the authors report suffers from perspective distortion and hallucinations as temporal depth grows.
• Runtime profile: Pose estimation runs at a consistent 10 Hz, 2D instance segmentation at 1 Hz, 3D mapping at 3 Hz, and 4D scene graph updates at 5 Hz.

Methodology in Plain English

SuperMap splits the problem into layers.

Geometric layer. The system uses SuperOdometry with image, depth/LiDAR, and IMU streams to estimate the robot pose and build a colorized dense 3D reconstruction. This metric backbone makes it possible to back-project 2D detections into 3D, compensate for motion, and keep the global map consistent.

Instance layer. For each frame, GroundingDINO provides language-grounded detection and SAM2 provides instance segmentation. Each tracklet carries a hybrid state (2D image centroid, bounding-box width and height, and image-plane velocity — six dimensions, or eight with scale velocity). Instead of assuming linear 2D motion, the tracker projects the 3D centroid of a known object through the current pose to predict where it should appear, and feeds that projection into a Kalman filter as a prior. This 3D-to-2D motion compensation lets identities survive occlusion and aggressive robot motion that defeat pure 2D trackers.

Map updating. Each existing map point is checked against the current depth image: the projected depth is compared with the measured depth to form a residual. Points whose residual falls within a threshold are treated as observable, points behind the surface as unobservable, and points in front of the surface as disappeared. Occupancy is updated in log-odds form, so transient or moved objects get pruned. Labels are fused with a Bayesian categorical update using the detector's confusion matrix, and object points whose posterior belief is too small are removed — suppressing momentary misclassifications.

Topological layer. The map becomes a graph whose nodes are object instances, with spatial edges built from class-dependent geometric predicates (for example "on," gated by an IoU threshold) and temporal edges that trace trajectories across the association results.

Language interface. A local subgraph is serialized into text: instance ID, semantic label, 3D centroid, and symbolic relations. A schema defining the graph structure and coordinate frame is prepended, and explicit spatio-temporal cues are injected to encourage trajectory- and relation-aware reasoning. The VLM must return target instance IDs inside <answer></answer> tags; a parser extracts them, looks up their 3D centroids in the map, and passes those as navigation waypoints.

Why This Matters

Impact on research. The paper argues that standard static perception metrics (mIoU, mAP) do not capture long-horizon consistency in dynamic scenes. SuperMap contributes a capability checklist showing that it is the only real-time, open-vocabulary, instance-level system in its comparison table that supports both short-term and long-term change and a scene graph. The authors state that, to the best of their knowledge, SuperMap is the first real-time, spatio-temporal, open-vocabulary, instance-level object mapping framework handling both short-term and long-term dynamic objects, and they release the system to provide a deployable baseline.

Real-world applications:

• Indoor service and logistics robots that must recognize that a cart was moved or a bin removed since their last visit.
• Assistive or delivery robots following relational instructions such as "return to the chair that was near the plant earlier."
• Emergency-response or inspection robots that reason about objects under spatial constraints, such as locating an extinguisher relative to a cone and a plant.
• Warehouse or facility monitoring, where appearance and disappearance of items is itself the signal of interest.

Industry relevance. Because everything runs onboard in real time, the system is positioned for deployment rather than offline processing. The reported memory and per-frame timing figures matter for embedded platforms, and the fact that semantic perception runs at 1 Hz while geometry and mapping run faster is a practical design point for anyone integrating heavy foundation models into a robotic feedback loop.

Future Directions

• Better tracking of highly dynamic objects. The authors identify this as the primary limitation of the current system, suggesting specialized tracking-by-detection modules or efficient segmentation-based tracking.
• Automatic prompting instead of fixed prompts. The pipeline currently relies on a pre-defined list of object prompts for open-vocabulary 2D detection; open-world object discovery could remove that dependency and let the system operate in truly novel environments without prior semantic knowledge.
• Quantified navigation evaluation. The paper demonstrates language-guided navigation qualitatively (choosing among four visually identical whiteboards, and retrieving the painting next to the fridge), but does not report numerical navigation success rates, leaving a natural target for future benchmarking.
• Richer evaluation protocols for temporal consistency. The supplementary material explicitly calls for metrics that extend beyond static instance segmentation to measure identity stability as objects are occluded, move, leave, and reappear.

Target Audience

Robotics researchers and engineers working on SLAM, semantic mapping, and embodied AI; practitioners building language-guided navigation stacks who need a real-time, open-vocabulary, object-centric map; and graduate students studying 3D scene graphs, long-term autonomy, or the integration of foundation models into robotic perception. Readers without a background in state estimation or probabilistic mapping will find the method sections demanding.

Authors’ abstract

Robotic navigation in human environments requires a spatio-temporal semantic representation that can rec- oncile open-vocabulary perception with long-term environmental changes. While foundation models provide strong zero-shot recognition, their predictions are intermittent and view-dependent, and naively integrating them into mapping pipelines leads to identity drift and stale semantics over time. We present SuperMap, a 4D spatio-temporal mapping framework for language-guided navigation that integrates high-frequency geometric SLAM with asynchronous open-vocabulary perception. Our core contribution is a consistency-driven mapping engine that combines 3D-aware instance association/re-activation with a principled existence-and-label confidence update to maintain stable object identities and prune outdated map content under occlusions and scene changes. SuperMap produces a queryable 4D scene-graph representation that interfaces naturally with Vision-Language Models by supporting compositional queries over object semantics, relations, We demonstrate SuperMap on benchmarks and real robots, including dynamic scenes with appearance/disappearance and relocation, and provide ablations and runtime analysis. We release the full system as open-source to provide the community with a deployable baseline for open-vocabulary spatio-temporal mapping. Project website: superodometry.com/supermap.