ACE-SLAM: Scene Coordinate Regression for Neural Implicit Real-Time SLAM

AI summary

Overview

Research area: visual SLAM (simultaneous localization and mapping), specifically neural implicit RGB-D SLAM and Scene Coordinate Regression (SCR).

Technical level: Advanced.

Scope: One sentence — the paper introduces ACE-SLAM, a neural implicit RGB-D SLAM system that uses a Scene Coordinate Regression network (a new triplane voting architecture called TriMLP) as its core map representation, and claims the first strictly real-time neural implicit RGB-D SLAM system of this kind.

What This Paper Is About

Most neural implicit SLAM systems represent a scene with rendering-based models such as NeRF, which require sampling along camera rays to extract geometry and therefore run far slower than the rate at which a camera produces frames. This paper asks whether a different implicit representation, Scene Coordinate Regression (SCR) — a small network that directly maps 2D image features to 3D global coordinates — can instead serve as the core map inside a live SLAM pipeline. The goal is a compact, continuously trained map that supports fast relocalization and tracking at frame rate, without the extra subsystems (decoupled tracking, bundle adjustment, explicit loop closure, semantic segmentation) that other pipelines add.

Key Contributions

1. The first RGB-D SLAM pipeline that fully leverages an SCR-based implicit map representation for strict real-time operation, capable of processing live data at frame rate, unlike previous neural implicit SLAM methods.
2. A novel SCR network architecture, TriMLP, specifically designed for continuous online training and inference within the SLAM loop, which factorizes 3D coordinate regression into three orthogonal 2D classification problems over the XY, XZ, and YZ planes.
3. A detailed analysis of design choices and efficiency trade-offs when adopting SCR representations for practical, real-time SLAM (including comparison against a homogeneous-coordinate MLP baseline, HomMLP, and against sparse versus dense features).
4. Extensive evaluation on established SLAM benchmarks, including dynamic scenes, demonstrating resilience and competitive performance against other neural implicit RGB-D SLAM pipelines.

Main Findings

• Real-time operation achieved: On the Replica efficiency evaluation (Table 2, Replica Room0 per the table caption), ACE-SLAM reaches 29.71 FPS and a real-time factor of 99.0%, compared with ESLAM at 7.35 FPS (24.5%), NICE-SLAM at 0.33 FPS (1.1%), Point-SLAM at 0.27 FPS (0.9%), and iMAP* at 0.15 FPS (0.5%) — the paper states this outperforms other methods by up to two orders of magnitude.
• Compact maps: ACE-SLAM's map size is reported as 1.11 MB (HomMLP variant 1.01 MB, SuperPoint-feature variant 0.86 MB), versus 27.23 MB for Point-SLAM, 45.46 MB for ESLAM, and 95.86 MB for NICE-SLAM. Figure 1 describes the implicit map as 1 MB.
• Latency per frame: Localizing a new frame end-to-end takes 11 ms (93 FPS equivalent) using ACE features, or 13 ms (76 FPS equivalent) using SuperPoint features.
• TriMLP beats HomMLP: Using the same small number of hidden layers, TriMLP vastly outperforms the homogeneous-coordinate HomMLP architecture in trajectory accuracy across all sequences, with the gap most visible in large, detailed scenes. For example, on Replica R1, TriMLP gives 0.044 ATE RMSE versus 0.069 for HomMLP; on ScanNet 0106, 0.319 versus 0.765.
• Trajectory accuracy versus other systems: On static Replica, TUM-RGBD and ScanNet (Table 1, ATE RMSE in meters), ACE-SLAM reports e.g. Replica R0 0.027, O0 0.030, TUM fr1/desk 0.083, fr2/xyz 0.016, fr3/office 0.082, ScanNet 0000 0.164 and 0181 0.219. It is competitive with iMAP* in Replica and TUM, surpasses iMAP* on parts of ScanNet, and approaches NICE-SLAM on several sequences, but remains below more recent methods and pipelines that use specialized submodules (for example GO-SLAM reports 0.003 on Replica R0, ESLAM 0.007, Point-SLAM 0.006, NICE-SLAM 0.017, CO-SLAM 0.006).
• Dynamic-scene robustness without special modules: On TUM-RGBD dynamic sequences (Table 3), ACE-SLAM reports fr3/s/xyz 0.037, fr3/s/rpy 0.119, fr3/s/static 0.007, fr3/s/half 0.049, fr3/w/xyz 0.072, fr3/w/rpy 0.253, fr3/w/static 0.012, fr3/w/half 0.150 — comparable to or better than specialized pipelines that use semantic priors (NID-SLAM, FND-SLAM) on several sequences, while not using semantics or extra modules itself.
• Sparse versus dense features: Dense ACE features and sparse SuperPoint features perform comparably in small-scale scenes, but ACE features are generally more reliable and accurate overall. With SuperPoint features, accuracy degrades noticeably and several sequences fail (marked X): Replica R2, ScanNet 0000 and 0181, for example.
• Implicit loop closure and relocalization emerge naturally: Because frames interact only through the compact implicit map rather than through explicit frame-to-frame correspondences, similar features cluster in 3D, producing soft loop-closure behavior and global consistency without additional modules. The paper notes this could be made explicit in future work.

Methodology in Plain English

The system takes a stream of RGB-D frames. Each frame is processed by a frozen, pre-trained feature extractor — the ACE encoder, which produces dense features at 1/8 resolution, or SuperPoint, which produces sparse features — in a separate thread. The color image is used for feature extraction; depth is used to unproject each 2D keypoint into a 3D local coordinate with the known camera intrinsics, and the depth information is otherwise ignored.

The map itself is a small neural network that takes a feature descriptor and outputs the 3D global coordinate of that feature in the scene. Rather than regressing coordinates directly (as in the HomMLP baseline), the proposed TriMLP predicts classification distributions over discretized bases on the three orthogonal planes XY, XZ, and YZ, applies a softmax, and estimates each coordinate component by weighted averaging over the corresponding plane bases, then averaging compatible components across planes. The authors argue this structured "voting" gives a many-to-one mapping — different descriptors of the same 3D point can reach the same coordinate — which speeds up adaptation for shallow networks.

The camera pose for each incoming frame is estimated purely by relocalization: predicted global coordinates are aligned to observed local coordinates with a rigid alignment solved in closed form using Kabsch–Umeyama, made robust by RANSAC over triplets of correspondences with up to H pose hypotheses, selecting the hypothesis with the highest inlier ratio. This inlier ratio also serves as a signal of tracking quality and map degradation.

Mapping and tracking run in parallel optimization cycles. Frames become keyframes if enough time has elapsed since the last keyframe or if their inlier ratio drops below a threshold. Each cycle samples an optimization window containing the newest keyframes, the latest frame, and additional keyframes sampled with a probability that favors frames with low inlier ratios (one term for uniform coverage, one weighted by a coefficient set to 1). Features are sampled from those frames with a similar inlier-ratio-driven strategy, and the map is refined by stochastic gradient descent over mini-batches, with the poses held fixed. The authors state that compute cost stays fixed across all stages because the number of sampled frames, hypotheses, and features per cycle is constant.

Why This Matters

This work argues that SCR is a viable, efficient alternative to rendering-based implicit representations for SLAM, and reframes the field's emphasis toward real-time operation rather than accuracy alone. The authors point out that many neural implicit SLAM systems report near real-time capability based on tracking-iteration efficiency even though they may take orders of magnitude longer than the data capture; as an example, they note that iMAP reportedly takes over 10 minutes to map Replica Room0, a 1-minute sequence at 30 FPS.

Real-world applications named or implied by the paper:

• Autonomous navigation, where embodied edge applications impose hard real-time constraints.
• Robotic manipulation, listed among embodied edge applications requiring efficiency and robustness.
• Wearables, also listed among embodied edge applications.
• Privacy-sensitive mapping, since stored geometric information can only be accessed through corresponding images captured within the scene.

Industry relevance: the system runs on consumer-grade hardware (the experiments use an NVIDIA RTX 4090 GPU with 64 GB RAM and an Intel i7-12700K CPU) with maps on the order of megabytes, which matters for deployment on resource-constrained platforms. The privacy-preserving property and native relocalization also matter for applications where a map must be shared or stored without revealing the scene.

Future Directions

• Extending TriMLP beyond real-time SLAM: the authors state that its applicability to other SCR tasks, such as large-scale relocalization, remains to be explored.
• Using depth more fully: they note that the current feature extractors use only the color image, and suggest future extensions could incorporate depth to produce true RGB-D features or leverage depth priors as in ACE-Zero, enabling operation on RGB-only streams.
• Online test-time refinement of the feature extractors, which are kept frozen throughout the reported experiments.
• Making the implicit loop-closure behavior explicit, and improving accuracy without sacrificing online operation; additional geometry results are deferred to supplementary material, since the quantitative evaluation here focuses on trajectory accuracy rather than high-fidelity geometry.

Target Audience

Researchers and engineers working on visual SLAM, neural implicit representations, 3D scene mapping, and robot perception, particularly those interested in real-time performance and deployment on constrained hardware. The paper is also relevant to readers following Scene Coordinate Regression and relocalization research, and to practitioners who need compact, privacy-preserving maps for embodied applications. The material assumes familiarity with SLAM fundamentals, camera pose estimation, and neural network training, so it is best suited to readers with an advanced background in computer vision.

Authors’ abstract

We present a novel neural RGB-D Simultaneous Localization And Mapping (SLAM) system that learns an implicit map of the scene in real time. For the first time, we explore the use of Scene Coordinate Regression (SCR) as the core implicit map representation in a neural SLAM pipeline, a paradigm that trains a lightweight network to directly map 2D image features to 3D global coordinates. SCR networks provide efficient, low-memory 3D map representations, enable extremely fast relocalization, and inherently preserve privacy, making them particularly suitable for neural implicit SLAM. Our system is the first one to achieve strict real-time in neural implicit RGB-D SLAM by relying on a SCR-based representation. We introduce a novel SCR architecture specifically tailored for this purpose and detail the critical design choices required to integrate SCR into a live SLAM pipeline. The resulting framework is simple yet flexible, seamlessly supporting both sparse and dense features, and operates reliably in dynamic environments without special adaptation. We evaluate our approach on established synthetic and real-world benchmarks, demonstrating competitive performance against the state of the art. Project Page: https://github.com/ialzugaray/ace-slam