TACO: Temporal Consensus Optimization for Continual Neural Mapping

AI summary

Overview

• Research area: Robotics, specifically dense 3D scene mapping with neural implicit representations, combined with continual learning and constrained optimization.
• Technical level: Advanced. The paper builds on neural implicit surface mapping (Co-SLAM), the method of multipliers from constrained optimization, and continual-learning regularization techniques such as Memory Aware Synapses (MAS).
• Scope: The paper proposes TACO (Temporal Consensus Optimization), a replay-free framework that lets a neural implicit map keep adapting to a changing environment by enforcing an importance-weighted agreement between the current model and its frozen past snapshots.

One note on the source: the provided paper content is truncated inside the Conclusion ("By treating"), so the closing discussion is not available here, and quantitative ScanNet results are referenced only as appearing in Appendix B.

What This Paper Is About

Online neural implicit mapping systems usually store and replay past observations or keyframes to avoid forgetting what they have already mapped, and they generally assume the scene is static. In real robot deployments, memory and computation are limited and the environment changes, so replayed observations conflict with new ones and degrade the map. TACO's goal is to keep a map both stable (retaining reliable past geometry) and adaptive (revising geometry that new observations contradict), without storing or replaying any past data.

Key Contributions

1. A temporal reformulation of continual mapping. TACO recasts continual neural mapping as a consensus problem over time, where frozen past model parameters ("historical snapshots") act as temporal neighbors of the current model rather than as replayed data.
2. An importance-aware consensus mechanism. A parameter-wise weighting scheme, derived from output-sensitivity importance estimates (inspired by MAS), decides which past parameters constrain the optimization and which are free to change, dynamically balancing stability against adaptation.
3. Replay-free operation. The method enforces consensus between model states instead of storing RGB-D frames or keyframes, enabling continual optimization under streaming observations and realistic memory constraints.
4. Validation in simulation and on real hardware. The authors test TACO on static RGB-D benchmarks (Replica, ScanNet), on dynamic simulated scenes from the Habitat Synthetic Scenes Dataset built with Habitat-Sim, on a larger Gibson scene, and on a real-world dynamic dataset collected with a TurtleBot.

Main Findings

• Static scenes: TACO matches the replay upper bound. On eight Replica scenes, TACO achieves performance comparable to Co-SLAM with replay and matches or outperforms all continual-learning baselines across the reported metrics (Artifacts, Holes, Chamfer Distance, Completion Ratio, Precision@5cm, and F1@5cm). It performs on par with or better than the strongest continual-learning baseline, MAS.
• Dynamic simulated scenes: TACO beats MAS on every reported metric. On the staged Habitat-Sim dynamic benchmark, TACO records Artifacts 6.38 versus MAS 8.53, Completion Ratio 74.37 versus 71.41, and F1@5cm 78.05 versus 73.43.
• Distinct failure modes in baselines. Replay-based methods such as Co-SLAM reconstruct moved objects at both their old and new locations, producing persistent "ghosting" artifacts because replay enforces consistency with observations that are no longer valid. Regularization-based MAS over-constrains parameter updates, causing mesh tearing and fragmented surfaces near objects that moved.
• Real-world adaptation with a single object move. When a yellow stool is moved from position A at time t0 to position B at time t1, TACO reconstructs the stool only at its current location, without ghost artifacts or geometric breakage, while Co-SLAM duplicates it and MAS distorts the affected geometry.
• Real-world adaptation under repeated changes. In a multi-stage hardware experiment involving relocating rows of cubes and then relocating the yellow stool into positions previously occupied by cubes, TACO revises outdated geometry while preserving stable regions; Co-SLAM accumulates severe ghost artifacts and MAS produces torn, fragmented geometry.
• Scalability to larger scenes. On a structurally complex Gibson scene, TACO produces more complete and smoother reconstructions than MAS with fewer discontinuities and less mesh fragmentation, though both show color inconsistency relative to replay-based Co-SLAM due to the absence of explicit appearance modeling.
• Memory footprint comparable to lightweight baselines, far below replay. Reported CPU resident set size (RSS) in MB: Co-SLAM 4165.1, KR (replay-based) 1912.4, UNIKD (knowledge distillation) 1945.1, Online MAS (regularization) 1800.6, and TACO 1864.5. The paper states TACO reduces memory usage by more than 50% relative to full replay.

Methodology in Plain English

The backbone is Co-SLAM, a real-time neural implicit mapping system that represents a scene with a multi-resolution hash feature grid plus geometry and color decoders, and renders depth and color by volume rendering along camera rays with a truncated signed distance function (SDF). TACO wraps this optimizer in a constraint.

The key idea is that instead of comparing the new model against stored images, TACO compares it against its own frozen previous parameters. At each time step, it defines a consensus target: a weighted average of the current parameters and the previous snapshot's parameters, where each parameter's weight reflects how important it has been. That target becomes an equality constraint, and the problem is solved with the method of multipliers, which alternates between a primal update (a few stochastic gradient descent steps on the reconstruction loss plus a penalty term) and a dual update that accumulates disagreement signals nudging the model back toward consensus. This is run for K iterations, and the final iterate becomes the new map.

Importance is measured without any extra network or ground truth. The authors define a proxy objective on the rendered outputs — the mean squared magnitude of predicted colors and depths over sampled rays — and accumulate the element-wise absolute gradient of that proxy with respect to the parameters across iterations. Parameters that consistently influence the rendered output get high importance and are strongly pulled toward their historical values; low-importance parameters adapt freely. Because accumulated importance grows over time, the weights are rescaled so their average magnitude matches the penalty parameter rho, preserving relative importance while keeping magnitudes stable. A masking threshold beta additionally zeroes out consensus constraints for parameters whose historical weight is below beta, so poorly observed or noisy parameters are not constrained early on.

Evaluation uses geometric metrics described in Appendix A, and all baselines — EWC, MAS, CNM, KR (with a replay buffer of 10 keyframes), and UNIKD — are implemented inside the Co-SLAM framework for controlled comparison. Co-SLAM with replay acts as an upper bound and Co-SLAM without replay as a lower bound.

Why This Matters

This work argues that the standard remedy for forgetting in neural mapping — storing and replaying old data — is actively harmful when the world changes, because it forces the map to agree with observations that are no longer true. Reframing consensus from the spatial domain (multi-robot agreement) to the temporal domain (agreement with your own past states) gives a principled way to be stable and adaptive at once, and it removes the memory cost of a replay buffer. That is a meaningful shift for both the neural mapping and continual learning communities, since it treats "forgetting" of outdated geometry as desirable rather than a failure.

Real-world applications:

• Service and logistics robots operating in warehouses or offices where shelves, pallets, and furniture are moved between shifts, and where long-term on-board memory is limited.
• Home assistant robots that must update maps as furniture is rearranged or objects are relocated, without keeping large buffers of old sensor data on a consumer-grade computer.
• Autonomous inspection and industrial monitoring, where machinery or inventory changes over time and the map must reflect the current state of the site.
• Search-and-rescue or field robotics, where scenes change during deployment (debris, moved obstacles) and connectivity or storage for replaying past data may be unavailable.

Industry relevance: the reported CPU RSS numbers speak directly to deployability on robot computers, where 4165.1 MB for full replay is a heavy burden compared with TACO's 1864.5 MB. Replay-free operation also reduces privacy and storage concerns, since raw RGB-D frames need not be retained. The released code at https://iconlab.negarmehr.com/TACO lowers the barrier to adoption.

Future Directions

• Extending consensus beyond the immediate predecessor. The paper's derivation simplifies to the current model and its immediate predecessor; the general formulation references a set of snapshots {Theta_{t-k}}. How to weight and combine many older snapshots is left open.
• Tuning the stability-adaptation trade-off. Penalty strength rho and the masking threshold beta are user-picked hyperparameters; making them adaptive or scene-dependent could improve robustness across environments.
• Generalization beyond Co-SLAM. Whether temporal consensus works with other neural implicit backbones, and with appearance modeling to close the color-quality gap observed against replay-based Co-SLAM on Gibson, is untested here.
• Long-horizon and large-scale deployment. Continued validation over very long robot runs, larger environments, and repeated heterogeneous changes would test whether consensus constraints remain well-behaved as history accumulates.

Target Audience

Robotics and 3D vision researchers working on neural implicit mapping, online dense SLAM, and continual learning, particularly those interested in deploying such systems under real memory and computation constraints. It is also relevant to practitioners building long-term autonomy stacks who need maps that update with the environment, and to readers interested in applying constrained optimization and consensus methods to learning-based robotic perception.

Authors’ abstract

Neural implicit mapping has emerged as a powerful paradigm for robotic navigation and scene understanding. However, real-world robotic deployment requires continual adaptation to changing environments under strict memory and computation constraints, which existing mapping systems fail to support. Most prior methods rely on replaying historical observations to preserve consistency and assume static scenes. As a result, they cannot adapt to continual learning in dynamic robotic settings. To address these challenges, we propose TACO (TemporAl Consensus Optimization), a replay-free framework for continual neural mapping. We reformulate mapping as a temporal consensus optimization problem, where we treat past model snapshots as temporal neighbors. Intuitively, our approach resembles a model consulting its own past knowledge. We update the current map by enforcing weighted consensus with historical representations. Our method allows reliable past geometry to constrain optimization while permitting unreliable or outdated regions to be revised in response to new observations. TACO achieves a balance between memory efficiency and adaptability without storing or replaying previous data. Through extensive simulated and real-world experiments, we show that TACO robustly adapts to scene changes, and consistently outperforms other continual learning baselines. Code is available at https://iconlab.negarmehr.com/TACO