Failure or Drift? Evaluating Monocular SLAM under Synthetic and Real-World Corruptions

AI summary

Overview

Research area: Visual SLAM (simultaneous localization and mapping) robustness evaluation, specifically monocular camera tracking under adverse visual conditions. The paper sits at the intersection of 3D computer vision, robotics benchmarking, and distribution-shift robustness.

Technical level: Intermediate. The paper explains its three tracking systems and its evaluation protocol in accessible terms, but readers benefit from some familiarity with trajectory error metrics (APE, RPE) and the distinction between visual odometry and full SLAM backends.

Scope in one sentence: The paper tests whether synthetic image corruptions are a valid stand-in for real adverse weather and lighting when benchmarking monocular SLAM, and finds that the choice of corruption changes which tracker looks best.

What This Paper Is About

Monocular SLAM systems are usually judged on clean benchmark trajectories, even though they are deployed in rain, fog, low light, blur, and sensor noise. Synthetic corruptions are a convenient way to study such conditions in a controlled way, but it is unclear whether conclusions drawn from a convenient image transformation match conclusions drawn from the real condition it is meant to imitate. The authors compare three monocular systems under synthetic image-space, geometry-aware, and compound corruptions, then check which of those conclusions survive on real 4Seasons recordings.

Key Contributions

1. A controlled evaluation of three monocular SLAM paradigms (ORB-SLAM2, DPVO, DROID-SLAM) across image-space, geometry-aware, compound, and natural adverse conditions, using 115 corrupted variants of KITTI sequence 00 plus four 4Seasons sequences.
2. A failure-aware evaluation protocol that reports trajectory validity first and only then reports APE and RPE conditioned on a valid output, so that a method cannot score well by returning nothing, and so that comparisons do not conflate tracking loss with differences in exported-pose density (for example, ORB-SLAM2 exporting keyframes).
3. Evidence that corruption fidelity changes tracker selection: the apparent ranking of DPVO versus DROID-SLAM flips depending on whether the corruption is image-space, geometry-aware, or compound.
4. A direct synthetic-to-real comparison showing which proxy conclusions transfer: rain and winter-like conditions preserve the compound-proxy ordering, while an illumination proxy does not.

Main Findings

• Two distinct failure modes. On clean KITTI sequences 00–10, ORB-SLAM2, DPVO, and DROID-SLAM each return an evaluable trajectory on all 11 sequences. Under synthetic corruption, DPVO and DROID-SLAM return an evaluable trajectory for every archived synthetic attempt, while ORB-SLAM2 is valid in only 37 of 63 image-space attempts, 15 of 23 geometry-aware attempts, and 10 of 15 compound attempts.

• Explicit failure versus silent drift. ORB-SLAM2 frequently stops returning an evaluable trajectory, whereas the learned systems keep producing poses whose aligned APE frequently exceeds 100 m under geometry-aware and compound effects. The image-space archive contains 25 empty ORB-SLAM2 trajectories and one synchronization failure; Gaussian noise, impulse noise, and snow produce no evaluable ORB-SLAM2 trajectory at any recorded severity, while brightness and JPEG compression are substantially less destructive.

• Clean data do not give a stable accuracy ordering. ORB-SLAM2 has the lowest median conditional APE on clean KITTI at 8.2 m, but reaches 518.0 m on sequence 01. DPVO and DROID-SLAM have median APEs of 64.9 m and 82.0 m respectively, and swap which is better depending on the route (DPVO better on sequence 01, DROID-SLAM better on sequence 08).

• Corruption type reverses the preferred learned tracker. Among the 43 paired image-space settings with complete learned-method results, DPVO has lower APE in 30. In the geometry-aware branch, DROID-SLAM has lower APE in all 33 paired settings, including every available severity of low light, fog, rain, flash, ISO noise, and focus changes. The compound branch largely preserves that ordering: DROID-SLAM is better in 14 of 15 settings, with fog with contrast at severity 5 the only reversal.

• No method-independent mapping across proxy layers. For DPVO, low light and brightness with contrast have identical severity ordering (Spearman ρ = 1.00), while rain and rain with contrast are almost unrelated (ρ = −0.10). DROID-SLAM shows the opposite pattern: its rain curves correlate strongly (ρ = 0.80) while its low-illumination curves do not (ρ = −0.10). Fog is method dependent, with ρ = 0.90 for DPVO and ρ = −0.20 for DROID-SLAM.

• Nominal severity is not a common difficulty scale. Under geometry-aware rain, DROID-SLAM APE decreases from 111.9 m at severity 1 to 87.9 m at severity 5, while DPVO stays between 125.5 m and 129.2 m. The authors interpret this as the corruption changing which image structure remains usable, not as the tracker improving with more rain.

• Two of three real conditions transfer. DROID-SLAM is better by 17.96 m for rain with contrast averaged over severities and by 16.40 m on the real rainy 4Seasons sequence. Fog with contrast also selects DROID-SLAM, and the real winter sequence agrees, with margins of 2.27 m and 5.15 m respectively.

• Illumination is the counterexample. Brightness with contrast favors DROID-SLAM by 5.51 m, but the real cloudy evening favors DPVO by 10.85 m. Notably, DPVO has ρ = 1.00 between the geometry-aware low-light and compound severity curves, so perfect agreement between two synthetic constructions still did not preserve the real ordering.

• Absolute error magnitudes are not comparable across datasets. Because the 4Seasons sequences differ in route, camera, motion, and scene geometry from KITTI, the paper explicitly limits its transfer claim to relative ordering, not numerical equivalence.

Methodology in Plain English

The authors pick three monocular systems that behave differently by design: ORB-SLAM2 (classical handcrafted keypoints with loop closure and relocalization), DPVO (learned patch correspondences with differentiable bundle adjustment in a local window, no loop closure), and DROID-SLAM (dense learned correspondences with global multi-frame optimization). None are fine-tuned on corrupted data, given test-time restoration, or adapted per condition.

They establish a clean baseline on the eleven KITTI odometry sequences with ground truth. The controlled experiment uses KITTI sequence 00 only, so that route, calibration, timestamps, and ground truth stay fixed while only the visual input changes. From that sequence they build 115 corrupted variants at five nominal severity levels: 65 image-space (13 transformations such as brightness, contrast, motion blur, JPEG compression, snow, and Gaussian noise), 35 geometry-aware (seven effects following the 3D Common Corruptions approach: far focus, near focus, flash, fog, ISO noise, low light, rain), and 15 compound (brightness with contrast, fog with contrast, rain with contrast).

For real-world validation they use four 4Seasons neighborhood sequences: neighborhood 2 train as a cloudy-afternoon reference, neighborhood 3 train as rainy afternoon, neighborhood 4 train as winter cloudy morning, and neighborhood 6 train as cloudy evening. These are matched by condition, not paired recordings of the same route.

All runs go through VSLAM-LAB, with every trajectory, log, runtime record, and evaluation output archived (more than 350 executions). ORB-SLAM2 runs on CPU; DPVO and DROID-SLAM use NVIDIA A100 or H100 GPUs. Missing configurations stay missing and are never imputed or converted into failures.

The evaluation is deliberately two-stage. First, a run is marked valid only if it returns a non-empty trajectory that can be associated with ground truth; the valid-run rate is the mean of this binary indicator over archived attempts. Only then are poses associated within 0.01 s and aligned by a similarity transform in Sim(3), and translation RMSE for APE and RPE is computed in the TUM RGB-D style. APE captures accumulated global disagreement; RPE captures local motion inconsistency. Comparisons between the two learned systems use the difference ΔAPE = APE_DPVO − APE_DROID-SLAM, where positive favors DROID-SLAM, and only conditions where both results exist are used.

Why This Matters

Impact on research. The paper argues that synthetic corruption benchmarks for stateful systems like SLAM cannot be validated by visual plausibility alone. Because errors propagate through tracking, optimization, and map state, a corruption that looks like rain can favor a different architecture than rain does. The finding that the preferred tracker flips between image-space and geometry-aware proxies means that a benchmark built only from convenient image transformations can recommend the wrong system for the condition it is meant to represent. The paper also pushes evaluation beyond a single trajectory error number toward reporting availability and trustworthiness separately.

Real-world applications:

• Autonomous driving, where a tracker that keeps emitting confident-looking poses while drifting can pass an incorrect pose into planning or control without triggering the fallback that an explicit tracking loss would cause.
• Mobile robotics and drones, where a system needs a signal for when to relocalize or hand off to another sensor rather than silently accumulating error.
• Augmented reality and online 3D reconstruction, where the same failure modes affect pose stability and the consistency of the reconstructed scene.
• Robustness benchmarking and safety validation for any camera-based perception pipeline deployed in weather, night, or degraded imaging conditions.

Industry relevance. Teams selecting a tracking architecture for a product need to know whether the robustness ranking they measure in a lab stress test holds up in the field. This paper shows that the answer depends on the physical fidelity of the perturbation, and that correspondence confidence weights already predicted by DPVO and DROID-SLAM are learned indirectly through the pose objective, not supervised as calibrated probabilities that the final trajectory is correct. In other words, a deployable system needs uncertainty signals tested against accumulated drift, not just against local residuals.

Future Directions

• Introducing temporally coherent, sensor-aware corruptions and a broader set of matched natural conditions, with map-quality and resource measurements alongside trajectory error.
• Establishing which controlled proxies reproduce specific field failure modes, rather than compressing accuracy, continuity, reconstruction, and efficiency into a single score.
• Extending robustness evaluation to map validity and system validity, not only trajectory validity: reconstruction accuracy and completeness, cross-view geometric consistency, rendering degradation, map growth, runtime stability, and memory consumption.
• Adding synthetic corruptions to more real-world datasets and extending the benchmarking study to more SLAM methods, including the dense reconstruction-coupled systems discussed in the paper.

Target Audience

Robotics and computer vision researchers who benchmark or deploy monocular SLAM; practitioners choosing between feature-based, sparse learned odometry, and dense learned SLAM architectures; and anyone building robustness or safety evaluations for camera-based perception, including those working with common-corruption protocols who want to understand their limits for stateful estimation systems. The paper is also useful to engineers who need to reason about the difference between a tracker that stops and a tracker that quietly drifts.

Authors’ abstract

Visual SLAM is commonly evaluated on clean trajectories, although deployment failures are often caused by adverse weather, illumination, blur, and sensor artifacts. Controlled corruptions are attractive because they isolate such factors, but a synthetic stress test is useful only when it leads to the same engineering conclusion as the condition it is intended to approximate. This work examines that question for monocular SLAM. We evaluate a classical feature-based system and two learned trackers under image-space, geometry-aware, and compound corruptions, and compare their behavior with adverse conditions from 4Seasons. Rather than reducing robustness to a single trajectory error, the evaluation separates explicit tracking failure from drift accumulated by methods that remain active. The results show that learned trackers largely replace catastrophic loss with sustained, and sometimes severe, drift. More importantly, the apparent ordering of the learned systems changes with the physical fidelity of the corruption: structured rain and fog proxies preserve the real-world ordering, whereas a simple illumination proxy does not. Code is available at: https://github.com/abhaythomas/master_thesis_vslamlab_robustness.