Changes in Real Time: Online Scene Change Detection with Multi-View Fusion

AI summary

Overview

Research area: Computer Vision, specifically Scene Change Detection (SCD) for robotics, with connections to 3D scene representation (3D Gaussian Splatting) and multi-view geometry.

Technical level: Advanced. The paper assumes familiarity with 3D Gaussian Splatting (3DGS), Neural Radiance Fields (NeRF), PnP pose estimation, SfM, and self-supervised losses.

Scope: This paper presents the first online, pose-agnostic, label-free, multi-view scene change detection method, achieving state-of-the-art performance at over 10 FPS while also introducing a fast change-guided 3DGS representation update.

What This Paper Is About

Robots revisiting a scene after time has passed need to detect which changes are real (e.g., an object moved) versus which are irrelevant (e.g., shadows or lighting shifts), often from viewpoints that do not match the original capture. Existing online SCD methods run in real time but are far less accurate than offline methods that have access to all pre- and post-change data before making predictions. This paper closes that gap by fusing change cues across multiple viewpoints using a self-supervised loss and a 3DGS-based change representation, detecting changes on the fly while beating even the best offline baselines.

Key Contributions

1. An online, pose-agnostic, label-free, multi-view SCD approach that infers change masks from unposed monocular images in real time (over 10 FPS), with no assumption of aligned trajectories or depth input.
2. A novel self-supervised fusion loss (L_SSF) that jointly integrates pixel-level and feature-level change cues across all observed viewpoints, replacing the hard-thresholding and intersection heuristics of prior work (MV3DCD) and achieving state-of-the-art performance in both online and offline settings.
3. An ultra-light PnP-based pose estimation module using XFeat descriptors and a GPU-parallel miniBA refinement, achieving constant-time O(1) pose estimation without drift accumulation.
4. A change-guided selective reconstruction and fusion strategy that updates the 3DGS representation in seconds by only reconstructing changed regions, reusing reference primitives elsewhere, and applying a lightweight global optimization.

Main Findings

• Offline SCD performance: On the PASLCD dataset, the proposed method achieves mIoU 0.552 and F1 0.694, versus MV3DCD's mIoU 0.478 / F1 0.628, an approximately 15% mIoU improvement over the strongest offline competitor while running nearly 3× faster (156s vs. 479s).
• Online SCD performance: In the online setting, the method reaches mIoU 0.486 / F1 0.638 at 11.2 FPS, roughly 2× higher mIoU than the strongest online competitor, CS+CYWS2D (mIoU 0.243 / F1 0.360 at 8.2 FPS). It surpasses the best offline methods even while operating online.
• Runtime breakdown: Total per-frame latency is 89.60 ms, with 58.17 ms (64.9%) spent on multi-view change cue fusion, 16.47 ms (18.4%) on pose estimation, and 11.50 ms (12.8%) on reference image retrieval. Change mask inference itself costs only 0.49 ms (0.6%).
• Speed–accuracy trade-off: The method operates between 11 and 20 FPS with only a 3.6% relative drop in F1 score, enabled by R_change acting as a persistent memory of previously learned changes.
• Ablation (change detection): Full model achieves mIoU 0.486 / F1 0.638. Removing the L1 term drops to 0.320 / 0.464; removing D-SSIM drops to 0.447 / 0.620. Using only pixel-level or only feature-level cues fails to converge. Replacing the self-supervised loss with MV3DCD's thresholding and heuristic fusion degrades to mIoU 0.350 / F1 0.495.
• Scene representation update (PASLCD): The method achieves PSNR 23.70, SSIM 0.7868, LPIPS 0.2491 in 42s, compared to 3DGS at 22.21 / 0.7558 / 0.2426 in 550s and 3DGS-LM at 22.26 / 0.7562 / 0.2422 in 340s — about 8× faster than 3DGS-LM and 13× faster than 3DGS.
• Scene representation update (CL-Splats): PSNR 30.54, SSIM 0.9356, LPIPS 0.1256 in 39s, versus 3DGS at 30.31 / 0.9319 / 0.1178 in 364s.
• Update ablation: Selective reconstruction alone (28s) introduces artifacts (PSNR 19.89), and global optimization alone (145s) reaches PSNR 22.64. Combined, the method reaches PSNR 23.70 in 36s.
• Qualitative advantage: Compared to MV3DCD, the method produces cleaner masks with fewer false negatives and false positives, capturing subtle appearance changes (e.g., red-to-blue T-shaped object, blue-to-black bench) that thresholding misses.
• Hardware and resolution: All experiments run on a GeForce RTX4090 at 1008×560 resolution.

Methodology in Plain English

The researchers split the problem into two phases. First, offline, they build a high-fidelity reference scene using 3D Gaussian Splatting (Speedy-Splat, following standard SfM for pose estimation) from the pre-change images. Then, online, they process incoming post-change frames one at a time.

For each new frame, they find approximate matches against a small subset (top-4) of reference images using XFeat keypoint descriptors, estimate the camera pose via PnP with RANSAC, and refine it with a GPU-parallel miniBA. This makes pose estimation fast and independent of drift.

With the pose known, they render the reference scene from the same viewpoint and compare the two images. They compute two types of change cues: pixel-level differences (a combination of L1 and D-SSIM with lambda = 0.2) that catch fine-grained appearance changes, and feature-level differences using SAM2-Tiny dense features that are more robust to distractors like shadows. These cues are simply summed into a combined map.

To aggregate information across viewpoints, they maintain a separate change representation (R_change) initialized from the reference 3DGS but with color parameters discarded and a learnable change parameter c added to each primitive. For each new frame, R_change is optimized for 16 iterations using a self-supervised loss that encourages predicted change values to be high where change cues are strong, with a regularization term to prevent the trivial "everything changed" solution. At each iteration, a past frame is sampled randomly, biased toward the most recent one with 1/3 probability. Rendering R_change yields the change mask.

Once all frames are processed, they refine R_change, then perform the scene update: they mask the inference images with the refined change masks, reconstruct only the changed regions as R_change*, fuse with reference primitives (excluding those contributing to changed pixels), and run a fast global optimization restricted to change-related primitives. This achieves rendering speeds exceeding 400 FPS for selective reconstruction and complete updates in seconds.

Why This Matters

This work is the first to combine online operation, pose-agnostic inference, label-free detection, multi-view consistency, and real-time throughput in a single SCD system, and it demonstrates that the online setting can match or exceed offline accuracy. It also shows that maintaining a 3D scene representation can be done incrementally and cheaply rather than re-optimized from scratch.

Real-world applications:

• Environmental monitoring, where repeated site visits detect vegetation, terrain, or structure changes.
• Infrastructure inspection, including detecting defects, corrosion, or component displacement across inspection rounds.
• Damage assessment after events such as storms or earthquakes, where rapid on-the-fly change reporting is essential.
• Long-term robotic asset management in industrial facilities, where robots autonomously re-traverse sites and need to flag true changes.

Industry relevance: The combination of label-free operation (no costly annotation), real-time performance (over 10 FPS), and few-second scene updates matters for any deployment where robots repeatedly visit a site autonomously — for example, in inspection, mining, agriculture, and asset monitoring. The paper is supported by the Australian Research Council Research Hub in Intelligent Robotic Systems for Real-Time Asset Management (ARIAM) and Abyss Solutions, indicating direct industrial interest in these use cases.

Future Directions

• Developing richer and more complementary change cues beyond pixel-level (L1 + D-SSIM) and feature-level (SAM2-Tiny) signals, which the authors suggest could improve both online and offline SCD performance.
• Addressing limitations of the XFeat features for pose estimation (the paper defers these to supplementary material), suggesting room for stronger lightweight descriptors.
• Extending the change-guided update strategy to handle more complex or long-term scene evolution patterns, given that CL-Splats scenes include only single object-level changes.
• Improving robustness to global illumination and appearance shifts across repeated inspections, which the current global optimization step only partially accounts for.

Target Audience

Robotics and computer vision researchers working on scene change detection, neural rendering, SLAM, or 3D Gaussian Splatting; engineers building inspection or monitoring robots that revisit sites; and practitioners interested in label-free, real-time perception pipelines. Readers should have some familiarity with 3D scene representations and multi-view geometry to follow the methodological details.

Authors’ abstract

Online Scene Change Detection (SCD) is an extremely challenging problem that requires an agent to detect relevant changes on the fly while observing the scene from unconstrained viewpoints. Existing online SCD methods are significantly less accurate than offline approaches. We present the first online SCD approach that is pose-agnostic, label-free, and ensures multi-view consistency, while operating at over 10 FPS and achieving new state-of-the-art performance, surpassing even the best offline approaches. Our method introduces a new self-supervised fusion loss to infer scene changes from multiple cues and observations, PnP-based fast pose estimation against the reference scene, and a fast change-guided update strategy for the 3D Gaussian Splatting scene representation. Extensive experiments on complex real-world datasets demonstrate that our approach outperforms both online and offline baselines.