MVGD-Net: A Novel Motion-aware Video Glass Surface Detection Method

AI summary

Overview

• Research area: Computer vision, specifically Video Glass Surface Detection (VGSD), with connections to segmentation, optical flow estimation, and multimodal (RGB + flow) fusion.
• Technical level: Advanced. The paper assumes familiarity with transformer backbones (Swin), attention mechanisms (self-attention, cross-attention, CBAM), optical flow estimation (RAFT), and standard segmentation losses (BCE, IoU).
• Scope: The paper proposes MVGD-Net, a motion-aware network that exploits motion inconsistency between glass and non-glass regions in video to segment glass surfaces, plus a new 312-video / 19,268-frame dataset (MVGD-D) and experiments against 11 competing methods on two VGSD datasets.

What This Paper Is About

Glass surfaces such as windows, walls and doors are transparent and colorless, which makes them hard for vision systems on robots and drones to detect. Prior work mostly operates on single images, and the first video-based method (VGSD-Net) relies on reflections that can be unreliable because no ground-truth reflection supervision exists. The authors observe that because reflected or transmitted content sits farther from the glass, it moves more slowly than objects in non-glass regions within the same spatial plane, and they build a network and dataset around this motion-inconsistency cue.

Key Contributions

1. MVGD-Net, a novel VGSD network that exploits motion inconsistency from video sequences, built from three modules: the Cross-scale Multimodal Fusion Module (CMFM) for cross-scale multimodal fusion, the History-Guided Attention Module (HGAM) and Temporal Cross Attention Module (TCAM) for temporal context, and the Temporal-Spatial Decoder (TSD) for balancing temporal and spatial feature integration.
2. MVGD-D, a large-scale dataset of 312 real-world video clips containing 19,268 frames, each with manually annotated glass surface masks, plus flow maps produced by RAFT.
3. Extensive experiments showing the proposed method outperforms relevant state-of-the-art methods on both VGSD-D and MVGD-D, with an ablation study isolating the contribution of each module and of the initial glass mask.
4. A formulation that handles special cases such as open doors and windows by using a primary glass mask to filter inconsistent motion cues from obvious non-glass regions, distinguishing the approach from prior mirror-detection work that used motion inconsistency.

Main Findings

• Motion inconsistency is a usable cue: The improvement of ablation model B (BS+RAFT+BF+BD, IoU 75.59) over model A (BS+BD, IoU 74.31) on MVGD-D demonstrates the effectiveness of motion cues in guiding glass surface detection.
• Full model is best on MVGD-D: The complete configuration G (BS+CMFM+TAM+TSD) reaches IoU 82.62, Fβ 89.14, MAE 0.090, BER 0.087 and ACC 0.930, the best values in the ablation table.
• Each module contributes: Comparisons of models C (79.80 IoU), D (78.74 IoU) and E (80.08 IoU) against the full model G show that each proposed module contributes positively to overall performance, and the comparison between D and G indicates CMFM brings a significant improvement.
• The primary mask matters: Ablation entry F ("w/o P", IoU 80.36) shows that the initial glass mask prediction P(N-1) used to refine the optical flow map plays a crucial role in improving performance.
• State-of-the-art results on VGSD-D: The proposed method achieves IoU 86.57, Fβ 92.53, MAE 0.064, BER 0.061 and ACC 0.935, with a reported time of 190.9 ms.
• State-of-the-art results on MVGD-D: The proposed method achieves IoU 82.62, Fβ 89.14, MAE 0.090, BER 0.087 and ACC 0.930.
• Margin over VGSD-Net on VGSD-D: Compared to the second-best method VGSD-Net (IoU 80.72, Fβ 88.57, MAE 0.099, BER 0.096, ACC 0.898), the proposed method improves IoU by 7.20%, Fβ by 4.46%, ACC by 4.12%, and reduces MAE by 35.35% and BER by 36.45%.
• Margin over GhostingNet on MVGD-D: Despite sharing the same backbone, the proposed method surpasses GhostingNet with IoU +3.26%, Fβ +0.44%, MAE −13.46%, BER −7.45% and ACC +1.63%.
• Single-image methods trail video methods: Image-based GSD methods perform worse than video-based GSD methods in most cases because they cannot exploit temporal information, and SOD methods perform worse than GSD methods because glass surfaces typically lack obvious appearances.
• Accuracy alone can mislead: FSNet obtains a higher ACC but a lower IoU on both datasets, because it relies heavily on optical flow and tends to over-detect glass regions in most scenes.
• Dataset statistics: MVGD-D exhibits more diverse glass surface locations (nine distinct spatial patterns) than VGSD-D and avoids center bias; its color contrast values (measured by χ² distance) are mainly distributed within [0.2, 0.6], indicating a more varied and challenging benchmark.

Methodology in Plain English

Three adjacent frames are fed into two parallel processing paths. One path uses RAFT to estimate optical flow maps between adjacent frames, capturing how pixels move; the other path uses a weight-shared Swin transformer backbone to extract RGB features at four scales. Before the flow is trusted, a coarse glass mask is decoded from intermediate features and used to reject motion cues coming from obvious non-glass areas such as open windows or doors.

The RGB and optical-flow features then meet in the Cross-scale Multimodal Fusion Module, a U-shaped arrangement of seven cross-scale cross-attention blocks that progressively downsample (left to right) and then upsample (right to left) the features, with element-wise multiplication pairing features at matching scales to produce spatial features. In parallel, the RGB features pass through two temporal modules: HGAM builds the current frame's key and value from earlier frames (combined by element-wise multiplication and channel-wise concatenation) so previous history refines the current prediction, while TCAM applies standard cross-attention between pairs of frames to capture both short-term motion trends and longer-range consistency.

Because temporal features and spatial features end up with different channel configurations, the Temporal-Spatial Decoder rebalances them using channel attention and spatial attention, gates the concatenated result by splitting it in half and multiplying the halves (borrowed from NAFNet's simple gate), and upsamples and fuses layer by layer to produce the final glass mask. Training supervises both the coarse primary mask and the three frame predictions using BCE loss and IoU loss, with the primary mask weighted by 1/8. The authors report training on an NVIDIA RTX 4090 (24 GB), resizing frames to 384×384, generating all optical-flow inputs with RAFT for every flow-dependent method, and deliberately using no data augmentation because it may disrupt temporal consistency. Optimizer, learning rate, batch size and training duration are not reported in the provided content.

Why This Matters

• Research impact: It moves glass surface detection from single images to video and introduces motion inconsistency as an explicit, supervised signal, providing a new dataset (MVGD-D) and a strong baseline for future VGSD work. The dataset is designed to reduce the center bias and exaggerated color contrast found in the prior VGSD-D dataset.
• Robotics and drone navigation: Detecting glass is a safety-critical perception problem for navigating robots and drones, and these platforms operate on video streams rather than isolated images.
• Depth estimation and 3D reconstruction: Misidentified glass corrupts depth sensing and scene reconstruction, so better glass masks directly improve downstream geometry.
• Autonomous driving and assistance systems: Vehicles and mobile agents encounter windows, glass doors and glass walls where a false negative can mean collision or a false positive can mean unnecessary avoidance.
• Industry relevance: Any product built on visual perception in built environments (service robots, warehouse automation, AR/VR headsets, smart cameras) can adopt the motion-cue framing. The method's reported runtime of 190.9 ms, however, indicates near-real-time rather than high-frame-rate operation on the tested hardware.

Future Directions

• Longer temporal context: The authors state as a limitation that the method takes only three consecutive frames as input, which limits its ability to capture long-term temporal dependencies and causes failures such as missing a top-left glass region in the 4th frame of the first failure scene and under-detecting a top-right glass region in the 4th frame of the second scene.
• Robustness under appearance changes: The reported failures highlight a limitation in maintaining consistent detection when appearance changes, which points toward memory or tracking mechanisms that carry detections forward.
• Hard negatives such as open doors and windows: The authors note that, like existing image-based GSD methods, their method may mis-detect some open doors or windows — specifically glass-like regions enclosed by their frames — as glass regions, so better discrimination of these cases remains open.
• Reducing reliance on optical flow: Because flow-based methods such as FSNet over-detect, and the pipeline depends on RAFT-generated flow at 190.9 ms, future work could explore cheaper or more robust motion representations.

Target Audience

Researchers and graduate students working on segmentation, video understanding, multimodal RGB–flow fusion, or transparent-surface perception; engineers building perception stacks for robots, drones and autonomous vehicles; and practitioners who need a benchmark dataset (MVGD-D) and a strong video baseline for glass surface detection tasks rather than single-image methods.

Authors’ abstract

Glass surface ubiquitous in both daily life and professional environments presents a potential threat to vision-based systems, such as robot and drone navigation. To solve this challenge, most recent studies have shown significant interest in Video Glass Surface Detection (VGSD). We observe that objects in the reflection (or transmission) layer appear farther from the glass surfaces. Consequently, in video motion scenarios, the notable reflected (or transmitted) objects on the glass surface move slower than objects in non-glass regions within the same spatial plane, and this motion inconsistency can effectively reveal the presence of glass surfaces. Based on this observation, we propose a novel network, named MVGD-Net, for detecting glass surfaces in videos by leveraging motion inconsistency cues. Our MVGD-Net features three novel modules: the Cross-scale Multimodal Fusion Module (CMFM) that integrates extracted spatial features and estimated optical flow maps, the History Guided Attention Module (HGAM) and Temporal Cross Attention Module (TCAM), both of which further enhances temporal features. A Temporal-Spatial Decoder (TSD) is also introduced to fuse the spatial and temporal features for generating the glass region mask. Furthermore, for learning our network, we also propose a large-scale dataset, which comprises 312 diverse glass scenarios with a total of 19,268 frames. Extensive experiments demonstrate that our MVGD-Net outperforms relevant state-of-the-art methods.