MVSMamba: Multi-View Stereo with State Space Model

AI summary

Overview

• Research area: Computer Vision — learning-based Multi-View Stereo (MVS) and dense 3D reconstruction.
• Technical level: Advanced. The paper assumes familiarity with convolutional feature pyramids, Transformer attention, cost volumes, and State Space Models (SSMs).
• Scope: The paper introduces MVSMamba, the authors' claim to the first Mamba-based MVS network, built around a "Dynamic Mamba module" with a reference-centered dynamic scanning strategy, and evaluates it on DTU, Tanks-and-Temples, and (in the appendix) ETH3D.

What This Paper Is About

Learning-based MVS estimates a depth map for a reference image by matching it against several source images, so the quality of feature matching drives the quality of the reconstruction. Transformers capture the long-range dependencies that CNNs miss, but their quadratic complexity makes the performance/efficiency trade-off difficult. The paper asks whether a Mamba-based state space model, which models long-range dependencies with linear complexity, can deliver global multi-view feature aggregation more cheaply.

Key Contributions

1. **MVSMamba, a first-of-its

Authors’ abstract

Robust feature representations are essential for learning-based Multi-View Stereo (MVS), which relies on accurate feature matching. Recent MVS methods leverage Transformers to capture long-range dependencies based on local features extracted by conventional feature pyramid networks. However, the quadratic complexity of Transformer-based MVS methods poses challenges to balance performance and efficiency. Motivated by the global modeling capability and linear complexity of the Mamba architecture, we propose MVSMamba, the first Mamba-based MVS network. MVSMamba enables efficient global feature aggregation with minimal computational overhead. To fully exploit Mamba's potential in MVS, we propose a Dynamic Mamba module (DM-module) based on a novel reference-centered dynamic scanning strategy, which enables: (1) Efficient intra- and inter-view feature interaction from the reference to source views, (2) Omnidirectional multi-view feature representations, and (3) Multi-scale global feature aggregation. Extensive experimental results demonstrate MVSMamba outperforms state-of-the-art MVS methods on the DTU dataset and the Tanks-and-Temples benchmark with both superior performance and efficiency. The source code is available at https://github.com/JianfeiJ/MVSMamba.