LiteMVS: Efficient Multi-View Stereo with Foundation Distillation and Expert Aggregation

AI summary

Overview

Research area: Computer Vision — multi-view stereo (MVS) depth estimation, knowledge distillation from vision foundation models, and embodied/robotic perception.

Technical level: Intermediate. The paper assumes familiarity with cost volumes, plane-sweep warping, and Mixture-of-Experts layers, but its high-level argument (combine cheap geometry with strong learned priors) is accessible.

Scope: The paper introduces LiteMVS, a lightweight multi-view depth model that injects semantic descriptors and foundation-model geometric priors into a plane-sweep cost volume with Mixture-of-Experts aggregation, and evaluates it on ScanNetv2, 7-Scenes, LIBERO, and RoboTwin 2.0.

What This Paper Is About

Multi-view stereo methods match pixels across calibrated images to estimate depth, but this matching breaks down on textureless surfaces, repetitive patterns, and object boundaries. Meanwhile, large monocular foundation models (e.g., Depth Anything V2, Segment Anything) understand scene structure well but lack multi-view geometric constraints. LiteMVS's goal is to transfer the monocular knowledge of those large models into a small, fast multi-view model so it can produce accurate depth and 3D reconstruction at real-time speeds — and then check whether the resulting representations also help a robot manipulate objects.

Key Contributions

1. Semantic-aware 4D feature volume. Features from a lightweight segmentation encoder (MobileSAM) are broadcast along the depth dimension and concatenated with the warped multi-view feature volume and geometric metadata, giving the cost volume high-level semantic guidance around object boundaries and textureless regions. The segmentation path is auxiliary — no segmentation head is trained, and the cues are optionally available at inference.
2. Mixture-of-Experts cost aggregation. Instead of one shared MLP applied uniformly across all depth hypotheses (as in SimpleRecon), a gating network predicts soft weights over several expert MLPs whose outputs are combined into the aggregated 3D cost, letting experts specialize in different depth regimes.
3. Pseudo-label distillation from vision foundation models. Relative depth pseudo-labels from Depth Anything V2 and surface normal pseudo-labels from StableNormal supervise training through scale-invariant and SSIM-plus-gradient losses, transferring geometric priors with no additional inference cost.
4. Validation beyond reconstruction. The learned representations are used as the target visual representation in the SpatialForcing framework on the LIBERO benchmark and compared against SigLIP, DINOv2, and VGGT, plus a comparison against SpatialForcing on RoboTwin 2.0.

Main Findings

• Depth accuracy leads all compared baselines. On ScanNetv2, LiteMVS reaches Abs Diff 0.0702, Abs Rel 0.0311, Sq Rel 0.0097, δ<1.05 of 82.45 and δ<1.25 of 98.52, versus the next-best DoubleTake at 0.0767, 0.0369, 0.0112, 79.94 and 98.35. Baseline numbers are taken from prior work or evaluated per method using their keyframes.
• Cross-dataset generalization without fine-tuning. On 7-Scenes, LiteMVS records Abs Diff 0.0898, Abs Rel 0.0480, Sq Rel 0.0143, δ<1.05 of 69.47 and δ<1.25 of 97.52, with DoubleTake at 0.0985, 0.0534, 0.0156, 64.76 and 97.01. Models trained on ScanNetv2 plus ScanNet++ were applied directly, using the 7-Scenes test split from prior work.
• Reconstruction quality with low latency. In the mesh evaluation, LiteMVS scores Comp 5.37, Acc 4.16, Recall 0.715, F-Score 0.715 at a 75 ms per-frame update latency. DoubleTake gets Acc 4.70, Recall 0.701, F-Score 0.714 at 76 ms; SimpleRecon gets Acc 6.09, Recall 0.658, F-Score 0.671 at 72 ms; VoRTX has the best Comp (4.31) but a 4550 ms latency; TransformerFusion is 326 ms and NeuralRecon 90 ms.
• Semantic descriptors matter most in the ablation. Removing semantic descriptors gives Abs Diff 0.0830 and Abs Rel 0.0412, which the authors describe as roughly a 25% relative-error reduction and over 15% absolute-difference reduction attributable to semantic priors.
• MoE aggregation beats a single shared MLP. Without the MoE MLP, Abs Diff is 0.0741 and Abs Rel 0.0357 (versus 0.0702 and 0.0311 with it), which the authors report as about a 13% absolute-relative-error reduction.
• Distillation improves results at no inference cost. Without the monocular knowledge distillation loss (L_MonoKD), Abs Diff is 0.0726, Abs Rel 0.0338, Sq Rel 0.0104, δ<1.25 98.34; adding it yields 0.0702, 0.0311, 0.0097, 98.52, with visibly sharper boundaries in the qualitative figure.
• Three experts is the best setting tested. With 8 experts: Abs Diff 0.0789, Abs Rel 0.0386, Sq Rel 0.0117, δ<1.25 98.21; with 5: 0.0763, 0.0369, 0.0112, 98.27; with 2: 0.0738, 0.0345, 0.0106, 98.36; with 3 (the chosen setting): 0.0702, 0.0311, 0.0097, 98.52. Adding more experts brings only limited gains, which the authors attribute to expert specialization rather than raw capacity.
• Competitive embodied manipulation at much higher speed. On LIBERO, LiteMVS reaches average success rate 94.1% at 74 ms inference, versus SigLIP 94.0% at 155 ms, DINOv2 94.1% at 186 ms, VGGT without PE 94.7% at 201 ms, and VGGT 96.9% at 201 ms. Per-task, LiteMVS scores 95.0 (Spatial), 94.5 (Object), 93.5 (Goal), 92.1 (Long). On RoboTwin 2.0, it achieves success rates comparable to SpatialForcing in both easy and hard settings.
• Speed-accuracy positioning. Figure 1(b) presents LiteMVS as having the lowest overall error and the fastest inference among the compared state-of-the-art methods.

Methodology in Plain English

LiteMVS starts from the classic plane-sweep recipe: encode every image with a small CNN, hypothesize a set of depth planes, warp source-image features into the reference view for each hypothesis using the known camera poses and intrinsics, and stack the result into a 4D volume laid out over channels, depth, height, and width.

The twist is what gets added to that volume. First, the reference image is also passed through a lightweight segmentation encoder (MobileSAM); those semantic features are copied across the depth dimension and concatenated onto the volume, so matching decisions can be informed by what objects are where, not just by pixel appearance. Second, the volume is compressed to a 3D cost using several expert MLPs rather than one shared MLP, with a small gating network deciding how much each expert contributes at each voxel — the idea being that different depth ranges and surfaces need different aggregation behavior.

Training is supervised primarily by the multi-view loss from SimpleRecon. On top of that, the researchers run large pretrained models offline to produce pseudo-labels: relative depth from Depth Anything V2 and surface normals from StableNormal. The network's predicted depth is compared to the relative-depth pseudo-label with a scale-invariant loss, and normals derived from the predicted depth (via a differentiable depth-to-normal operator) are compared to the normal pseudo-labels with SSIM and gradient terms. Because these large models are only used to make labels, nothing extra runs at test time. Training used ScanNetv2 and ScanNet++ data; details of the implementation are deferred to supplementary material, and λ_rel, λ_norm, dataset sizes, and model parameter counts are not reported in the main content.

Why This Matters

For research, the paper argues that distillation is a viable alternative to simply running a large foundation model inside an MVS pipeline: you can inherit its geometric priors at zero inference overhead. It also broadens the evaluation of an MVS model beyond depth and mesh metrics into robot manipulation, suggesting that geometry-aware lightweight features are useful visual targets for embodied policies, and that they could serve as a foundation for temporally consistent 4D representations.

Real-world applications implied by the framing:

• Robotics manipulation, where the LIBERO and RoboTwin 2.0 experiments show geometry-aware visual features supporting object-centric spatial reasoning.
• Augmented reality, listed as a motivation for real-time depth estimation.
• 3D scene reconstruction from posed images on indoor scenes, as demonstrated on ScanNetv2 and 7-Scenes with a depth-based pipeline.
• Embodied intelligence / interactive agents that need real-time perception and, per the authors, temporally coherent 4D scene representations.

Industry relevance centers on the accuracy-efficiency trade-off: LiteMVS reports better reconstruction than prior lightweight methods at 75 ms per-frame update latency and faster inference than SigLIP, DINOv2, and VGGT in the manipulation comparison, which matters for deployments with tight compute budgets.

Future Directions

• Handling inherently ambiguous geometry. The authors state LiteMVS remains challenged by reflective surfaces, transparent objects, and low-texture regions — the classic failure modes of matching-based stereo.
• Improving downstream transfer. They note that performance of the learned 4D representations on downstream tasks is "not yet optimal," leaving room for improvement in how these features are used by embodied policies.
• Extending to temporally consistent 4D representations. The paper repeatedly frames static geometry as a stepping stone toward spatiotemporal modeling, but does not itself build a temporal model.
• Better MoE design. The team defers further studies on expert homogeneity and expert parameter size to supplementary material, implying open questions about how to organize experts most effectively.

Target Audience

Researchers and engineers working on multi-view stereo, real-time depth estimation, or efficient 3D perception will get the most from this paper, along with practitioners in robotics and embodied AI who care about visual representations that are cheap enough to run online. Readers interested in knowledge distillation from foundation models into small task-specific networks will also find the pseudo-label training scheme relevant. Some background in cost-volume MVS and deep learning is needed to follow the method section.

Authors’ abstract

Real-time 3D perception is crucial for robotics, augmented reality, and embodied intelligence applications. Existing multi-view stereo (MVS) methods primarily rely on geometric correspondences, which often fail in textureless or repetitive regions, while monocular depth models leverage strong image-level priors but lack robust multi-view geometric constraints. More importantly, in robotics and embodied manipulation scenarios, high-quality 3D geometry is not only essential for static reconstruction, but also serves as a critical foundation for learning temporally consistent 4D representations. To obtain visual representations with stronger structural awareness and greater potential for spatiotemporal extension, we present LiteMVS, a lightweight multi-view depth estimation model that integrates plane-sweep geometric reasoning with strong monocular semantic and structural priors. The central idea of LiteMVS is to efficiently inject high-level monocular knowledge, obtained from lightweight segmentation models and large-scale vision foundation models, into a multi-view stereo framework. In particular, LiteMVS enriches the cost volume with semantic descriptors and employs a Mixture-of-Experts (MoE) formulation to enable adaptive geometric aggregation across depth hypotheses. Moreover, geometric priors distilled from vision foundation models further strengthen monocular guidance without increasing inference cost. Through this design, LiteMVS not only improves depth estimation and 3D reconstruction quality in static scenes, but also provides a more reliable geometric foundation for subsequent temporal modeling and 4D representation learning. Experiments on ScanNetv2 and 7-Scenes demonstrate that LiteMVS achieves high-quality depth prediction and 3D reconstruction while maintaining competitive efficiency.