Fin3R: Fine-tuning Feed-forward 3D Reconstruction Models via Monocular Knowledge Distillation

AI summary

Overview

• Research area: Computer vision, 3D reconstruction, feed-forward geometry models, monocular knowledge distillation, parameter-efficient fine-tuning.
• Technical level: Advanced.
• Scope: The paper introduces Fin3R, a single lightweight fine-tuning stage that distills monocular geometry knowledge from a teacher model into the encoders of four feed-forward 3D reconstruction models (DUSt3R, MASt3R, CUT3R, and VGGT) while leaving their decoders frozen, and evaluates the resulting models on monocular depth, two-view and multi-view pose, video depth, and pointmap regression benchmarks.

What This Paper Is About

Feed-forward 3D reconstruction models such as DUSt3R, MASt3R, CUT3R, and VGGT regress pointmaps for all input images in a single network pass, but their depth and pointmap outputs remain coarse: fine structures are over-smoothed, object boundaries are blurred, and transparent or glossy surfaces are reconstructed inaccurately. The authors attribute this to two causes, namely scarce high-fidelity depth and pose supervision in existing datasets, and geometric misalignment that accumulates during multi-view pointmap regression over long sequences. Fin3R addresses both by freezing the decoder and fine-tuning only the image encoder, using a custom re-normalization LoRA adapter, so that the encoder absorbs fine geometric detail distilled from a strong monocular teacher on large unlabeled data without losing multi-view matching ability.

Key Contributions

1. A general encoder-only distillation strategy that improves local geometric detail and robustness in feed-forward 3D reconstruction models, applied identically across four architectures: DUSt3R's pairwise prediction with relative depth, MASt3R's pairwise prediction with metric depth, CUT3R's recurrent network, and VGGT's parallel transformer.
2. A feature-shift mitigation approach that combines a customized re-normalization LoRA adapter with multi-view data replay to reduce distribution shift over long sequences, motivated by the observed increase in encoder feature norms caused by single-view distillation.
3. A comprehensive evaluation across DUSt3R, MASt3R, CUT3R, and VGGT showing improved depth fidelity and correspondence accuracy while preserving or slightly improving global multi-view performance.
4. Efficiency: the method adds only the tiny LoRA weights, leaving test-time memory and latency virtually unchanged, and the same implementation works for relative and metric depth models.

Main Findings

• Monocular depth improves across the board: On the averaged scale-invariant relative depth benchmark spanning NYUv2, KITTI, ETH3D, iBims-1, DDAD, DIODE, and HAMMER, DUSt3R improves from Rel 7.03 / δ1 92.3 to 5.58 / 94.8, CUT3R from 6.46 / 92.9 to 5.59 / 94.7, and VGGT from 5.77 / 94.0 to 4.29 / 96.7. The teacher model MoGe scores 4.14 / 96.9, so fine-tuned VGGT performs almost as well as the state-of-the-art expert model.
• Metric depth also improves: MASt3R's averaged metric depth Rel / δ1 improves from 49.62 / 30.2 to 27.60 / 64.3, while the teacher MoGe-2 scores 14.82 / 78.6. On NYUv2 specifically, MASt3R+Ours Rel is slightly higher than the baseline (11.71 versus 10.79), while the other six datasets improve substantially.
• Relative pose estimation on ScanNet1500: AUC@5 / AUC@10 / AUC@20 improves from 31.61 / 53.77 / 70.99 to 33.73 / 55.67 / 72.66 for DUSt3R, from 37.60 / 59.96 / 76.24 to 37.93 / 60.21 / 76.68 for MASt3R, and from 28.40 / 47.36 / 61.51 to 35.21 / 56.70 / 72.80 for VGGT. The fine-tuned VGGT outperforms Reloc3r (34.79 / 58.37 / 75.56) at the 5° threshold, despite Reloc3r being designed only for pose regression.
• Multi-view pose on RealEstate10k: RRA@5 / RTA@5 / AUC@30 improves from 94.01 / 42.39 / 62.40 to 95.41 / 47.07 / 64.81 for DUSt3R, from 94.89 / 52.21 / 73.45 to 95.02 / 53.74 / 73.87 for MASt3R, from 96.66 / 61.66 / 78.95 to 96.99 / 62.15 / 79.13 for CUT3R, and from 95.28 / 53.14 / 74.18 to 96.27 / 56.54 / 75.35 for VGGT.
• Video depth estimation preserved and improved: CUT3R improves on T&T (Rel 0.209 to 0.180, δ1 69.5 to 76.2), KITTI (0.123 to 0.112, 87.4 to 89.8), Sintel (0.428 to 0.406, 47.4 to 58.4), and Bonn (0.077 to 0.062, 93.9 to 96.8), with ETH3D roughly flat (0.126 to 0.130, 83.1 to 82.8). VGGT improves on ETH3D (0.044 to 0.041, 97.9 to 99.2), T&T (0.137 to 0.115, 85.3 to 88.0), Sintel (0.301 to 0.252, 68.4 to 72.7), and Bonn (0.052 to 0.048, 97.3 to 97.5), with KITTI roughly flat (0.072 to 0.069, 96.5 to 96.6).
• Pointmap regression improves: On 7-Scenes and NRGBD, models with the distillation method consistently achieve lower accuracy and completeness errors and improved normal consistency. On DTU, VGGT+Ours reaches Acc mean 0.948 / median 0.520 and Comp mean 1.879 / median 0.905 versus VGGT's 1.187 / 0.715 and 2.229 / 1.309; on ETH3D it reaches Acc 0.209 / 0.112 and Comp 0.170 / 0.085 versus VGGT's 0.290 / 0.196 and 0.371 / 0.230. The concurrent model Pi3 scores 1.151 / 0.622 and 1.793 / 0.629 on DTU, and 0.194 / 0.130 and 0.220 / 0.135 on ETH3D.
• Feature norm drift is real and measurable: Average L2 norms of encoder patch tokens are 9.61 for VGGT, 10.53 for LoRA only, 10.34 for LoRA plus replay, and 9.73 for the full method with re-normalization LoRA and multi-view data replay.
• Ablation confirms each component: Distillation ablation moves VGGT from Rel 5.68 / δ1 94.1 / Acc 0.017 (no label supervision, no teacher, no SA-1B) to 5.21 / 95.0 / 0.014 (depth labels), to 5.00 / 95.3 / 0.013 (monocular teacher), to 4.35 / 96.3 / 0.012 (monocular teacher plus SA-1B). Fine-tuning strategy ablation on ScanNet relative pose gives VGGT AUC@5 / AUC@10 / AUC@20 of 28.40 / 47.36 / 61.51, decoder-full 28.42 / 51.59 / 67.30, encoder-full 32.06 / 52.29 / 68.04, encoder-and-decoder-full 26.35 / 45.90 / 60.02, encoder LoRA 32.96 / 54.21 / 70.40, and encoder LoRA plus re-normalization 35.21 / 56.70 / 72.80.
• Cross-head generalization: Although training distills only the depth head of VGGT with pseudo-labels, the pointmap head shows similar improvements, indicating that a robustly trained encoder benefits downstream heads without direct supervision.
• Confidence calibration: The paper reports that VGGT often produces blurry geometry with low confidence scores, and that after fine-tuning the model becomes more confident and produces sharper geometry with better calibrated confidence.
• CO3Dv2 caveat: Multi-view pose on CO3Dv2 improves DUSt3R (RRA@5 80.49 to 85.75, RTA@5 75.22 to 78.02, AUC@30 81.03 to 82.83) but negatively affects CUT3R (70.83 to 70.89, 64.39 to 63.76, 74.10 to 73.74) and only marginally changes VGGT. The authors suspect CO3Dv2 is used to train the pose head, causing strong memorization.
• Multi-view feature quality preserved: On the Feat2GS benchmark, DUSt3R improves from 19.56 / 0.6504 / 0.3181 to 19.60 / 0.6512 / 0.3181 on geometry, and VGGT_e from 19.66 / 0.6558 / 0.3123 to 19.70 / 0.6561 / 0.3115. The authors note Feat2GS performance typically falls in a narrow range, often PSNR between 19.40 and 19.70.

Methodology in Plain English

The authors start from the observation that these models share a common structure: a shared encoder extracts features from images, and a decoder correlates those features across views. They argue that detail recovery failures originate mainly in the encoder, so they leave the decoder, which handles view matching, untouched and enrich only the encoder.

Their teacher is the monocular geometry estimator MoGe, whose pseudo-labels are distilled into the encoder using the diverse SA-1B dataset. Because MoGe's depth is affine-invariant, they subtract the shift in the z-component and then apply DUSt3R's normalization. During each epoch they sample 20,000 images from SA-1B, 1,000 from Hypersim, and 1,000 from TartanAir; training runs for 10 epochs on four NVIDIA L20 GPUs over a single day.

A naive attempt at encoder-only distillation improved single-frame accuracy but increased the norms of encoder features. That drift pushed features outside the range the frozen decoder expects, hurting multi-view matching. Combining LoRA with multi-view data replay only partially fixed it. The authors therefore embed customized re-normalization inside each LoRA block: given a weight matrix W and its LoRA update ΔW, they rescale the combined weight as W' = (W + ΔW) · ||W||₂ / ||W + ΔW||₂, so the updated weight keeps the original L2 norm. Both the rank and alpha of LoRA are set to 8.

Training optimizes two losses. The monocular distillation loss aligns predicted depth with the teacher's high-fidelity pseudo-label, weighted by an aleatoric uncertainty term. The pointmap regression loss enforces multi-view matching and is gated by an indicator function that equals 1 only for images belonging to the multi-view dataset, so it does not apply to single-view samples. The overall objective averages the two losses over all N images, with uncertainty terms modeled as in prior work.

Details differ per model. DUSt3R uses 2-view data with distillation supervision

Authors’ abstract

We present Fin3R, a simple, effective, and general fine-tuning method for feed-forward 3D reconstruction models. The family of feed-forward reconstruction model regresses pointmap of all input images to a reference frame coordinate system, along with other auxiliary outputs, in a single forward pass. However, we find that current models struggle with fine geometry and robustness due to (\textit{i}) the scarcity of high-fidelity depth and pose supervision and (\textit{ii}) the inherent geometric misalignment from multi-view pointmap regression. Fin3R jointly tackles two issues with an extra lightweight fine-tuning step. We freeze the decoder, which handles view matching, and fine-tune only the image encoder-the component dedicated to feature extraction. The encoder is enriched with fine geometric details distilled from a strong monocular teacher model on large, unlabeled datasets, using a custom, lightweight LoRA adapter. We validate our method on a wide range of models, including DUSt3R, MASt3R, CUT3R, and VGGT. The fine-tuned models consistently deliver sharper boundaries, recover complex structures, and achieve higher geometric accuracy in both single- and multi-view settings, while adding only the tiny LoRA weights, which leave test-time memory and latency virtually unchanged. Project page: \href{http://visual-ai.github.io/fin3r}{https://visual-ai.github.io/fin3r}