CompMVR: Compression-Aware Multi-View Restoration Using Diffusion Models for Geometrically Consistent 3D Reconstruction

AI summary

Overview

Research area: Computer vision — lossy image/video compression, diffusion-based image restoration, and multi-view 3D reconstruction (novel view synthesis, camera pose estimation, feature matching).

Technical level: Advanced. The paper assumes familiarity with latent diffusion models, LoRA fine-tuning, multi-view attention, Gaussian Splatting, and structure-from-motion style geometry evaluation.

Scope: The paper defines compression-induced cross-view geometric inconsistency as a distinct failure mode, and proposes CompMVR, a two-stage compression-aware multi-view diffusion restoration framework evaluated across four codecs and two unseen quantization levels.

What This Paper Is About

Real-world multi-view captures are usually stored or transmitted with lossy codecs such as JPEG, H.265/HEVC, or H.266/VVC, but existing restoration methods are built for single images or video frames and only optimize per-image visual quality. The authors show that this view-by-view approach leaves geometry unreliable: compression artifacts differ per view, so feature matches, dense correspondences, and camera poses degrade even when images look clean. The goal is a restoration method that jointly refines multiple compressed views so downstream 3D reconstruction becomes more accurate, not just prettier.

Key Contributions

1. A new problem formulation and benchmark. The authors frame compression-induced cross-view geometric inconsistency as a distinct failure mode and build a codec-synthesized multi-view benchmark with clean/compressed pairs, codec labels, and QP annotations across H.264/AVC, H.265/HEVC, VP9, and AV1.
2. CompMVR, a two-stage compression-aware diffusion framework. Stage 1 learns a compression-aware latent using codec classification and QP regression supervision; Stage 2 restores N = 4 views jointly with a diffusion model conditioned on that prior.
3. Spatially varying artifact modeling. A compression artifact estimator (CAE), an 11-layer fully convolutional network with 0.375M parameters, predicts per-pixel residual maps from the compressed input alone — no high-quality reference needed at inference — and injects them into the diffusion latent.
4. Explicit cross-view geometric supervision. A confidence-weighted correspondence loss built on a pre-trained VGGT (used only during training), combined with 3D self-attention layers trained from scratch with zero initialization rather than initialized from Stable Video Diffusion weights as SIR-Diff does.

Main Findings

• Compression measurably breaks view matching. On RealEstate10K with HEVC at QP42, uncompressed ground truth yields 2,001 mean matches and AUC@5 of 24.55, while the compressed input drops to 1,655 matches and 22.59 AUC@5. CompMVR restores 1,659 matches and 24.34 AUC@5, above the fine-tuned CODiff (1,625 / 23.36), DiQP (1,643 / 23.45), and SIR-Diff (1,627 / 23.09) baselines.
• Novel view synthesis improves substantially. Training 3D Gaussian Splatting on restored Free-dataset inputs, CompMVR is best under every codec and QP setting. Against the second-best baseline it gains 2.02 dB PSNR and 0.071 SSIM while cutting LPIPS by 0.134. The uncompressed upper reference is 28.30 PSNR, 0.885 SSIM, 0.112 LPIPS.
• Geometry tasks improve consistently. CompMVR achieves the best AUC@15/30 for VGGT camera pose estimation across all codecs and compression levels, and the highest average AUC@5 for view matching. Under a matched fine-tuning protocol it reaches the lowest MEt3R score of 0.132 (CODiff 0.135, DiQP 0.136, SIR-Diff 0.141).
• Trade-offs with perceptual baselines exist. In the matched fine-tuning comparison (Table 5), DiQP has higher PSNR (27.23 vs. 23.89) but far lower CLIPIQA (0.393 vs. 0.673) and worse MEt3R (0.136 vs. 0.132). CompMVR leads on SSIM (0.897), NIQE (3.26), and CLIPIQA (0.673). The paper also notes that SIR-Diff achieves a lower point-map Chamfer distance, with details deferred to the supplementary material.
• Generalization to an unseen codec holds. On VVC/H.266, which is not in the training set, CompMVR improves over CODiff on PSNR (23.6228 vs. 23.4970), NIQE (3.2610 vs. 3.3412), and CLIPIQA (0.6981 vs. 0.6917).
• The correspondence loss drives the geometry gains. Removing it reduces inlier ratio from 0.805 to 0.801, raises Sampson error from 0.078 to 0.082, drops AUC@5 from 0.531 to 0.516, and raises MEt3R from 0.132 to 0.138.
• The learned prior helps but is not dominant. Replacing the Stage 1 prior with a null embedding worsens Sampson error from 0.078 to 0.081 and MEt3R from 0.1357 to 0.1361. Random conditioning degrades more sharply: inlier ratio 0.7747 versus 0.8047, Sampson 0.0960 versus 0.0806, AUC@5 0.5164 versus 0.5275, MEt3R 0.1494 versus 0.1404.
• Codec and QP supervision are complementary. Training with only the codec loss gives 23.49 PSNR, only the QP loss gives 23.77, and both together give 23.89. t-SNE plots show codec loss separates codec types while QP loss organizes representations by compression severity.
• Cost is higher than baselines. CompMVR uses 276.171M trainable parameters, 1.073 s forward time, and 11.01 GB peak GPU memory, versus SIR-Diff at 815.545M / 0.826 s / 2.48 GB and DiQP at 79.373M / 0.422 s / 1.02 GB.

Methodology in Plain English

The pipeline has two training stages that share a latent representation of "how badly and in what way is this image compressed."

Stage 1 trains an encoder to produce a latent that captures the compression regime. A small head predicts the codec type (trained with cross-entropy) and the quantization parameter (trained with L1 regression). A decoder reconstructs the image at the same time, so the latent must retain content as well as compression context. A key design point: codec and QP labels are used only during training. At inference the model infers the compression prior directly from the input.

Stage 2 performs the actual restoration with a Stable Diffusion 2.1 backbone fine-tuned via LoRA, processing four views jointly rather than one at a time. Two mechanisms inject compression awareness. First, the Stage 1 latent enters through cross-attention and is fused with the UNet input. Second, a lightweight estimator predicts a per-pixel artifact map. The authors motivate this with a gradient-energy observation: regions with dense texture and edges accumulate larger compression residuals than smooth regions at the same QP, so distortion depends on local content, not just a global setting. The artifact map is supervised with an asymmetric loss (which penalizes under-estimation more than over-estimation, with α = 0.3) plus a cosine similarity term for spatial alignment, then projected and concatenated onto the diffusion latent.

For cross-view consistency, the model uses two complementary strategies. Implicit consistency comes from 3D self-attention and Spatial-3D ResNet layers that let views exchange features. Explicit consistency comes from a correspondence loss: VGGT is run on high-quality reference images during training to establish pixel matches via 3D nearest-neighbor matching in world coordinates, filtered by confidence and geometry. Restored pixels are then compared against matched ground-truth pixels with confidence weighting, so unreliable matches under severe compression are down-weighted instead of being enforced. VGGT is discarded at inference. The total objective combines the artifact-estimator loss, the correspondence loss, and a perceptual loss made of reconstruction plus DISTS.

Training used 146 scenes from DTU, LLFF, Mip-NeRF 360, and Tanks and Temples (140 for training, 6 for validation), with six QPs (20, 26, 32, 38, 44, 50) and testing at QPs 27 and 42, both unseen. Stage 1 ran 200K iterations at batch size 8; Stage 2 ran 100K iterations at batch size 2 with 4 views, on a single NVIDIA A6000.

Why This Matters

The work reframes restoration as a geometry problem rather than a pixel problem. Most restoration benchmarks reward perceptual quality, which can actively push results away from what a 3D pipeline needs — a smoothed image may look clean while destroying the correspondences a matcher depends on. By showing that compression alone drops AUC@5 from 24.55 to 22.59 on RealEstate10K and that restoring it back to 24.34 changes downstream reconstruction, the paper gives the field a concrete metric and benchmark for a previously implicit assumption.

Real-world applications:

• Consumer 3D capture: phone or drone photo sets stored as JPEG/HEVC before being uploaded to NeRF or Gaussian Splatting pipelines.
• Street-level and real-estate mapping: large compressed image collections (the RealEstate10K setting) used for pose estimation and view synthesis.
• Streaming and cloud XR: where multi-view content is compressed for bandwidth and then reconstructed on the client or in a datacenter.
• Archival and heritage digitization: compressed historical or remote captures that need 3D reconstruction without re-capture.

Industry relevance is direct: codecs like AVC, HEVC, VP9, and AV1 are the default storage and delivery formats, so any 3D reconstruction product operating on real user data already faces this degradation. The demonstrated generalization to VVC/H.266 — a codec not seen in training — matters for deployment across heterogeneous capture devices and delivery stacks.

Future Directions

• Reducing cost and moving to feed-forward inference. The paper explicitly flags that the framework needs additional compute and GPU memory (11.01 GB peak, 1.073 s per forward pass) and is not yet a fully feed-forward pipeline.
• Codec-adaptive priors. The authors note that different codecs produce different artifact structures and that priors tailored to coding tools such as AV1 — which differ from HEVC — are needed for stronger cross-codec generalization.
• Validating on untested 3D tasks. The paper states that direct applicability to 3D tasks beyond those evaluated requires further validation.
• Better handling of "plausible but inaccurate" detail. The supplementary material reportedly analyzes textured, edge, and smooth regions to examine where restoration produces realistic-looking but geometrically wrong content, which is the central risk of using generative priors for measurement tasks.

Target Audience

Researchers and engineers working on 3D reconstruction from casual captures, computational photography under compression, and diffusion-based restoration. It is most useful for those who already know diffusion backbones and multi-view geometry and want a concrete, benchmarked argument for why view-wise restoration is insufficient. Practitioners building photogrammetry, Gaussian Splatting, or neural rendering products on compressed user data will find the benchmark and the complexity table practically relevant; readers new to diffusion models or structure-from-motion will find the method sections dense.

Authors’ abstract

Real-world 3D reconstruction pipelines often rely on multi-view images degraded by lossy compression. While compression artifacts have been extensively studied in single-image or video restoration, their impact on multi-view geometry remains largely overlooked. In this work, we identify compression-induced cross-view geometric inconsistency as a critical failure mode, where view-dependent artifacts disrupt feature matching, dense correspondence, camera parameter estimation, and downstream 3D reconstruction tasks such as novel view synthesis. To address this problem, we propose CompMVR, a compression-aware multi-view diffusion restoration framework that jointly refines compressed views while preserving cross-view consistency, and construct compressed datasets for systematic evaluation. Unlike conventional restoration methods that optimize each image independently for 2D perceptual quality, CompMVR leverages learned compression priors and cross-view correspondence to recover visual details while improving geometric reliability across views. Experiments across diverse datasets, codecs, and compression levels demonstrate gains in restoration quality and downstream 3D tasks, including novel-view synthesis, camera pose estimation, and view matching. These results establish compressed multi-view restoration as a distinct problem and highlight its importance for 3D reconstruction under lossy compression. Code and data for this paper are available at https://mchaewon.github.io/CompMVR-project-page/.