Depth-Supervised Fusion Network for Seamless-Free Image Stitching

AI summary

Overview

Research area: Computer vision, specifically image stitching and multi-view image fusion, with supporting use of monocular depth estimation.

Technical level: Advanced. The paper assumes familiarity with homography estimation, mesh-based warping, deep feature correlation, UNet-style decoder architectures, dilated convolutions, reparameterization, and standard stitching evaluation metrics (PSNR, SSIM, SIQE, LPIPS).

Scope: The paper proposes and evaluates a two-branch deep pipeline—depth-aware transformation estimation plus soft-seam multi-view fusion—for stitching image pairs with large parallax, tested on the UDIS-D and IVSD datasets.

What This Paper Is About

When two photographs of the same scene are taken from different positions, objects at different distances shift by different amounts, producing parallax. Stitching them with a single homography—the standard planar assumption—produces ghosting and structural misalignment. This paper's goal is to use estimated depth as a supervisory signal to align features across depth layers, and then to blend the aligned images using an adaptive "soft-seam" region rather than a hard seam or a naive average, so the final wide-field-of-view image shows no visible transition artifacts.

Key Contributions

1. A depth-supervised image stitching method aimed at the alignment challenges caused by large parallax from significant depth differences, enabling seamless fusion of multi-view images.
2. A depth-aware two-stage transformation estimation (coarse homography followed by mesh-based refinement) coupled with a reparameterization strategy, which the authors report significantly enhances alignment performance in large-parallax scenarios.
3. Determination of a soft-seam region that allows flexible adjustment of multi-view fusion, avoiding misalignment and ghosting.
4. Extensive experiments reported to outperform state-of-the-art methods in large-parallax alignment and seamless fusion.

Main Findings

• Quantitative comparison on UDIS-D: The proposed method records PSNR 25.467, SSIM 0.839, SIQE 43.732, and LPIPS 0.462. The closest competitor is UDIS++ at PSNR 25.426, SSIM 0.837, SIQE 43.184, LPIPS 0.469. Other compared methods include APAP (23.792 / 0.794 / 41.707 / 0.472), ELA (24.012 / 0.808 / 41.781 / 0.470), LPC (22.595 / 0.736 / 43.616 / 0.467), SPW (21.606 / 0.687 / 41.060 / 0.466), UDIS (21.171 / 0.648 / 42.186 / 0.475), TRIS (24.476 / 0.821 / 41.621 / 0.476), and SRS (24.828 / 0.811 / 41.857 / 0.473).
• Quantitative comparison on IVSD: The proposed method records PSNR 26.778, SSIM 0.820, SIQE 46.568, and LPIPS 0.436, against UDIS++ at 26.649 / 0.819 / 46.383 / 0.439. On the LPIPS metric specifically, ELA reports 0.435, which is lower (better) than the proposed method's 0.436 as listed in Table 1.
• Efficiency: At an image size of 512 × 512, the proposed method requires 67.04 ms, the lowest among the compared methods: UDIS++ 79.73 ms, SRS 83.17 ms, TRIS 107.98 ms, UDIS 193.66 ms, APAP 6683.14 ms, ELA 8347.79 ms, SPW 11651.68 ms, and LPC 13435.47 ms. The authors state that even with depth estimation and inference, the overall running time is better than the other methods.
• Qualitative results: On UDIS-D, the authors report clearer fusion of a wall area without blurring or ghosting, and preservation of complete bicycle content inside a marked region; on IVSD, they report consistent performance across varying scene depths, with visualized alignment errors in the lower-right corner of figures appearing smaller than for competitors.
• User study: 50 participants (30 researchers or students with a computer vision background and 20 without specific expertise) rated stitched images from 1 to 5 on ghosting, misalignment, structural accuracy, and realistic scene restoration. The proposed method consistently received higher ratings.
• Ablation on losses (Table 3, UDIS-D): Full model 25.470 / 0.839 / 43.732 / 0.462; w/o L_smooth 25.431 / 0.833 / 43.156 / 0.466; w/o L_cost 25.438 / 0.836 / 43.186 / 0.463; w/o L_reg 25.432 / 0.837 / 43.651 / 0.463; w/o L_mesh 25.473 / 0.840 / 43.701 / 0.463; w/o L_depth 25.434 / 0.838 / 43.703 / 0.463. The authors note that removing mesh constraints marginally improves certain metrics but that this is not meaningful visually, since it comes from relaxing image distortion limits.
• Complementary constraints: The paper states that L_edge and L_angle have complementary effects—retaining one alone renders the other nearly ineffective—so they are combined into a single mesh loss L_mesh for ablation.
• Reparameterization threshold: Ten thresholds were tested. c^ = 1 represents the original model and c^ = 0 the model without the Reparameterization Block Adaption (RBA). Both extremes perform worse than the model with c^ = 0.25, which achieves the best performance while maintaining efficient training.

Methodology in Plain English

The pipeline has two learned parts that are trained separately.

Part one—figuring out how to warp one image onto the other. A reference image and a target image are passed through ResNet50 to extract features at 1/16 and 1/8 resolution. At the coarse 1/16 scale, a Feature Correlation Aggregation (FCA) block computes inter-view correspondences, and a regression block predicts offsets for four quadrilateral vertices. Those four point pairs are turned into a coarse 3 × 3 homography using Direct Linear Transformation. The target features are then warped by that homography, and a mesh-based refinement stage predicts grid-wise offsets, interpolated into a continuous deformation field with radial basis functions using a Gaussian basis. The final warping field is the coarse homography plus the residual deformation.

Training losses combine an alignment term (measured as mean pixel error in the overlapping region after coarse and residual transformations, using masks for the homography, inverse homography, and residual), a mesh edge loss that limits mesh stretching, a mesh angle loss that encourages adjacent edges in non-overlapping regions to stay parallel, and a depth loss. The depth loss applies the same alignment formulation to depth maps produced by Depth Anything, normalized within the overlapping region to reduce relative error caused by depth discontinuities outside it. The paper notes that inverse losses are not designed because nonlinear transformations do not always support inverse operations.

Part two—blending the aligned pair. Rather than choosing one hard seam, the method treats any region in the overlap that needs fusion as a potential seam. Region masks from the aligned images go into a Soft-Seam Estimation (SSE) module built on a UNet architecture, where 3 × 3 convolutions are replaced with dilated convolutions at dilation rates 1, 2, 3, 4, and 5. At the four skip connections, same-scale features from both inputs are upsampled with nearest-neighbor interpolation and passed through a 1 × 1 convolution to reduce channels; a difference map is computed by pixel-wise subtraction, concatenated with the upsampled features along the channel dimension, and fed through two dilated convolution layers. The output soft-seam mask is combined with the original masks via a single filter and a sigmoid to yield two flexible pixel-level masks used for weighted fusion. Training losses here are a terminal loss (restricting fusion mask boundaries to the intersection area and controlling endpoints), a cost loss built on squared pixel differences, a smoothness loss over adjacent pixels in the fusion region, and a depth regularization loss applied to aligned and locally recalibrated depth images.

Part three—making the regression cheaper. The 3 × 3 convolution in the regression block behind the FCA block is replaced by a RepBlock containing a 1 × 1 convolution (Conv¹) and a 3 × 3 convolution (Conv³), combined in a linear manner with trainable weights and a ReLU. The contribution c₁ of Conv¹ is computed from the summed weights of the two branches. During training, if c₁ falls below a threshold hyperparameter c^, the 1 × 1 branch is folded into the 3 × 3 layer by padding its weights and summing weighted parameters, avoiding feature degradation from the extra branch.

Training setup: PyTorch on an NVIDIA RTX 3090 GPU, Adam optimizer, learning rate starting at 10⁻⁴ and decaying exponentially. The transformation model trains for 100 epochs with λ, γ, η set to 3, 3, 1; λ′, γ′, η′ equal to those; and μ, ζ, ξ set to 10, 10, 0.3. The multi-view fusion model trains for 50 epochs on the training set with ρ, τ, ι, σ set to 10000, 1000, 1000, 10. Training uses the UDIS-D training set; evaluation is on the UDIS-D testing set and real-world data from the IVSD dataset. UDIS-D test sets are categorized into three complexity levels following UDIS++.

Why This Matters

Impact on research. The paper shows that depth consistency can serve as a supervisory prior inside the transformation estimation step, not just as a post-hoc refinement, and that relaxing the notion of a "seam" into a diffused soft-seam region gives the fusion network room to correct local alignment errors. It also demonstrates that a reparameterization scheme tuned with a threshold (c^ = 0.25) can reduce the cost of shift regression without discarding the extra branch outright, which is a transferable engineering idea for other regression heads.

Real-world applications (categories named in the paper):

• Panoramic photography
• Remote sensing
• Medical imaging
• Virtual reality

Industry relevance. The efficiency result—67.04 ms at 512 × 512, faster than all eight compared methods including UDIS++ at 79.73 ms—matters for deployment, because the method still runs depth estimation and inference and yet stays within a practical runtime. The code is released at https://github.com/DLUT-YRH/DSFN.

Future Directions

• Multi-image panoramic stitching. The method is designed for two-image stitching and the authors state it currently lacks full capability for multi-image panoramic stitching, specifically citing difficulties in maintaining loop consistency and mitigating global error propagation in complex scenarios.
• Robust stitching under dynamic conditions. The conclusion notes that dynamic elements in a scene make accurate depth information hard to obtain, and states that robust stitching under dynamic conditions is a future focus.
• Extending the reparameterization study. The threshold analysis tested 10 values and found c^ = 0.25 best, leaving open how the threshold should be chosen for other architectures or datasets.
• Depth estimation as a bottleneck. Because the depth supervision depends on Depth Anything and on relative rather than absolute depth, the accuracy of the depth prior under scene changes remains an open question raised by the design.

Target Audience

Researchers and graduate students working on image stitching, panoramic imaging, and multi-view fusion; practitioners building wide-field-of-view reconstruction systems in photography, remote sensing, medical imaging, or VR; and engineers interested in depth-guided learning, seam-based blending strategies, or reparameterization for efficient regression heads. Readers without a background in homography estimation, mesh warping, or deep feature matching will find the methodology section demanding, as the paper is written for a computer vision research audience.

Authors’ abstract

Image stitching synthesizes images captured from multiple perspectives into a single image with a broader field of view. The significant variations in object depth often lead to large parallax, resulting in ghosting and misalignment in the stitched results. To address this, we propose a depth-consistency-constrained seamless-free image stitching method. First, to tackle the multi-view alignment difficulties caused by parallax, a multi-stage mechanism combined with global depth regularization constraints is developed to enhance the alignment accuracy of the same apparent target across different depth ranges. Second, during the multi-view image fusion process, an optimal stitching seam is determined through graph-based low-cost computation, and a soft-seam region is diffused to precisely locate transition areas, thereby effectively mitigating alignment errors induced by parallax and achieving natural and seamless stitching results. Furthermore, considering the computational overhead in the shift regression process, a reparameterization strategy is incorporated to optimize the structural design, significantly improving algorithm efficiency while maintaining optimal performance. Extensive experiments demonstrate the superior performance of the proposed method against the existing methods. Code is available at https://github.com/DLUT-YRH/DSFN.