Unaligned Image Guided Denoising via Cross-modal Conditional Flow Matching

AI summary

Overview

Research area: Computer Vision — low-level image restoration, specifically guided (cross-modal) image denoising using near-infrared (NIR) guidance under spatial misalignment.

Technical level: Advanced. The paper assumes familiarity with flow matching / diffusion-style generative models, U-Net architectures, cross-attention and optical-flow-based correspondence.

Scope (one sentence): The paper proposes UGD-FM, a cross-modal conditional flow matching framework that denoises a heavily noisy RGB image with the help of an unaligned NIR image, reaching state-of-the-art PSNR/SSIM/LPIPS with only one or two inference steps.

What This Paper Is About

Denoising a very noisy RGB photo is easier if a second, cleaner image of the same scene (such as a near-infrared image) is available as a structural guide. Most existing guided denoising methods assume the two images are pixel-aligned, or at best handle small horizontal offsets in rectified stereo pairs. This paper targets a harder, more realistic mobile-imaging case: the RGB image is severely noisy and the NIR guide is misaligned by a large two-dimensional displacement, so the system must simultaneously denoise and figure out where each RGB pixel corresponds to in the NIR image.

Key Contributions

1. UGD-FM framework: A general cross-modal conditional flow matching formulation for unaligned guided denoising, where the noisy RGB image is progressively transported toward the clean image, and the improving RGB state in turn makes cross-modal matching easier — a mutually reinforcing loop between restoration and guidance aggregation.

2. Stage-focused and inference-consistent training: A training strategy that places inference timesteps around two task-critical stages (a highly corrupted stage and a relatively cleaner stage), aligns training timesteps to these anchors via a mixture of truncated Gaussians, and replaces the naive interpolation state with a solver-induced state to reduce training–inference mismatch — enabling effective one-step and two-step inference.

3. Adaptive sparse guidance propagation: A coarse-to-fine matching scheme that keeps only the top-K candidates at the coarsest scale over a 2D range, then propagates them as 2×2 child locations to finer scales, avoiding prohibitively expensive dense large-range matching while aggregating aligned guidance features.

4. Auxiliary regularization and backbone compatibility: A perceptual consistency loss (LPIPS between the aggregated guidance image and the clean target) plus a flow consistency loss against pseudo flow annotations, both used only at training time; the framework is shown compatible with NAFNet, Uformer and Restormer backbones.

Main Findings

• Best overall results on FANVID: UGD-FM with the Restormer block at two-step inference reaches PSNR 30.77 / SSIM 0.8535 / LPIPS 0.2098 under Gaussian noise (σ = 0.3), 31.93 / 0.8764 / 0.1870 under Poisson-Gaussian noise (α = 0.02, σ = 0.2), and 30.58 / 0.8506 / 0.2121 under (α = 0.03, σ = 0.3).
• Best overall results on RGB-NIR Scene: Restormer two-step reaches 29.87 / 0.8188 / 0.2551 (σ = 0.3), 30.98 / 0.8437 / 0.2315 (α = 0.02, σ = 0.2), and 29.63 / 0.8141 / 0.2585 (α = 0.03, σ = 0.3).
• Best overall results on MS2: Restormer two-step reaches 33.07 / 0.9343 / 0.1592 (σ = 0.3), 34.34 / 0.9452 / 0.1444 (α = 0.02, σ = 0.2), and 32.82 / 0.9329 / 0.1609 (α = 0.03, σ = 0.3).
• Gains over all ten baselines: UGD-FM outperforms five single-image denoisers (MPRNet, NAFNet, Uformer, Restormer, Xformer) and five guided methods (CUNet, DVN, RFFNet, NAID, SANet) across every dataset and noise setting; notably, SANet under σ = 0.3 on FANVID obtains 29.15 PSNR versus 30.50 for the one-step Restormer variant.
• Two steps beat one step consistently: For example, on FANVID (α = 0.03, σ = 0.3), NAFNet goes from 30.12 to 30.42 PSNR and Restormer from 30.29 to 30.58 when moving from one to two steps.
• Beats diffusion/flow baselines: On FANVID under (α = 0.03, σ = 0.3), RDBM scores 28.80 PSNR, 3M-TI 28.89, and RFfusion 28.53, compared with 30.12 (NAFNet one-step) and 30.42 (NAFNet two-step) for UGD-FM.
• Transfers to thermal super-resolution: Following the 3M-TI protocol, UGD-FM obtains the best SSIM (0.8710) and MUSIQ (36.94), with PSNR 30.36, LPIPS 0.2012 and MANIQA 0.4008 — versus 3M-TI's 30.09 / 0.8610 / 0.1787 / 0.4443 / 36.66.
• Flow formulation and training schedule help cumulatively (FANVID, α = 0.03, σ = 0.3): direct regression gives 30.27 PSNR at one step (no two-step result reported); standard flow matching gives 30.28 (one-step) and 30.34 (two-step); adding stage-focused sampling gives 30.31 / 30.42; adding inference-consistent state construction reaches 30.29 / 30.58.
• Dense matching is impractical: In the guidance-aggregation ablation, global attention causes out-of-memory errors, while sparse propagation achieves 30.29 / 0.8401 / 0.2215 (one-step) and 30.58 / 0.8506 / 0.2121 (two-step) versus 28.97 / 0.7920 / 0.3247 and 29.10 / 0.7976 / 0.3244 with no attention.
• Both auxiliary losses matter: Removing both drops two-step PSNR from 30.58 to 29.78; keeping only the flow consistency loss gives 30.45 and only the perceptual loss gives 30.27.

Methodology in Plain English

The team treats denoising as a gradual journey rather than a single jump. The noisy RGB image is the starting point, the clean image is the destination, and the model learns to predict a "velocity" that moves the image along a straight line between the two. Because the starting image is so noisy, trying to match it to the NIR guide all at once fails; instead, as the image gets cleaner along the journey, matching becomes easier and the extracted guidance makes the next step of the journey more accurate.

To keep inference short, they noticed that two moments of the journey matter most: the very beginning, where only coarse RGB structure can be recovered, and a later, cleaner moment where fine details and reliable cross-modal matching are possible. They therefore aim their one- or two-step solver at exactly those two moments, and change training so that the model sees the same kinds of intermediate images the solver will actually produce, rather than idealized interpolations.

For matching, searching every possible offset over a ±80-pixel 2D range at every scale would be far too costly. Instead, at the coarsest scale they search the full range once per pixel, keep only the top-K best-matching NIR candidates, and carry those candidates down to finer scales as 2×2 child locations, repeating the top-K selection in that small set. The resulting weighted sum of NIR features is injected into the RGB features through spatially varying weights and a gate that can suppress guidance when the match looks wrong. Two training-only losses stabilize this: one compares the aggregated guidance image against the clean RGB target using LPIPS, and the other compares the implied displacement field against pseudo flow computed between the clean target and the guide by an existing flow network.

The architecture is a two-branch U-Net, with the multi-scale RGB and NIR features fused in the decoder; the same guidance machinery is dropped into NAFNet, Uformer and Restormer blocks to make three variants. For FANVID and RGB-NIR Scene, which ship aligned pairs, the authors synthesize misalignment by reconstructing the NIR image as a 3D Gaussian representation with SHARP and rendering new NIR views from eight virtual camera positions; MS2 already contains real multi-camera misalignment.

Why This Matters

Impact on research: The paper reframes guided denoising from a one-shot regression problem into a progressive transport problem, and shows that a generative-model formulation can be squeezed into one or two steps without losing accuracy. It also provides a way to retrofit existing single-image restoration backbones (NAFNet, Uformer, Restormer) with cross-modal guidance, and demonstrates that sparse candidate propagation is a tractable substitute for dense large-displacement matching.

Real-world applications:

• Smartphone night photography, where a wide RGB camera and a separate NIR or secondary sensor cannot be perfectly aligned and the RGB frame is severely noisy.
• Surveillance and security cameras that pair a low-light color sensor with an NIR illuminator.
• Automotive and robotics imaging, where multi-camera rigs must fuse modalities without precise calibration.
• Thermal/RGB imaging systems, as demonstrated by the paper's RGB-guided thermal super-resolution experiment.

Industry relevance: Mobile and camera manufacturers already ship multi-sensor modules; a method that works with uncalibrated, large-baseline misalignment removes a costly calibration and mechanical-alignment burden, and the one-to-two-step inference is attractive for on-device latency budgets.

Future Directions

• Extending the framework to displacements larger than the tested search radius of D = 80, and to non-frontal, rotation- or scale-heavy misalignment that a pure 2D translational search may not capture.
• Reducing dependence on pseudo flow annotations produced by an external flow network between the clean target and the guide — a training-time dependency whose removal would simplify data preparation and avoid inheriting that network's errors.
• Testing whether the sparse top-K propagation generalizes beyond two modalities (e.g., RGB plus thermal, depth, or event sensors) and beyond denoising to other degradations such as blur or low-light enhancement.
• Pushing toward truly single-step inference without quality loss, and characterizing when the two-stage anchor schedule (τ_hc = 0 plus one intermediate τ_rc) should be replaced by more anchors.

Target Audience

Graduate students and researchers in computational photography and low-level vision who work on guided/cross-modal restoration or on diffusion and flow-matching models for inverse problems. It is also useful for applied engineers in mobile imaging, automotive sensing, or camera ISP teams who need denoising that tolerates uncalibrated multi-sensor geometry. Readers without background in flow matching or attention-based feature alignment will find the method sections dense and should start with the introduction and Figure 1.

Authors’ abstract

This work presents UGD-FM, a novel unaligned image guided denoising framework based on cross-modal conditional flow matching. Unlike previous approaches that assume spatially aligned inputs or only handle small-baseline rectified pairs, we address a more challenging setting where severe target noise and large-range two-dimensional cross-modal misalignment coexist. Specifically, we formulate guided denoising as a progressive flow matching process, allowing image restoration and aligned guidance aggregation to mutually reinforce each other. To support efficient few-step inference, we introduce a stage-focused and inference-consistent training strategy that focuses on task-critical denoising stages, and mitigates the training-inference mismatch. We further propose adaptive sparse guidance propagation for efficient and reliable large-range matching, regularized by perceptual and geometric auxiliary losses. Experiments demonstrate that UGD-FM achieves state-of-the-art denoising performance with only one or two inference steps, and is compatible with modern single-image restoration backbones.