ReflexSplit: Single Image Reflection Separation via Layer Fusion-Separation

AI summary

Overview

Research area: Computer vision, specifically single image reflection separation (SIRS) — the task of splitting one photograph of a reflection-contaminated scene into its transmission (the scene behind the glass) and reflection layers.

Technical level: Advanced. The paper assumes familiarity with attention mechanisms, transformer architectures, multi-scale encoder-decoder design, and curriculum learning schedules.

Scope in one sentence: ReflexSplit is a dual-stream network that explicitly alternates between fusing and separating transmission/reflection features at every decoder level, rather than relying on implicit feature mixing.

What This Paper Is About

When a photo is taken through glass, the camera records a mixture of the scene behind the glass (transmission) and the reflected scene in front of it (reflection). The goal of SIRS is to recover both layers cleanly from a single image.

The authors identify a persistent failure mode they call transmission-reflection confusion: as networks get deeper, characteristics of the two layers bleed into each other, so the network starts mixing up which parts of the image belong to which layer. This is especially pronounced in deep decoder layers and under nonlinear mixing (strong light sources, over-exposure, specular highlights). ReflexSplit's goal is to prevent that confusion across the entire network hierarchy.

Key Contributions

1. Explicit layer fusion-separation framework. A paradigm that alternates between a fusion step (extracting shared structure/degradation from both streams) and a differential separation step (enforcing layer-specific disentanglement), preventing transmission-reflection confusion throughout the decoder hierarchy.

2. Cross-scale Gated Fusion (CrGF). A module at decoder levels {4, 3, 2} that adaptively aggregates semantic priors from a Swin Transformer branch, texture details from a MuGI-based CNN branch, and decoder context across hierarchical depths using bidirectional gating, to stabilize gradient flow and maintain feature consistency.

3. Layer Fusion-Separation Blocks (LFSB). Blocks combining early bidirectional cross-stream projection, dual-dimensional attention (self-attention plus cross-attention), and a differential operator that performs cross-stream subtraction (A^t − λ_ℓ A^r) to suppress inter-layer interference — extending the Differential Transformer's attention-cancellation idea from single-stream noise suppression to dual-stream layer separation.

4. Curriculum training. Depth-dependent initialization plus epoch-wise warmup that progressively strengthens differential separation, letting the network learn holistic reconstruction before focusing on layer-specific disentanglement.

Main Findings

• Synthetic benchmark results (Table 1). ReflexSplit reaches an average over 540 images of 26.40 PSNR / 0.898 SSIM, versus RDNet at 26.38 / 0.890 and DSIT at 25.93 / 0.886. Per-dataset: Real20 25.22 / 0.846, Objects 27.08 / 0.929, Postcard 25.38 / 0.927, Wild 27.30 / 0.933, Nature 27.03 / 0.854. The paper reports that ReflexSplit leads consistently on SSIM while PSNR is competitive.

• Real-world cross-dataset generalization on OpenRR-1K (Table 2). ReflexSplit achieves 26.5824 PSNR, 0.9372 SSIM, 0.1087 LPIPS, 2.9764 NIQE, 0.0684 DISTS — the highest SSIM and lowest LPIPS and DISTS among compared methods. DSIT obtains marginally higher PSNR at 26.6672 but has worse LPIPS (0.1137), NIQE (2.9872), and DISTS (0.0700). Models were tested without fine-tuning after training on PASCAL VOC, Real20, and Nature.

• Parameter efficiency. ReflexSplit has 174M trainable parameters, which the authors describe as 34% fewer than RDNet (266.4M) and 28% more than DSIT (136M), positioning it between the two on efficiency while outperforming both on SSIM.

• CrGF ablation (Table 3). Replacing CrGF with alternatives drops performance: no fusion (direct aggregation) gives 24.01 / 0.838 on Real20; simple concatenation 24.89 / 0.828; element-wise addition 25.01 / 0.830; full CrGF 25.22 / 0.846. On SIR² the gap is larger (26.33 / 0.896 for CrGF vs. 25.32 / 0.881 for no fusion).

• LFSB component ablation (Table 4). Progressive addition on Real20: baseline dual-stream decoder 23.87 / 0.812; plus early fusion 24.32 / 0.823; plus self-attention 24.61 / 0.831; plus SA+CA 24.89 / 0.838; plus differential separation 25.08 / 0.842; full LFSB 25.22 / 0.846.

• Curriculum training ablation (Table 5). Fixed λ = 0.5 gives 24.83 / 0.821 on Real20; warmup only 24.95 / 0.835; depth initialization only 25.08 / 0.840; the full combined strategy 25.22 / 0.846. On SIR², the full strategy reaches 26.33 / 0.896.

• Removing LFSB causes a large drop. The paper reports that without LFSB, PSNR falls to 19.946 and Normalized Cross-Correlation between streams rises to 0.9254 (lower NCC is better, so a higher value indicates more entanglement).

• Qualitative differences. Against DSRNet, MaxRF, DSIT, and RDNet on Postcard, Wild, and SolidObject, the authors report DSIT suffers transmission-reflection confusion, DSRNet fails in high-frequency regions, RDNet introduces color distortion, and MaxRF loses edge details, while ReflexSplit produces cleaner separation and preserves texture, boundaries, and color.

• Supplementary comparison with RDNet. On OpenRR-1K, ReflexSplit is reported to improve over RDNet on 63% of test images.

• Attention visualization. Cross-attention maps between transmission and reflection overlap in the baseline; after differential separation via A^t − σ(λ_ℓ)A^r the attention distributions become balanced and layer-specific rather than overlapping.

• Not reported. Inference latency, FLOPs, and memory figures are not given in the provided content (the supplementary lists a "More Complexity Comparison" section, but no numbers appear in the text supplied).

Methodology in Plain English

The system takes a mixed image I of size H × W × 3 and produces three outputs: transmission T̂, reflection R̂, and a residual RR̂ that captures the nonlinear interaction between layers.

Two encoders, two viewpoints. One branch is a pretrained Swin Transformer (the Global Feature Extractor Block, GFEB) that produces semantic priors at scales {P₂, P₃, P₄, P₅}. The other is a MuGI-based CNN (the Local Feature Extractor Block, LFEB) that produces texture features at resolutions E₀ through E₅, with H_ℓ = H/2^ℓ. Splitting the job this way keeps global meaning and fine detail from tangling in a single stream.

Fusing across scales (CrGF). At decoder levels 4, 3, and 2, CrGF combines the current decoder context F_{ℓ+1}, the semantic prior P_ℓ, and the texture feature E_ℓ. It computes two complementary gated paths — one emphasizing the current level gated by context, one propagating context gated by current cues — and blends them with softmax-normalized weights after 1×1 convolutions. At levels 1 and 0, where no P_ℓ exists, it simply adds F_{ℓ+1} + E_ℓ.

Alternating fuse and separate (LFSB). Each block first applies bidirectional projection (concatenating transmission and reflection features and projecting them) so the two streams can share information about common degradations. Then it computes attention two ways: self-attention on features stacked along the batch dimension (for within-layer spatial structure) and cross-attention on features stacked along the sequence dimension (for between-layer dependencies). The distinctive step is that instead of just adding SA and CA outputs together — as DSIT does — the model subtracts one stream's attention from the other's with a learnable, sigmoid-scaled coefficient: A^t_diff = (A^t_SA + A^t_CA) − σ(λ_ℓ)(A^r_SA + A^r_CA), and symmetrically for the reflection stream. This actively cancels the interference each layer leaks into the other. A feed-forward network with residual connections then produces the next-level features.

Curriculum over the differential strength. The differential strength λ_ℓ is set by depth as λ_ℓ^init = 0.8 − 0.6·e^(−0.3ℓ), so deeper layers separate more aggressively (approaching 0.8) and shallow layers stay gentler (approaching 0.2) to preserve fine detail. Separately, an epoch-wise factor λ_diff(e) rises linearly from 0.1 to 1.0 over E_warmup = 30 epochs and stays at 1.0 afterward. The effective coefficient is the product, λ_ℓ(e) = λ_ℓ^init · λ_diff(e).

Training setup. Training uses a composite dataset of 7,643 synthetic pairs from PASCAL VOC, 90 real pairs, and 200 Nature pairs; synthetic pairs are blended with I_syn = γ₁T_syn + γ₂R_syn − γ₁γ₂(T_syn ∘ R_syn) with γ₁ ∈ [0.8, 1.0] and γ₂ ∈ [0.4, 1.0], inspired by the "screen" blending mode. Each epoch samples 5,000 pairs at a 0.6:0.2:0.2 synthetic:real:Nature ratio. Images are resized to 384 × 384. Optimization uses Adam with learning rate 10⁻⁴, weight decay 0, batch size 1, CosineAnnealingLR (T_max = 10, η_min = 8×10⁻⁶), for 200 epochs on a single NVIDIA RTX 4090. LFSB uses window size W = 12 and attention heads {2, 4, 8, 8, 8} at decoder levels {0, 2, 3, 4, 5}.

Losses. The objective sums six terms: Charbonnier reconstruction loss on transmission (ε = 10⁻⁶), ℓ₁ reflection loss, VGG perceptual loss using features from layers {2, 7, 12, 21, 30}, color consistency loss (matching mean and standard deviation of the reflection), exclusion loss (penalizing gradient overlap between T̂ and R̂), and a reconstruction consistency loss enforcing T̂ + R̂ + RR̂ ≈ I. Weights given are λ_rec = 1.0, λ_refl = 0.5, λ_vgg = 0.1, λ_exclu = 1.0, λ_recons = 0.2; the value of λ_color is not reported in the provided text.

Why This Matters

Impact on research. The paper argues that the field's implicit fusion mechanisms — feature exchange without an explicit separation constraint — are the root cause of progressive layer confusion in deep decoders, and that this failure worsens with depth. It offers a concrete architectural answer (alternating fusion and differential separation) plus a training schedule, and it extends the Differential Transformer's cancellation idea from noise suppression within a single stream to disentanglement between two streams. This reframes reflection separation as a differential-attention problem rather than a feature-exchange problem.

Real-world applications:

• Autonomous driving, where reflections on windshields or wet roads corrupt perception input.
• Industrial inspection, where photographing products through glass or transparent packaging introduces reflections that obscure defects.
• Photography and image restoration, letting users recover the scene behind a window or the reflected subject in front of it as separate usable images.
• General image enhancement pipelines, where reflection artifacts degrade downstream vision tasks and the residual term RR̂ explicitly accounts for nonlinear mixing such as over-exposure and specular highlights.

Industry relevance. The method is a single-stage model, which the supplementary material highlights as a practical advantage over two-stage training approaches such as RDNet. At 174M parameters with a public checkpoint link (https://github.com/wuw2135/ReflexSplit), it sits between DExNet (9.6M) and RDNet (266.4M) in size while delivering the strongest SSIM and LPIPS on the real-world OpenRR-1K benchmark.

Future Directions

• Quantifying efficiency. The supplementary promises a complexity comparison but the provided content gives no latency, FLOPs, or memory figures, so the single-stage efficiency claim remains unverified in numbers.
• Fixing the stated failure cases. The supplementary identifies complex outdoor lighting, specular reflections, and mixed indoor-outdoor scenes with extreme brightness differences as scenarios where ReflexSplit struggles. Extending the method to these regimes is the obvious next step.
• Untangling the differential parameter. λ_ℓ is set by a fixed hand-designed formula (depth exponential) times a hand-designed warmup ramp. Whether these schedules can be learned or adapted per image is open.
• Deeper architectural questions. The paper positions explicit separation against RDNet's reversible-encoder philosophy, noting RDNet improves on only some images. The supplementary discusses complementary architectural philosophies between reversible designs and explicit layer separation — whether the two can be combined is unaddressed.

Target Audience

Researchers and graduate students working on computational photography, image restoration, and reflection removal; practitioners building perception systems for autonomous driving or industrial inspection that must handle glass and reflective surfaces; and anyone studying attention mechanism design, specifically differential attention and dual-stream disentanglement. The paper is written for readers comfortable with transformer-based vision architectures and multi-scale encoder-decoder training; it is not an introductory read.

Authors’ abstract

Single Image Reflection Separation (SIRS) disentangles mixed images into transmission and reflection layers. Existing methods suffer from transmission-reflection confusion under nonlinear mixing, particularly in deep decoder layers, due to implicit fusion mechanisms and inadequate multi-scale coordination. We propose ReflexSplit, a dual-stream framework with three key innovations. (1) Cross-scale Gated Fusion (CrGF) adaptively aggregates semantic priors, texture details, and decoder context across hierarchical depths, stabilizing gradient flow and maintaining feature consistency. (2) Layer Fusion-Separation Blocks (LFSB) alternate between fusion for shared structure extraction and differential separation for layer-specific disentanglement. Inspired by Differential Transformer, we extend attention cancellation to dual-stream separation via cross-stream subtraction. (3) Curriculum training progressively strengthens differential separation through depth-dependent initialization and epoch-wise warmup. Extensive experiments on synthetic and real-world benchmarks demonstrate state-of-the-art performance with superior perceptual quality and robust generalization. Our code is available at https://github.com/wuw2135/ReflexSplit.