PhaSR: Generalized Image Shadow Removal with Physically Aligned Priors

AI summary

Overview

• Research area: Computer vision — single-image shadow removal, ambient light normalization, and physically guided image restoration.
• Technical level: Advanced. The paper assumes familiarity with Retinex intrinsic decomposition, Transformer attention (including differential attention), and pretrained vision backbones (DINO-v2, DepthAnything-v2).
• Scope: A mask-free shadow removal framework, PhaSR, that applies closed-form illumination normalization followed by cross-modal geometric-semantic attention, and that generalizes from single-light outdoor shadows to multi-source indoor ambient lighting.

What This Paper Is About

Shadows are hard to remove because the network must separate illumination from the surface's true reflectance, and dark objects look similar to shaded regions. Prior methods inject physical cues (depth, normals, semantics) but these cues often conflict: geometry responds sharply to local shading, while semantics stay stable across lighting, and naive fusion blurs boundaries or adds noise. PhaSR attacks this "prior misalignment" with two aligned stages — a parameter-free normalization step and a cross-modal attention that subtracts geometric attention from semantic attention — aiming to work not only on single-light direct shadows but also on multi-source ambient lighting where earlier methods degrade.

Key Contributions

1. Physically Aligned Normalization (PAN): a closed-form, model-free preprocessing module combining Gray-world color normalization, log-domain Retinex decomposition, and dynamic range recombination, which suppresses chromatic bias. The authors report it consistently improves existing architectures by 0.15–0.34 dB across diverse lighting conditions.
2. Geometric-Semantic Rectification Attention (GSRA): an extension of differential attention to cross-modal alignment, computing A_rect = A_sem − λ · A_geo over depth-derived geometry (DepthAnything-v2) and DINO-v2 semantic embeddings, to harmonize physically grounded geometric precision with semantic stability.
3. State-of-the-art results under mask-free evaluation: reported best performance on the WSRD+ and Ambient6K benchmarks, with generalization from outdoor direct shadows to indoor indirect and ambient lighting, while remaining computationally efficient.
4. Cross-model plug-in validation: PAN is shown to improve OmniSR and DenseSR as well as PhaSR, and is compared against classical parameter-free color correction methods (ACE, White-balance, White-Patch, CIELab) on WSRD+.

Main Findings

• Benchmark results on five datasets: PhaSR reaches 30.73 PSNR / 0.960 SSIM on ISTD, 34.48 / 0.960 on ISTD+, 30.38 / 0.961 on INS, 28.44 / 0.842 on WSRD+, and 23.32 / 0.834 on Ambient6K.
• Ambient lighting is where the gap is largest: On Ambient6K, PhaSR (23.32 / 0.834) outperforms dedicated ambient light normalization methods IFBlend (21.44 / 0.819) and RLN²-Lf (21.71 / 0.825), as well as ReHiT (19.98 / 0.798) and DC-ShadowNet (17.73 / 0.711).
• Not uniformly best on every metric: On ISTD+, StableShadowDiffusion reports a higher PSNR of 35.19 / 0.970 and DenseSR reports 33.98 / 0.974, both above PhaSR's 34.48 / 0.960 PSNR/SSIM pairing. On INS, DenseSR reports 30.64 / 0.981 and OmniSR 30.38 / 0.973, versus PhaSR's 30.38 / 0.961. On ISTD, PhaSR has the highest PSNR (30.73) but a lower SSIM (0.960) than DenseSR (0.976).
• PAN reduces residual error: measured as mean pixel-wise difference from a shadow-free reference, PAN lowers residual error by +26.4% (ISTD train), +17.3% (ISTD test), +4.3% (INS test), +1.3% (Ambient6K test), +4.4% (WSRD+ test), +3.4% (SRD test), +3.5% (CL3AN SH test) and +7.4% (CL3AN CR test). The authors summarize steady gains of 1–8% in ambient conditions.
• PAN helps other models too: adding PAN raises OmniSR by +0.22 PSNR on ISTD+, +0.22 on WSRD+, +0.24 on Ambient6K; DenseSR by +0.05, +0.33, +0.24 respectively; and PhaSR by +0.15, +0.27, +0.34 respectively.
• PAN beats generic color correction on WSRD+: PAN scores 28.4421 PSNR, 0.8418 SSIM, 0.0469 LPIPS, 0.9452 RMSE, versus White-balance at 27.1237 / 0.8125 / 0.0548 / 0.9762 and CIELab at 25.4175 / 0.8016 / 0.0715 / 1.1562.
• Ablation confirms both components matter: removing PAN drops ISTD+ to 33.15 / 0.952 and WSRD+ to 28.17 / 0.825; removing GSRA (replaced by cross-attention) drops to 32.56 / 0.934 and 26.92 / 0.820; disabling rectification with λ = 0 drops to 32.89 / 0.951 and 27.32 / 0.832. Removing the geometric prior gives 33.52 / 0.956 and 27.85 / 0.838; removing the semantic prior gives 33.38 / 0.955 and 27.71 / 0.837.
• Efficiency: PhaSR runs in 87.9 ms per 640 × 480 image with 55.632 G FLOPs and 18.949 M parameters — the lowest FLOPs and second smallest parameter count among compared models. For contrast, ShadowDiffusion runs 506.9 ms / 174.658 G / 55.376 M, OmniSR 120.1 ms / 78.316 G / 24.553 M, DenseSR 124.6 ms / 81.127 G / 24.698 M, and StableShadowDiffusion 452.8 ms / 678.577 G / 1329.824 M. ShadowFormer is faster (43.7 ms) and smaller (11.352 M).
• Evaluation is mask-free: methods requiring explicit shadow masks as input are excluded from comparison, reflecting real deployment where automatic mask detection often fails.

Methodology in Plain English

PhaSR uses two alignment stages.

Stage 1 — PAN (global illumination correction). It is a closed-form preprocessing pipeline with no learned parameters. First, Gray-world normalization rescales each color channel toward the image's spatial average to remove color casts from warm or cool light (a small ε = 10⁻⁶ avoids division by zero). Second, following the Retinex image formation model I_norm(x) = R(x) ⊗ S(x), it takes the logarithm so the terms become additive: log I_norm = log R + log S. It estimates global lighting as the spatial average in log space, obtains pseudo-reflectance as the residual, and exponentiates both back. Third, it recombines the pseudo-components and min-max normalizes them into a stable, illumination-consistent image Î.

Stage 2 — GSRA (local cross-modal rectification). The backbone is a hierarchical Transformer encoder-decoder with base channel dimension C = 32 and seven Transformer blocks (N₁–N₇), each with 2 layers. Frozen DINO-v2 provides semantic embeddings and DepthAnything-v2 provides depth and normal maps. Depth-to-normal conversion assumes a pinhole camera with FOV = 60°. Each prior is injected into the shared input feature with a learnable weight (α_geo for geometry, α_sem for semantics), then the two branches produce their own keys and values through lightweight linear projections. Using a shared query, the model computes geometric and semantic attention maps (with relative position bias) and rectifies via A_rect = A_sem − λ · A_geo, where the learnable λ balances context-dependent illumination variation against geometric regularization. The output concatenates A_rect·V_geo and A_rect·V_sem.

Training. Supervision combines Charbonnier loss (λ = 0.95, ε = 10⁻⁶) and SSIM loss (λ = 0.05). Training uses 256 × 256 random crops, AdamW (β₁ = 0.9, β₂ = 0.999, ε = 1 × 10⁻⁸), batch size 9, 1400 epochs, a learning rate starting at 2 × 10⁻⁴ with cosine annealing, and random flipping and rotation augmentation. Reported dataset sizes are not given in the paper content.

Why This Matters

Impact on research. The paper reframes shadow removal as a prior-alignment problem rather than a fusion problem, and shows that a parameter-free normalization step can be dropped into existing architectures to gain 0.15–0.34 dB. It also demonstrates that structured geometric and semantic priors help on ambient light normalization, a task previously handled by specialized methods.

Real-world applications.

• Computational photography and smartphone camera pipelines, where shadows are removed before display or saving.
• Surveillance and autonomous driving, where shadow boundaries cause false object detections and tracking errors.
• Remote sensing and satellite imagery, where shadow removal improves downstream segmentation and change detection.
• AR/VR and 3D reconstruction, where inconsistent lighting impairs relighting and geometry recovery.

Industry relevance. The mask-free design avoids the fragile shadow-detection step that often fails on soft shadows or unusual lighting. At 87.9 ms per 640 × 480 image with 55.632 G FLOPs and 18.949 M parameters, the model is far lighter than diffusion-based alternatives like StableShadowDiffusion (1329.824 M parameters, 678.577 G FLOPs), which the authors position as suitable for real-time or embedded deployment.

Future Directions

• Extending to more complex multi-source illumination: the paper notes that disentangling multiple overlapping light contributions remains an open, ill-posed challenge, even though PhaSR generalizes to ambient lighting.
• Resolving failure cases: the supplementary material identifies dark intrinsic materials and specular surfaces as challenging scenarios where the method still struggles.
• Improving SSIM on standard benchmarks: PhaSR trails DenseSR on SSIM across ISTD, ISTD+, and INS, so structural fidelity on classic single-light datasets is an unresolved gap.
• Cross-dataset transfer: the supplementary material describes Ambient6K ↔ ISTD cross-dataset experiments testing generalization from single-source outdoor shadows to multi-source indoor ambient lighting; further work could extend this to unseen lighting conditions and other sensor modalities.

Target Audience

Researchers and engineers working on shadow removal, intrinsic image decomposition, illumination normalization, and multimodal vision transformers. It is most useful for readers already comfortable with attention mechanisms and Retinex theory who want a compact, mask-free architecture that plugs physical priors into a restoration backbone. Practitioners seeking a deployable, low-complexity model for camera or embedded pipelines will also find the efficiency numbers and plug-in PAN results directly relevant.

Authors’ abstract

Shadow removal under diverse lighting conditions requires disentangling illumination from intrinsic reflectance, a challenge compounded when physical priors are not properly aligned. We propose PhaSR (Physically Aligned Shadow Removal), addressing this through dual-level prior alignment to enable robust performance from single-light shadows to multi-source ambient lighting. First, Physically Aligned Normalization (PAN) performs closed-form illumination correction via Gray-world normalization, log-domain Retinex decomposition, and dynamic range recombination, suppressing chromatic bias. Second, Geometric-Semantic Rectification Attention (GSRA) extends differential attention to cross-modal alignment, harmonizing depth-derived geometry with DINO-v2 semantic embeddings to resolve modal conflicts under varying illumination. Experiments show competitive performance in shadow removal with lower complexity and generalization to ambient lighting where traditional methods fail under multi-source illumination. Our source code is available at https://github.com/ming053l/PhaSR.