PI-Light: Physics-Inspired Diffusion for Full-Image Relighting

AI summary

Overview

Research area: Computer vision and computer graphics, specifically full-image relighting via diffusion models and physically-based rendering (PBR).

Technical level: Advanced. The paper assumes familiarity with latent diffusion models, self-attention, the Principled BRDF, Lambertian diffuse reflection, and the Cook-Torrance microfacet specular model.

Scope: The paper proposes and evaluates π-Light, a two-stage physics-inspired diffusion framework for relighting entire images (foreground and background), together with a newly constructed object-and-scene dataset rendered under controlled lighting.

What This Paper Is About

Relighting an entire image — not just a person or a foreground object — is hard because it is nearly impossible to capture real photos of the same scene under many different lighting conditions, so large structured paired datasets are scarce. Purely data-driven models tend to produce results that violate basic light transport rules, and they generalize poorly to unseen materials and lighting. The paper's goal is to combine diffusion priors with physical light-transport constraints so that a model can decompose an image into its material properties and then re-render it under a new, precisely controllable light direction and intensity.

Key Contributions

1. A diffusion-based neural relighting framework for physically-aware full-image relighting, built as a two-stage pipeline of inverse neural rendering followed by neural forward rendering.
2. Physics-inspired light-transport priors imposed as training regularization on the neural forward rendering module, which the authors report helps the model converge toward physical principles and improves generalizability with less data and computation than prior works.
3. A new high-quality dataset of diverse objects and scenes rendered under controlled lighting conditions, providing both training data and a downstream benchmark.
4. Two supporting design choices: batch-aware (global-aware) attention that lets batches communicate for consistent intrinsic predictions, and a lighting representation using only the front hemisphere of a rendered gray ball to avoid interference from self-luminous objects and built-in scene lighting.

Main Findings

• Inverse rendering results: On the Object50 test set, the method reports an average PSNR of 20.53, SSIM of 0.8934, and LPIPS of 0.0761, versus RGB↔X at 19.32 / 0.8908 / 0.0877, Kocsis et al. at 19.15 / 0.8844 / 0.0.0893 (as printed), and Zhu et al. at 17.43 / 0.8584 / 0.1042. IntrinsicAnything is reported only for albedo (21.92 / 0.9157 / 0.0677).
• Scene-level inverse rendering: On Scene200, the method reports average PSNR 12.95, SSIM 0.5646, LPIPS 0.3741, compared with RGB↔X at 11.75 / 0.5546 / 0.3892, Kocsis et al. at 12.21 / 0.5694 / 0.3778, and Zhu et al. at 11.29 / 0.4629 / 0.3616. The authors note their metallic scores are lower than some baselines on this set (7.02 PSNR / 0.2001 SSIM / 0.6966 LPIPS).
• Metallic is the weakest intrinsic: On Object50, the method's metallic scores (13.96 / 0.8357 / 0.1216) are lower in PSNR than RGB↔X (15.97 / 0.8404 / 0.1226) and Kocsis et al. (14.02 / 0.8335 / 0.1250).
• Forward rendering beats prior object relighting methods: On Object50, the method reports 14.09 PSNR / 0.9211 SSIM / 0.0553 LPIPS, versus Neural Gaffer at 11.72 / 0.9069 / 0.0844 and DiLightNet at 12.21 / 0.8938 / 0.0890. Under the aligned "reconstruction" setting, the method reports 15.06 / 0.9332 / 0.0458, versus Neural Gaffer at 13.69 / 0.9186 / 0.0765, DiLightNet at 12.72 / 0.9035 / 0.0816, and RGB↔X at 9.64 / 0.8878 / 0.0783.
• Ablation confirms both decomposition and physics losses matter: Trained for 30k iterations on Object50, the baseline without decomposition and without shading losses scores 10.88 PSNR / 0.8964 SSIM / 0.0712 LPIPS. Adding decomposition gives 11.95 / 0.9072 / 0.0616; adding decomposition plus the diffuse shading loss gives 11.97 / 0.9054 / 0.0617; adding decomposition plus the physical-based shading loss gives 12.18 / 0.9070 / 0.0620; and using all three gives 12.68 / 0.9139 / 0.0572.
• Comparison against foreground relighting: The paper argues IC-Light fails to preserve albedo and roughness and lacks precise lighting control — in its example, a requested right-side light still appears to come from the left, and carpet colors are altered across rows.
• Real-image generalization: The authors report that their model generates specular highlights, including on highly metallic objects such as a desk lamp, and produces shadows on the ground or wall for a basketball and water cup, while still being trained with fewer samples than prior works and without fully realistic datasets.

Methodology in Plain English

The approach splits the job into two learned stages, both built on a pretrained image diffusion model (Stable Diffusion), with the U-Net initialized from GeoWizard.

Stage 1 — take the image apart. A batch of four copies of the input image is fed through one diffusion model at once. Each copy carries a one-hot label telling it which property to output: albedo, normal, roughness, or metallic. The standard self-attention layers are extended to be "global-aware" so the four batches can exchange information, which the authors say improves both efficiency and consistency of the four predicted intrinsics. The CLIP image embedding of the condition inputs goes into cross-attention layers. A per-sample mask marks unreliable regions (transparent or translucent materials, sky normals, non-Principled-BRDF regions), and the masked loss is applied directly in V-prediction latent space with the mask downsampled by a factor of 8.

Lighting representation. Instead of using irradiance or a full HDRI, the method renders a gray ball with 50% roughness from the camera position, and uses only the front hemisphere. This avoids the original scene's built-in lights and self-luminous objects influencing the target lighting control.

Stage 2 — render under new light. Based on the surface reflection model I_rendered = A ⊙ D + S, three batched inputs are used: (input image, albedo) → relit image; (normal, light, mask) → diffuse map; and (normal, light, metallic, roughness, mask) → specular map. Splitting the batches this way lets the diffuse and specular branches focus on light and shadow structure, while the first branch focuses on final color.

Physics-inspired losses in RGB space. Because physical laws are meaningless in latent space, the physics losses are applied after the VAE decoder reconstructs the output. A diffuse shading loss computes a diffuse map by converting the gray-ball lighting into a 32×16×3 environment map, deriving UV coordinates from the normal map, and grid-sampling — no ground-truth diffuse annotation needed. A physical-based shading loss enforces the rendering equation against the prediction, and a DINO-feature perceptual reconstruction loss keeps the image content unchanged. These are combined with the V-prediction loss and weighted by 1/t.

Dataset. Over 10,000 objects with BRDF materials were filtered from Objaverse and rendered with 10 views and 10 lighting conditions each (100 images per object), with lighting sampled to favor the upper hemisphere, using point lights in 80% of cases and point plus environment maps in the remaining 20%, with environment maps drawn from over 700 Poly Haven maps. For scene data, 400 scenes (300 indoor, 100 outdoor) were downloaded from BlenderKit and 300 were kept after manual filtering. Intrinsics are extracted directly from Blender's Principled BRDF nodes rather than the composition layer.

Training and evaluation setup. 80k iterations for stage one and 90k for stage two on eight 40GB A100 GPUs, batch size 8, learning rate 1e-5, Adam optimizer, random seed fixed at 42. Evaluation used 50 held-out objects (4 camera views × 6 lighting conditions each) and 20 held-out scenes rendering 200 images, using PSNR, SSIM, and LPIPS. CFG was set to 1.0 in stage 1 and 1.5 in stage 2.

Why This Matters

Impact on research. The work argues that injecting physics-inspired regularization into a diffusion training loop can substitute for some of the large-scale, hard-to-collect structured paired data that full-image relighting normally requires, and it offers a new benchmark dataset for downstream evaluation.

Real-world applications:

• Film and video production, where a shot's lighting must be changed after capture.
• Augmented reality, where virtual objects and lighting must match a real scene.
• Digital content creation and e-commerce, where product images are re-shot under new studio lighting without physically re-shooting.
• Image editing tools that require direction- and intensity-controllable relighting rather than prompt-based guesses.

Industry relevance. Accurate specular highlights and correct shadow direction are the visual cues that make composited imagery believable; the paper positions its contributions (precise lighting control, preserved albedo and roughness, background consistency) as direct answers to the failure modes of foreground-only relighting methods used in current pipelines.

Future Directions

• Improve intrinsic prediction for challenging materials. The reported metallic scores lag behind baselines on both test sets, so better handling of metals and rough/specular surfaces is a clear open problem.
• Close the remaining synthetic-to-real gap. The authors state that existing attempts to bridge this gap remain suboptimal and that their own method relies on rendered data rather than fully realistic datasets; they also note the latent and RGB spaces are not strictly pixel-aligned and that they have not found a better alternative.
• Extend beyond a front-hemisphere lighting control. The gray-ball representation deliberately discards rear lighting to avoid interference from built-in lights, which is a trade-off worth investigating.
• Broaden scene coverage. The dataset covers indoor and outdoor scenes but excludes heavily stylized content such as fog and rain, and resolution mismatches with baselines (480×640 and 240×320 versus 768×768) complicate direct comparison — a consistent-resolution benchmark would clarify results.

Target Audience

Researchers and graduate students in computer vision and graphics working on relighting, intrinsic image decomposition, inverse and forward rendering, and diffusion-based image generation; practitioners in VFX, AR/VR, and computational photography who need controllable, physically plausible relighting; and dataset builders looking for a benchmark for full-image relighting under controlled lighting.

Authors’ abstract

Full-image relighting remains a challenging problem due to the difficulty of collecting large-scale structured paired data, the difficulty of maintaining physical plausibility, and the limited generalizability imposed by data-driven priors. Existing attempts to bridge the synthetic-to-real gap for full-scene relighting remain suboptimal. To tackle these challenges, we introduce Physics-Inspired diffusion for full-image reLight ($π$-Light, or PI-Light), a two-stage framework that leverages physics-inspired diffusion models. Our design incorporates (i) batch-aware attention, which improves the consistency of intrinsic predictions across a collection of images, (ii) a physics-guided neural rendering module that enforces physically plausible light transport, (iii) physics-inspired losses that regularize training dynamics toward a physically meaningful landscape, thereby enhancing generalizability to real-world image editing, and (iv) a carefully curated dataset of diverse objects and scenes captured under controlled lighting conditions. Together, these components enable efficient finetuning of pretrained diffusion models while also providing a solid benchmark for downstream evaluation. Experiments demonstrate that $π$-Light synthesizes specular highlights and diffuse reflections across a wide variety of materials, achieving superior generalization to real-world scenes compared with prior approaches.