Relighting as a Probe of Visual Priors via Augmented Latent Intrinsics

AI summary

Overview

Research area: Computer vision — image-to-image relighting, visual representation learning, and generative (diffusion-based) image editing.

Technical level: Intermediate. The paper assumes familiarity with diffusion decoders, latent-space representations, and standard image-quality metrics, but its central argument is conceptual: which visual features are useful for physically grounded image transformations.

Scope in one sentence: The paper uses image-to-image relighting as a controlled diagnostic to test whether pretrained visual encoders preserve the physical information needed for illumination transfer, and introduces a framework that combines dense pixel-aligned features with latent intrinsic representations to balance semantic context against photometric fidelity.

What This Paper Is About

Relighting an image means moving the light from one picture onto the content of another, which requires knowing which brightness changes come from illumination and which come from the surface itself. The authors argue that this task is a good "probe" of visual representations, because recognition-oriented features are trained to be invariant to exactly the lighting, texture, and local appearance cues that relighting needs. Their goal is to systematically test how different pretrained encoders affect relighting quality, and to build a model that keeps both semantic context and fine photometric detail.

Key Contributions

1. Proposing generative relighting as a probe of physically grounded visual priors. The authors define "relightability" operationally as the ability to support (i) photometric consistency under light transfer, (ii) material fidelity for view- and material-dependent effects, and (iii) illumination robustness while retaining dense per-pixel detail, then use it to quantify what visual encoders encode about illumination, material, and geometry.

2. Introducing Augmented Latent Intrinsics (ALI). ALI is a controlled image-to-image relighting framework built on LumiNet that fuses frozen visual encoder features into latent intrinsic representations through a lightweight fusion adapter, aligned to a diffusion decoder via a three-stage training schedule. It exposes and balances the semantic–photometric trade-off.

3. A systematic analysis of encoder families. Experiments comparing CLIP, DINOv2, DINOv3, MAE, and RADIOv2.5 within the same fixed backbone, data, and decoder objective show that strong semantic encoders can fail at relighting, while dense pixel-aligned representations better preserve the physical cues required for light transfer.

4. State-of-the-art results on MIIW. ALI reports state-of-the-art performance on the MIIW benchmark among open-sourced diffusion-based methods, improving RMSE by 4.5% and SSIM by 4.9% over LumiNet, with the largest gains on glossy and specular materials.

Main Findings

• Stronger semantics can hurt relighting. Features from encoders optimized for pure semantics show relighting quality (PSNR) inversely correlated with recognition performance (ImageNet-1K linear probing as reported in the original papers). The paper frames this as a semantic–photometric trade-off between abstraction and physical fidelity.

• Reconstruction objectives beat contrastive ones here. MAE and RADIOv2.5 consistently outperform CLIP and DINO-based features in the ablation. In the Stage I&II comparison, RADIOv2.5 reaches RMSE 0.1260, LPIPS 0.2440, PSNR 18.3426, and SSIM 0.5958, versus CLIP at RMSE 0.1627, LPIPS 0.3153, PSNR 16.1333, SSIM 0.5039. MAE reaches RMSE 0.1286, LPIPS 0.2554, PSNR 17.9861, SSIM 0.4852.

• Feature density alone is not the answer. Going from DINOv2 to DINOv3 improves metrics (for example, Stage I&II RMSE from 0.1686 to 0.1654 and SSIM from 0.4815 to 0.5299), but the authors state this is insufficient when the underlying objective promotes invariance to texture, lighting, and local appearance.

• Gains concentrate on hard materials. Semantic augmentation in Stage I improves every material category, with the largest gains on Glossy, Metallic, and Specular regions. For Metallic, SSIM moves from 0.4164 (LumiNet) to 0.4855 (Stage I&II); for Specular, from 0.3778 to 0.4423.

• Stage-wise ablation shows progressive improvement. On MIIW in-scene evaluation, ALI reports PSNR 18.872, RMSE 0.119, LPIPS 0.213, and SSIM 0.671, against LumiNet at 18.568, 0.123, 0.228, and 0.645. The authors describe the result as consistent performance and the second-best LPIPS without privileged labels.

• Human evaluation favors ALI on lighting. In the in-the-wild user study, ALI scores 0.93 for lighting alignment and 0.90 for lighting realism, versus LumiNet at 0.42 and 0.75, and Latent-Intrinsics at 0.13 and 0.28. Latent-Intrinsics leads on identity preservation at 0.68, with ALI at 0.63.

• Stage III improves perceptual quality. In the stage-wise study, lighting realism rises from 0.25 (LumiNet) to 0.35 (Stage I), 0.68 (Stage I&II), and 0.89 (all stages), while identity preservation rises to 0.96 with all stages.

• Pixel metrics are not the whole story. The paper notes that PSNR, SSIM, and RMSE correlate imperfectly with perceived lighting quality and often emphasize low-frequency tone consistency over directional shadows or specular effects. Methods trained only on MIIW, such as SA-AE and Latent-Intrinsic, can achieve stronger pixel-level scores — for example Latent-Intrinsic at RMSE 0.222 and SSIM 0.571 on color-corrected cross-scene evaluation — which the authors attribute to this metric bias.

• Intrinsics and lighting stay disentangled. Holding intrinsic features fixed while varying the lighting input produces coherent relighting, and interpolating the extrinsic code yields smooth lighting transitions within the same scene.

• Consistency with concurrent work. The authors note that RAE (Zheng et al., 2025) similarly found an MAE-based representation paired with a diffusion decoder outperforms a DINO-based counterpart for image reconstruction under the same decoding architecture.

Methodology in Plain English

The authors start from LumiNet, a latent-intrinsic relighting system in which an encoder splits an image into lighting-invariant intrinsic features and a global lighting embedding, and a decoder recombines intrinsics from one image with lighting from another. They deliberately hold the backbone, training data, and decoder objective fixed, and vary only which frozen pretrained encoder is used to augment the representation — this is what makes the setup a controlled probe rather than a new architecture race.

The method, ALI, runs in three stages:

• Stage I — Augmenting latent intrinsics. Features from a frozen visual encoder (for example RADIOv2.5) are extracted from several intermediate layers, upsampled to input resolution, and concatenated into a pixel-wise "hypercolumn" descriptor. A learnable projection layer aligns this with the intrinsic features, and the result is added to them. Only the relighting encoder and projection layers are trained, using a reconstruction loss, a hyperspherical regularization loss, and an improved intrinsic-invariance loss that pulls a scene's intrinsic features toward their mean across different lighting conditions.

• Stage II — Aligning the generative decoder. With the encoder and projection frozen, the LumiNet diffusion decoder is fine-tuned to condition on the new augmented intrinsics, using the standard DDPM denoising score-matching loss.

• Stage III — Self-refinement. Because paired real-world data is scarce, the authors generate a "Lighting Zoo" of pseudo-relit image pairs by transferring illumination between randomly sampled images within a batch, then fine-tune the decoder on these pairs, periodically mixing in same-image reconstructions to preserve content fidelity.

Training uses AdamW with a learning rate of 4×10⁻⁵ at 512×512 resolution and an effective batch size of 64, on 8 NVIDIA A6000 Ada 48GB GPUs. Data comes from MIIW (985 scenes under 25 lighting conditions) and BigTime (460 scenes under 20–50 natural illumination conditions). Stage I uses a scene-aware batch sampler, Stage II and III use random sampling across scenes, and Stage III draws 1,000 randomly sampled scenes and includes 10% identity relighting samples. The Lighting Zoo consists of roughly 6,000 images from IIW, RealEstate-10K, and DL3DV, with seven relit versions generated per scene. Stage I runs 4 epochs (about 8 hours on a single GPU) and Stages II and III run 2 epochs each (about 10 hours each).

At inference, intrinsics come from a content image and the lighting embedding from a separate illumination image, so the model generalizes to unpaired relighting. The authors note that although LumiNet's bypass decoder is used by default to preserve identity, all feature-fusion experiments disable it to isolate the effect of the augmented representation.

Why This Matters

Impact on research. The paper reframes relighting from a graphics task into a diagnostic for representation learning. Its central claim — that recognition-oriented features suppress the illumination-sensitive detail needed for physical image transformation — suggests that benchmark accuracy on semantic tasks is a poor proxy for whether a representation supports physically meaningful generation. The finding that prior design can substitute for supervision scale also points toward a more data-efficient route than large synthetic datasets, dense physical annotations, or explicit inverse-graphics supervision.

Real-world applications mentioned or implied:

• Augmented reality, where virtual objects must be lit consistently with a real scene.
• Computational photography, including post-capture lighting edits.
• Digital content creation and visual effects, enabling realistic lighting changes without synthetic data or privileged supervision.
• Reflectance editing, shadow manipulation, view-consistent generation, and material decomposition, which the authors name as extensions of the same probing framework.

Industry relevance. Any pipeline that selects a pretrained visual backbone for a graphics or editing model faces the trade-off this paper quantifies. The practical takeaway is that under a fixed decoder and fixed dataset, swapping CLIP or DINO for a dense, pixel-aligned encoder such as RADIOv2.5 measurably changes material fidelity — particularly on metallic and specular surfaces — which matters for product visualization, real-estate imagery, and AR content.

Future Directions

• Extending the probe to related tasks. The authors suggest applying the same diagnostic framework to reflectance editing, shadow manipulation, view-consistent generation, and material decomposition to clarify which representation properties support physical reasoning.

• Combining image-only relighting with 3D structure or physical supervision. The limitations section notes ALI can shift fine details, conflate surface color with illumination, and hallucinate shadows from learned priors rather than explicit geometry — problems the authors attribute to the absence of 3D structure or physical supervision.

• Going beyond a single backbone family. The paper argues relightability depends less on a specific backbone family than on the inductive biases imposed by reconstruction, spatial alignment, and feature granularity, leaving open which combination of these biases is optimal.

• Closing the metric gap. Because standard full-reference metrics correlate imperfectly with perceived lighting quality, the field needs better ways to measure directional shadows, specular effects, and material fidelity, which the paper currently addresses through qualitative comparison and human evaluation.

Target Audience

Researchers and graduate students in computer vision and generative modeling who work on relighting, intrinsic image decomposition, or diffusion-based image editing. It is also useful for practitioners selecting pretrained visual backbones for graphics or AR pipelines, and for representation-learning researchers interested in what self-supervised and contrastive objectives discard. Readers need some familiarity with diffusion decoders and latent-space representations to follow the training stages, but the paper's core argument — that semantic strength and physical usefulness diverge — is accessible without deep implementation knowledge.

Authors’ abstract

Image-to-image relighting requires representations that separate illumination from scene properties while preserving dense geometry, material, and photometric cues. We use this task as a probe of visual priors: unlike recognition tasks that reward invariance, relighting tests whether visual features retain the information needed for light transfer. Through a controlled generative relighting framework, we find that strong semantic encoders can degrade relighting quality, exposing a semantic--photometric trade-off between abstraction and physical fidelity. We introduce Augmented Latent Intrinsics (ALI), which balances this trade-off by fusing dense, pixel-aligned visual features into a latent-intrinsic relighting model and refining it with self-supervision on unlabeled real image pairs. ALI improves relighting quality, especially on glossy, metallic, and transparent materials, and demonstrates that generative relighting is an effective tool for quantifying what visual encoders encode about the physical world.