UniLumos: Fast and Unified Image and Video Relighting with Physics-Plausible Feedback

AI summary

Overview

Research area: Computer vision and generative modeling, specifically image and video relighting with diffusion/flow-matching models.

Technical level: Advanced. The paper assumes familiarity with diffusion and flow-matching generative models, latent-space representations, velocity-field training, path consistency distillation, and geometry estimation (depth and surface normals).

Scope: The paper proposes a single flow-matching framework, UniLumos, that relights both images and videos under textual, image, or video lighting conditions while enforcing geometric (depth/normal) consistency in RGB space, and introduces a structured lighting annotation protocol and a corresponding vision-language-model-based benchmark.

What This Paper Is About

Relighting means changing the illumination in an image or video while keeping the subject's geometry, appearance, and content unchanged. Existing diffusion-based relighting methods operate in a semantic latent space where "looking similar" does not guarantee physical correctness, so they frequently produce overexposed highlights, misaligned shadows, and incorrect occlusions. UniLumos aims to fix this by bringing explicit geometric supervision (depth and normal maps) from the RGB output space into a flow-matching backbone, while also enabling fast few-step inference.

Key Contributions

1. Unified relighting with physics-plausible feedback. UniLumos is a single framework for both image and video relighting that injects RGB-space depth and normal feedback into a flow-matching backbone (built on Wan2.1), explicitly aligning lighting effects with scene structure. Path consistency learning is used so this supervision remains viable under few-step training and inference.

2. A structured six-dimensional illumination annotation protocol. The protocol covers direction, intensity, color temperature, light source type, temporal dynamics, and optical phenomena. Attributes are generated automatically by vision-language models such as Qwen2.5-VL and embedded into the conditioning signal for fine-grained control.

3. LumosData and LumosBench. LumosData is a scalable data pipeline that extracts relighting pairs from real-world videos (using BiRefNet for masks and IC-Light for degradation), producing roughly 110K high-quality video pairs plus 1.2M additional relit images. LumosBench is a disentangled, attribute-level benchmark that uses large vision-language models to automatically score whether generated outputs match the intended lighting across each of the six attributes.

4. Extensive validation including a 20x speedup. Experiments report state-of-the-art relighting quality with improved physical consistency and a 20x speedup for both image and video relighting.

Main Findings

• Image relighting quality. UniLumos reports PSNR 26.719, SSIM 0.913, and LPIPS 0.089, the best of the compared image methods (SwitchLight: 20.483 / 0.901 / 0.094; DiLightNet: 21.894 / 0.860 / 0.131; IC-Light: 24.316 / 0.884 / 0.108; SynthLight: 25.572 / 0.905 / 0.102).

• Lumos consistency on images. UniLumos scores an average Lumos Score of 0.912 with a Dense L2 Error of 0.103, versus 0.791 / 0.214 for the next-best method (SynthLight).

• Video relighting quality. UniLumos reports PSNR 25.031, SSIM 0.891, and LPIPS 0.109, compared with IC-Light per frame (20.132 / 0.851 / 0.133), Light-A-Video + CogVideoX (19.851 / 0.859 / 0.124), and Light-A-Video + Wan2.1 (20.784 / 0.876 / 0.129).

• Temporal consistency. On the R-Motion metric (lower is better, per the table), UniLumos records 1.436, better than IC-Light per frame (2.437), Light-A-Video + CogVideoX (1.784), and Light-A-Video + Wan2.1 (1.582).

• Video Lumos consistency. UniLumos achieves an average Lumos Score of 0.871 and Dense L2 Error of 0.147, versus 0.672 / 0.432 (IC-Light per frame), 0.641 / 0.383 (Light-A-Video + CogVideoX), and 0.682 / 0.371 (Light-A-Video + Wan2.1).

• Efficiency. Under a standardized setting generating 49-frame videos at 480p, UniLumos completes generation over 20 times faster than prior methods, which the authors attribute to geometry-free inference and few-step generation.

• Physics feedback ablation. Removing all feedback drops video metrics to PSNR 21.433, SSIM 0.862, LPIPS 0.139, R-Motion 1.473, Lumos Score 0.859, Dense L2 Error 0.297. Removing only depth gives 23.472 / 0.883 / 0.118 / 1.443 / 0.870 / 0.265; removing only normals gives 22.115 / 0.874 / 0.123 / 1.446 / 0.863 / 0.173. The authors note that omitting normal supervision hurts more than omitting depth, concluding that surface orientation matters more than distance for light-shadow interactions.

• Path consistency ablation. Removing path consistency yields PSNR 25.317, SSIM 0.902, LPIPS 0.113, R-Motion 1.438, Lumos Score 0.875, Dense L2 Error 0.153. The paper describes this as a minor drop in physical metrics while noting it maintains competitive SSIM and LPIPS, justifying inclusion for its efficiency benefit in few-step regimes.

• Training modality ablation. Training on video only gives PSNR 22.487, SSIM 0.863, LPIPS 0.119, R-Motion 1.487, Lumos Score 0.857, Dense L2 Error 0.173. Training on images only gives 24.471 / 0.872 / 0.123 / 2.429 / 0.841 / 0.182. The unified model (25.031 / 0.891 / 0.109 / 1.436 / 0.871 / 0.147) balances both, with image-only training sacrificing temporal smoothness.

• Generalization check. Additional evaluation was run on the public object-centric relighting benchmarks StanfordOrb and Navi, which the paper states are completely disjoint from the training data.

Methodology in Plain English

The model starts from Wan2.1-T2V-1.3B-480P, a pretrained video generation model that works by flow matching: it learns a velocity field that transports noise into data along a straight interpolated path. UniLumos adds new trainable projection and fusion layers, all initialized to zero so the pretrained behavior is preserved at the start of training.

For supervision, the researchers need pairs of "degraded" and "real" videos. Their LumosData pipeline takes real videos, extracts subject masks with BiRefNet, applies a pretrained relighter (IC-Light) to the subject region under many lighting prompts and light directions, and fills the background with pure color or Gaussian noise rather than doing full inpainting. The result is a training tuple of (degraded video, background video, mask, lighting condition) that maps to the original real video.

The distinctive step is the physics feedback. After the model decodes a predicted latent into RGB frames, a frozen dense estimator (Lotus) extracts depth and surface normals from those frames. These are compared against pseudo-ground-truth depth and normals from the reference input, and the relative L2 error is used as a loss. Crucially, depth and normals are used only as training-time supervision, never as inference inputs, so the test pipeline stays simple and does not propagate estimator errors.

Because RGB-space supervision needs high-quality outputs that normally require full multi-step denoising, the authors adopt path consistency learning, which enforces that a large integration step matches two smaller ones. This lets geometric feedback remain useful while training and sampling with very few steps. Training mixes three losses: a standard flow-matching loss, the path consistency loss, and the physics loss, with fixed weights of 1.0, 0.1, and 0.1. Each batch is split 80/20: 20 percent computes the path consistency loss (three forward passes, one backward pass), and of the remaining 80 percent, half receives the physics-guided loss.

Optimization used AdamW at a learning rate of 1e-5, batch size 8, 5,000 iterations, on 8 NVIDIA H20 GPUs with 96GB RAM. Testing used internal samples split evenly between images at 768x512 and videos at 480p (832x480) with 49 frames each.

Why This Matters

The paper attacks a well-known weakness of generative relighting: latent-space similarity is not physical correctness. By turning geometry signals into a training-time feedback loop, it shows a practical route to lighting that respects scene structure without complicating inference. It also argues that FID/LPIPS-style metrics miss lighting-specific errors, and responds with an interpretable, attribute-level benchmark scored by vision-language models.

Real-world applications mentioned in the paper:

• Film production, where seamless lighting integration is critical to visual fidelity
• Gaming, where lighting shapes how environments and characters are perceived
• Augmented reality, where inserted content must match scene illumination
• Real-time relighting and scalable deployment scenarios, enabled by the reported efficiency gains

Industry relevance: The 20x speedup claim and the geometry-free inference pipeline make this relevant to production systems where latency and cost matter. The data pipeline reduces dependence on expensive hardware captures or manual annotation by deriving relighting pairs from existing real-world video (built on Panda70M), which matters for teams without paired-light-stage data.

Future Directions

• Extending the physics feedback signals. The appendix discusses why albedo, shadow masks, and material properties were rejected; the open question is whether more informative physical signals can be incorporated without reintroducing ambiguity, domain sensitivity, or dataset dependence.

• Going beyond training-time-only geometry. The authors chose depth and normals as supervision rather than inference inputs. Whether the trade-off between accuracy and pipeline simplicity changes with better estimators is left open.

• Scaling the data pipeline. LumosData relies on a pretrained 2D relighter (IC-Light) to synthesize degradations. Whether the model's quality ceiling is set by that relighter, and how the pipeline scales past roughly 110K video pairs and 1.2M relit images, is not resolved.

• Benchmark scope. LumosBench uses vision-language models to score six lighting attributes automatically. The reliability of VLM-based scoring as an evaluation standard, and whether it should be complemented by stronger physical metrics, is an open question.

• Limitations. The paper references a limitations discussion in Appendix E, but that content is not included in the provided material, so the specific limitations the authors identify are not reported here.

Target Audience

Researchers and engineers working on diffusion-based image and video editing, relighting, and inverse-rendering-adjacent generative modeling. It is most useful to readers already comfortable with flow matching, latent diffusion, and consistency/distillation techniques who want to understand how geometric supervision can be injected into a generative relighting pipeline. Practitioners building production relighting or lighting-integration tools will also benefit from the efficiency results and the structured annotation protocol, while benchmark-focused researchers will find the attribute-level evaluation design relevant.

Authors’ abstract

Relighting is a crucial task with both practical demand and artistic value, and recent diffusion models have shown strong potential by enabling rich and controllable lighting effects. However, as they are typically optimized in semantic latent space, where proximity does not guarantee physical correctness in visual space, they often produce unrealistic results, such as overexposed highlights, misaligned shadows, and incorrect occlusions. We address this with UniLumos, a unified relighting framework for both images and videos that brings RGB-space geometry feedback into a flow matching backbone. By supervising the model with depth and normal maps extracted from its outputs, we explicitly align lighting effects with the scene structure, enhancing physical plausibility. Nevertheless, this feedback requires high-quality outputs for supervision in visual space, making standard multi-step denoising computationally expensive. To mitigate this, we employ path consistency learning, allowing supervision to remain effective even under few-step training regimes. To enable fine-grained relighting control and supervision, we design a structured six-dimensional annotation protocol capturing core illumination attributes. Building upon this, we propose LumosBench, a disentangled attribute-level benchmark that evaluates lighting controllability via large vision-language models, enabling automatic and interpretable assessment of relighting precision across individual dimensions. Extensive experiments demonstrate that UniLumos achieves state-of-the-art relighting quality with significantly improved physical consistency, while delivering a 20x speedup for both image and video relighting. Code is available at https://github.com/alibaba-damo-academy/Lumos-Custom.