LuxRemix: Lighting Decomposition and Remixing for Indoor Scenes

AI summary

Overview

Research area: Computer Vision — indoor scene relighting, lighting decomposition, and 3D scene representation.

Technical level: Advanced. The paper assumes familiarity with diffusion transformers, LoRA fine-tuning, multi-view diffusion U-Nets, HDR tone mapping, and 3D Gaussian splatting.

Scope: This paper presents LuxRemix, a three-stage pipeline that decomposes indoor scene lighting into individually controllable light sources from a single multi-view capture and encodes them in a real-time relightable 3D Gaussian splatting representation.

What This Paper Is About

When an indoor scene is photographed or reconstructed in 3D, the lighting of the moment is baked into the result, so photographers and artists cannot later switch a lamp off, change its color, or dim it without physically reshooting. LuxRemix attacks this problem by factorizing a multi-view indoor capture into separate contributions from each light source plus an ambient term, propagating that factorization consistently across all views, and then storing it in a fast 3D representation where individual lights can be edited interactively.

Key Contributions

1. A single-image lighting decomposition model (LuxRemix-SV) that factorizes complex indoor illumination into ambient lighting and individual one-light-at-a-time (OLAT) sources using a fine-tuned diffusion transformer.
2. A multi-view lighting harmonization method (LuxRemix-MV) that propagates a decomposition performed on sparse reference views consistently across all captured viewpoints using geometric constraints.
3. A relightable 3D Gaussian splatting representation with per-light HDR RGB coefficients, enabling real-time interactive control over the intensity, color, and on/off state of individual near-field light sources from any viewpoint.
4. A large-scale synthetic dataset of over 12,000 generated indoor scenes with ground-truth per-light decompositions, which the authors are publicly releasing.

Main Findings

• Single-image decomposition quality: On 30 held-out synthetic test scenes, LuxRemix-SV reaches 27.676194 PSNR, 0.898317 SSIM, and 0.081680 LPIPS, versus ScribbleLight at 14.393476 PSNR / 0.395403 SSIM / 0.687826 LPIPS and Qwen-Image at 18.233805 PSNR / 0.713984 SSIM / 0.237084 LPIPS.
• Ablation on mask conditioning: Replacing the final channel-wise token addition design with a FLUX-token in-context LoRA gives 25.198421 PSNR / 0.864664 SSIM / 0.101492 LPIPS, and using a U-Net-based latent diffusion model instead gives 27.134128 PSNR / 0.857183 SSIM / 0.099066 LPIPS — both below the full model.
• Multi-view harmonization is necessary: LuxRemix-MV scores 30.762927 PSNR / 0.866890 SSIM / 0.090679 LPIPS, outperforming both processing each view independently with LuxRemix-SV (25.137302 / 0.807102 / 0.148779) and the mask-guided editing variant LuxRemix-MV-Edit (26.371356 / 0.793793 / 0.136203).
• HDR handled through exposure bracketing: The harmonization stage runs three times per lighting condition at high/medium/low exposure (matching EV0, EV-2, and EV-4 of the target HDR OLAT), then merges the results using the Debevec and Malik method.
• Dataset scale and cost: The synthetic dataset contains over 12,000 procedurally generated indoor scenes, up to six controllable lights per scene, roughly 49,600 equirectangular views, about 2,800 GPU hours on NVIDIA A100s, and approximately 9 TB of data.
• Storage-efficient rendering choice: Storing 1,000 equirectangular 360° images (2048×1024) takes 6.8 GB and 27,000 seconds to render, which is 19% of the render time and 40% of the storage of 20,000 pre-rendered perspective images (512×512, 140,000 seconds, 17.2 GB).
• Training configuration: LuxRemix-SV uses LoRA rank 32 for 3,000 iterations at batch size 192 with the Prodigy optimizer, on 48 NVIDIA A100-40GB GPUs in about 12 hours. LuxRemix-MV uses full-parameter U-Net fine-tuning for 30,000 iterations (15,000 at 4 views, 10,000 at 8 views, 5,000 at 15 views) with AdamW, on 48 A100-40GB GPUs in about 28 hours.
• Real-world evaluation setup: Real-world scenes use standard SfM for camera poses and typically 32–96 images covering the target lights; qualitative results draw on captures from Zip-NeRF and VR-NeRF.
• Limitations acknowledged: Models are trained only on static synthetic indoor scenes, the limited light-source diversity biases decompositions toward light cones over diffuse lighting, and distant global illumination editing via HDRIs is not supported. The paper does not report quantitative results for outdoor, dynamic, or HDRI-relighting cases.

Methodology in Plain English

The authors build the system in three stages.

First, they create training data at scale. They take over 12,000 procedurally generated indoor 3D models and add procedurally generated lights, including ceiling, wall, floor, and table lamps plus environment lighting. Light colors are sampled from black-body color temperatures and then jittered with 10% HSV variation. Each scene is rendered with Blender's Cycles path tracer with all lights on, with each single light on alone (OLAT), and with ambient only. To avoid the cost of pre-rendering every possible camera angle, they store four equirectangular 360° HDR images per scene at roughly eye height and cut perspective training views out of them on the fly, using depth maps and light masks to pick views where the target light is actually visible.

Second, they fine-tune a pretrained image-editing diffusion transformer with LoRA so it can perform two related tasks from a text instruction plus a light mask: isolate a single selected light at a requested brightness level, and turn off only the selected light while leaving the rest unchanged. The mask is encoded by a single-layer MLP and added channel-wise to the input image latents, which the authors find works better than concatenating image and mask as tokens. Training pairs are generated on the fly, including synthesized combinations of multiple OLAT images to diversify the lighting.

Third, because single images cannot keep a decomposition consistent across viewpoints, they fine-tune a pretrained multi-view diffusion U-Net that takes the original views, the sparsely decomposed views, Plücker ray embeddings, and reference view masks, and produces harmonized decompositions for every view. Running this three times per lighting condition and merging the exposures yields HDR per-light images.

Finally, they pretrain a standard 3D Gaussian splatting model on the original images, freeze its geometry and appearance, then add per-light HDR RGB coefficients to each Gaussian and optimize those against the decomposed multi-view images in linear HDR space with differentiable tone mapping. At render time these per-light contributions are linearly recombined with user-chosen light intensities and colors, preserving the real-time rendering speed of Gaussian splatting.

Why This Matters

Impact on research: LuxRemix is described as the first method for multi-view lighting decomposition and remixing in indoor scenes. It bridges a gap the authors identify between single-image lighting methods that lack 3D consistency and multi-view relighting methods that require controlled multi-light captures or only produce global relighting without per-light control. The released dataset of over 12,000 synthetic indoor scenes with ground-truth per-light decompositions is intended to support follow-up work.

Real-world applications:

• Virtual production and filmmaking, where lighting configurations need to be revised after a shoot without recapturing the set.
• Photography post-production, giving per-light control over intensity and color in already-captured indoor images.
• Interactive architectural and interior design review, letting clients toggle individual lamps in a reconstructed room in real time.
• Immersive and mixed-reality experiences that need temporally and spatially consistent lighting across many viewpoints.

Industry relevance: The work originates from Meta Reality Labs with academic collaboration from the University of Toronto, and targets workflows where fine-grained, interactive control over lighting is a practical requirement rather than a research curiosity. Real-time performance is a design goal, since the relighting is built on 3D Gaussian splatting and demonstrated with recordings from a modified Splatfacto renderer in the Nerfstudio viewer.

Future Directions

• Extending beyond static synthetic indoor scenes to outdoor and dynamic scenes, which the authors explicitly identify as outside the current training distribution.
• Reducing the bias introduced by limited light-source diversity in the training data, which currently favors light cones over more diffuse lighting configurations.
• Supporting editing of distant global illumination via HDRIs, which the current model does not handle and which the authors leave for future work.
• Evaluating the approach on larger real-world multi-view captures and characterizing how single-view versus multi-view consistency behaves in harder capture conditions, which the supplement touches on.

Target Audience

Researchers and practitioners in computer vision, graphics, and computational photography working on relighting, inverse rendering, diffusion-based image editing, or 3D Gaussian splatting. It is also relevant to technical artists and virtual production engineers interested in post-capture control of individual light sources, though the method's reliance on multi-view capture and diffusion model training makes the core technical content most useful to readers with a graphics and generative-modeling background.

Authors’ abstract

We present a novel approach for interactive light editing in indoor scenes from a single multi-view scene capture. Our method leverages a generative image-based light decomposition model that factorizes complex indoor scene illumination into its constituent light sources. This factorization enables independent manipulation of individual light sources, specifically allowing control over their state (on/off), chromaticity, and intensity. We further introduce multi-view lighting harmonization to ensure consistent propagation of the lighting decomposition across all scene views. This is integrated into a relightable 3D Gaussian splatting representation, providing real-time interactive control over the individual light sources. Our results demonstrate highly photorealistic lighting decomposition and relighting outcomes across diverse indoor scenes. We evaluate our method on both synthetic and real-world datasets and provide a quantitative and qualitative comparison to state-of-the-art techniques. For video results and interactive demos, see https://luxremix.github.io.