RelightFormer: Feed-forward Generative Transformer for Multiview Object Relighting

AI summary

Overview

Research area: Computer Vision / Computer Graphics — generative image relighting, inverse rendering, and diffusion/flow-matching Transformers.

Technical level: Intermediate to Advanced. The paper assumes familiarity with latent diffusion, flow matching, Transformers, inverse rendering, and light-transport concepts such as the rendering equation and environment maps.

Scope: The paper presents a single feed-forward generative Transformer that relights objects given one or many input views and a target environment map, plus the large-scale relighting dataset built to train it.

What This Paper Is About

Editing the lighting of a photograph while keeping the object's shape, material, and identity unchanged is hard because the underlying scene physics (light, geometry, material) is unknown from 2D images alone. Traditional inverse-rendering pipelines try to recover those hidden properties and often fail or require hours of per-scene optimization, while existing generative relighting models mostly work on a single image and ignore multi-view cues that reveal 3D geometry. This paper's goal is to skip explicit property estimation entirely and train one feed-forward generative model that directly outputs relit images from single-view or multi-view input under a new illumination.

Key Contributions

1. RelightFormer, a large feed-forward generative Transformer for direct image relighting from single- or multi-view input, which bypasses the ill-posed steps of explicit inverse rendering.
2. A latent illumination module that adapts a video foundation model (Wan2.1) by mapping target environment maps into latent codes and injecting them into image features through cross-attention, plus permutation-invariant positional encoding (PRope) so all input views are treated symmetrically without sequential bias.
3. The Laval Objaverse Dataset (LOD), described as a large-scale, fully open-source multi-view relighting dataset rendered from 90,545 objects in Objaverse and 39,008 illuminations from the Laval Database.
4. State-of-the-art results reported for both single-view and multi-view object relighting, with strong zero-shot generalization, including real-world OLATverse data and the Stanford-ORB benchmark.

Main Findings

• Single-view relighting: RelightFormer reports sPSNR 23.80, PSNR 21.16, SSIM 0.894 and LPIPS 0.112 on the authors' held-out test set, compared with DiLightNet (17.83 / 15.95 / 0.777 / 0.232), Neural Gaffer (20.79 / 17.78 / 0.863 / 0.116), fine-tuned Neural Gaffer (22.27 / 20.25 / 0.883 / 0.095), LightSwitch (18.63 / 16.56 / 0.778 / 0.235) and Reli3D (18.52 / 15.54 / 0.814 / 0.254).
• Performance improves with more views: RelightFormer reports 24.83 / 22.62 / 0.905 / 0.084 with 16 views and 25.07 / 22.83 / 0.906 / 0.080 with 32 views, a monotonic gain the authors attribute to the model extracting underlying 3D geometry and fusing it with global incident lighting.
• Real-world zero-shot generalization (OLATverse, 42-object validation set): Under environment-map relighting RelightFormer reports sPSNR 20.48, PSNR 17.23, SSIM 0.964 and LPIPS 0.047, compared with DiLightNet (15.06 / 16.32 / 0.748 / 0.298), Neural Gaffer (18.56 / 15.98 / 0.958 / 0.048), fine-tuned Neural Gaffer (19.61 / 16.64 / 0.961 / 0.049) and LightSwitch (12.22 / 9.88 / 0.917 / 0.144).
• Rotating point-light relighting: Under spatially concentrated illumination, RelightFormer reports the best PSNR (29.39) and SSIM (0.826), with sPSNR 31.16 and LPIPS 0.209, while Neural Gaffer obtains slightly better sPSNR (21.05) and LPIPS (0.122). The authors note that perceptual details under rapidly varying specular illumination remain challenging.
• Material-stratified results: RelightFormer performs best on most material categories, including glossy, translucent and furry objects, but Neural Gaffer achieves a higher PSNR on plastic and stone, which the authors attribute to the inherent ill-posedness of image-based relighting for challenging materials.
• Novel-view relighting on Stanford-ORB: The post-trained variant RelightFormer-Post reports PSNR-H 20.99, PSNR-L 26.84, SSIM 0.952 and LPIPS 0.061 at roughly 2 minutes inference, on par with or slightly better than Reli3D (20.26 / 25.13 / 0.943 / 0.077 at ~2 min) despite Reli3D being trained with ground truth 3D object shapes. Optimization-based inverse-rendering methods obtain higher reconstruction quality (for example Neural-PBIR at 26.01 / 33.26 / 0.979 / 0.023) but require roughly 1 to 20 hours.
• 3DGS+RelightFormer trade-off: Combining 3D Gaussian Splatting reconstruction with the relighting model yields 23.25 / 30.12 / 0.967 / 0.038 at roughly 10 minutes, an intermediate quality–efficiency point.
• Ablation — illumination conditioning: Channel-wise concatenation of illumination tokens reaches sPSNR 23.51 vs. 23.27, PSNR 20.95 vs. 20.10, SSIM 0.892 vs. 0.882 and LPIPS 0.111 vs. 0.121 for the full model, showing a significant drop without cross-attention injection.
• Ablation — training view flexibility: Fixed single-view training collapses to sPSNR 16.07, PSNR 14.87, SSIM 0.768, LPIPS 0.315; sampling 1–8 views gives 22.91 / 20.17 / 0.882 / 0.121; the default 1–16 views gives 23.51 / 20.95 / 0.892 / 0.111.
• Ablation — permutation invariance: Removing permutation invariance drops the model to sPSNR 17.53, PSNR 15.98, SSIM 0.797 and LPIPS 0.269.
• Cross-dataset generalization: The model achieves competitive results on 484 pairs from the TensoIR dataset, which the authors describe as out-of-distribution relative to the training data.

Methodology in Plain English

The method takes a set of reference images of an object, their camera parameters, and a target environment map, and directly predicts the relit images without ever estimating shape or material explicitly.

The backbone is Wan2.1, a pretrained latent video diffusion model based on flow matching. A variational autoencoder compresses images and environment maps into a latent space, and a Diffusion Transformer learns a velocity field that transports Gaussian noise to the target latent along a straight-line path. Camera parameters are converted into Plücker ray maps and added to the latents so the model knows the geometry of every pixel. Reference tokens and noisy target tokens are concatenated along the sequence dimension, and after the Transformer processes them, the reference half is discarded and the remaining tokens predict the velocity field.

The distinctive piece is the latent illumination module. Because the rendering equation integrates incoming light weighted by surface properties, and cross-attention computes a weighted sum of value vectors, the authors design illumination cross-attention where image tokens act as queries and ray-embedded illumination tokens act as keys and values. Attention weights thereby implicitly learn to approximate the material–geometry term, dynamically deciding how much each light direction contributes at each spatial location. Each Transformer block combines geometry-aware multi-view self-attention with this illumination cross-attention before a feed-forward network.

Because multi-view images are an unordered set, the standard frame-indexed rotary embeddings of the video model would impose an artificial ordering. The authors replace them with PRope, which computes rotation angles from physical camera configurations and decomposes each token's transform into a projective part and a positional rotary part, giving permutation invariance by construction.

For training data, they render the Laval Objaverse Dataset: 90,545 objects, environment maps from the Laval Indoor and Laval Outdoor HDR datasets augmented with 16× horizontal rotations to reach 39,008 unique illuminations, 16 randomly sampled environment maps per object (8 indoor, 8 outdoor), 16 rendered views for training and 200 for validation/testing per object-lighting pair, with strict splits across objects, illumination conditions and camera viewpoints. The model is fine-tuned on this data at 256×256 resolution for 80K steps on 4 NVIDIA H200 GPUs with a global batch size of 128 and an initial learning rate of 1×10⁻⁴ under cosine annealing, randomly sampling 1–16 reference images per iteration. A separate novel-view post-training stage of 20K steps at a fixed learning rate of 1×10⁻⁴ adds dense supervision over 16 target viewpoints, producing RelightFormer-Post.

Why This Matters

The work argues that direct generative relighting can compete with, and in several settings beat, both inverse-rendering pipelines and single-image generative models, while doing so in minutes rather than hours. It also questions the common practice of concatenating illumination along channels, showing empirically that cross-attention injection is substantially better, and it contributes a large open-source dataset that the authors state was previously unavailable for multi-view object relighting. The authors position their latent-space pipeline as complementary to optimization-based inverse rendering rather than a replacement, noting it can naturally accommodate additional conditioning modalities such as text.

Real-world applications:

• Film production and visual effects, where lighting an object can be re-authored after capture.
• Digital content creation and asset pipelines for games and virtual production.
• Augmented and virtual reality, where virtual objects must match the lighting of a live scene from multiple viewpoints.
• E-commerce and product visualization, where a single object can be shown under many lighting conditions without re-shooting.

Industry relevance: The reported inference cost — roughly 2 minutes for the feed-forward variant and roughly 10 minutes for the 3DGS+RelightFormer combination, against 1 to 20 hours for optimization-based methods — is the kind of quality–efficiency trade-off that matters for interactive creative tools and production pipelines. The open dataset and code release (https://github.com/vLAR-group/RelightFormer) lower the barrier for others to build on the approach, and the paper appears in SIGGRAPH Asia 2026 Conference Papers.

Future Directions

• Specular and rapidly varying illumination: The authors acknowledge that perceptual details under quickly changing specular illumination remain challenging, and Neural Gaffer still slightly leads on sPSNR and LPIPS in the rotating point-light setting.
• Challenging materials: Plastic and stone remain cases where Neural Gaffer achieves higher PSNR, and the paper attributes this to the inherent ambiguity of image-based relighting — an open problem for material-specific fidelity.
• Closing the gap with optimization-based inverse rendering: Methods such as Neural-PBIR, NVDIFFREC, NVDIFFREC-MC, InvRender, NeRFactor and PhySG still report higher PSNR and LPIPS on Stanford-ORB at the cost of hours of optimization, leaving headroom for feed-forward approaches.
• Extending conditioning modalities: The authors explicitly note that the latent-space formulation can naturally accommodate additional conditioning such as text, suggesting a route toward more controllable relighting.

Target Audience

Researchers and practitioners in generative computer vision, differentiable and inverse rendering, and neural rendering; graphics engineers building relighting or asset-authoring tools for film, games, AR/VR and e-commerce; and anyone working on multi-view or video-diffusion models who is interested in how permutation-invariant encoding and cross-attention conditioning can be adapted from a video foundation model to a non-sequential, unordered input setting.

Authors’ abstract

Image relighting is traditionally tackled via complex inverse rendering pipelines, which suffer from ill-posed optimization, or single-image generative models that ignore crucial multi-view cues necessary for understanding 3D geometry and material interactions. To address these limitations, we introduce a feed-forward generative Transformer for direct single- and multi-view image relighting that entirely bypasses explicit intrinsic property estimation. Adapted from a video foundation model, our architecture features a latent illumination module that dynamically injects target environment maps into spatial features via cross-attention. Furthermore, we employ permutation-invariant positional encodings to symmetrically process unordered multi-view inputs without sequential bias. To train this robust data-driven model, we construct the massive Laval Objaverse Dataset (LOD), comprising 90K objects and 39K unique illuminations. Extensive experiments demonstrate state-of-the-art visual quality, photorealistic relighting quality, and strong zero-shot generalization across single-view, multi-view, and novel-view relighting tasks.