FlashRelight: Portrait Video Relighting with Dynamic Lighting

AI summary

Overview

Research area: Computer vision — portrait video relighting using generative video diffusion models. Sub-areas include intrinsic-style decomposition (delighting), controllable video generation, and lighting representation/transfer.

Technical level: Advanced. The method builds on a pre-trained DiT-based video generative model (Wan2.1-1.3B), latent diffusion training schedules, LoRA fine-tuning, environment-map tokenization, and cross-attention conditioning. Readers should already understand diffusion models and video generation to get the most from it.

Scope (one sentence): The paper proposes a two-stage delighting-then-relighting framework that edits the illumination of portrait videos under time-varying lighting, driven by environment-map sequences, lighting text, or a lighting reference video, and trained on a hybrid of static OLAT data plus generated dynamic-lighting videos.

What This Paper Is About

Portrait video relighting means changing how a person is lit in an existing video while keeping their identity, expressions, and motion intact. Most existing methods assume the light stays roughly fixed, but real footage often has strobing flashes, drifting shadows, moving highlights, and color changes, which are hard to synthesize and hard to obtain training data for. FlashRelight addresses this by splitting the problem into two stages — first strip the original lighting away, then generate the new, time-varying lighting — and by training on a cheap mix of static one-light-at-a-time (OLAT) captures and synthetic dynamic-lighting videos produced by a video generator.

Key Contributions

1. A two-stage framework for dynamic portrait video relighting. A delighting model suppresses the source illumination to produce an approximately illumination-neutral video, and a DiT-based relighting model then synthesizes the target dynamic lighting on top of it, preserving identity, motion, and temporal coherence.

2. Three complementary lighting control modalities in one model. Envmap sequences give explicit frame-level lighting control (with token-level temporal alignment in cross-attention), dynamic lighting text gives semantic control through a T5-encoded text branch, and lighting reference videos give direct visual guidance for cross-video lighting transfer via channel-wise latent concatenation.

3. A scalable hybrid supervision strategy. The Hybrid Dynamic Portrait Video Dataset combines Static OLAT Data with Dynamic Gen Data, avoiding the need for large-scale dynamic light-stage capture or fully rendered paired datasets, and covering both controlled illumination variation and realistic portrait motion.

4. Strong quantitative results across two of the three settings plus an ablation. The method leads on all five metrics for envmap-sequence relighting and on FID and ID similarity for text-based relighting, with an ablation confirming that the two data sources are complementary.

Main Findings

• Envmap-sequence relighting (Table 1): best on every metric. FlashRelight reaches LE 0.0584, LPIPS 0.1635, ID 0.8097, PSNR 23.1099, and tLPIPS 0.1303, versus DiffusionRenderer at 0.1120 / 0.3781 / 0.6143 / 17.3683 / 0.2458, Relighting-Cai at 0.1585 / 0.4051 / 0.5193 / 14.2131 / 0.2684, and Lite2Relight at 0.1423 / 0.4154 / 0.5653 / 15.6086 / 0.2721.

• Text-based relighting (Table 2): best FID and identity, comparable smoothness. FlashRelight achieves FID 31.16 and ID 0.7973, against Light-A-Video at 51.78 / 0.7126, Light-X at 66.27 / 0.6762, and IC-Light at 134.98 / 0.4332. Its tLPIPS of 0.1417 is marginally above Light-A-Video's 0.1395, which the paper describes as temporal stability comparable to the strongest video-based baseline.

• Reference-video transfer is evaluated qualitatively only. Because paired ground truth is unavailable for that setting, the paper reports no metrics and instead shows visual examples of illumination dynamics transferred from a reference portrait video while source identity and motion are preserved.

• Hybrid data beats either source alone (Table 3, combined evaluation across all three settings). OLAT-only gives FID 87.9401 / ID 0.5538 / tLPIPS 0.2380; Gen-only gives 86.9547 / 0.5470 / 0.2543; the full hybrid gives 55.8078 / 0.6734 / 0.1906.

• Frame-wise baselines flicker. Lite2Relight, Relighting-Cai, and IC-Light are applied per frame and show unstable illumination, while DiffusionRenderer (also video-based) improves quality but does not consistently follow fine-grained temporal lighting evolution.

• Inference is substantially cheaper than the compared video baseline. On a single NVIDIA H20 for 49-frame clips at 832×480, FlashRelight takes 185.9 s total with 15.23 GiB peak reserved memory (87.5 s / 15.17 GiB for delighting, 98.4 s / 15.23 GiB for relighting at 50 steps each), versus 503.3 s and 44.22 GiB for the evaluated DiffusionRenderer configuration at 15 steps.

Methodology in Plain English

The system is built as a pipeline rather than a single network.

Stage one — delighting. A model takes the source video and suppresses its original lighting and shadows. The authors are explicit that this does not have to be a physically exact decomposition; the output only needs to be a stable, illumination-neutral-looking version of the person that preserves identity and motion. The source-video latent is concatenated with the noisy target latent along the channel dimension to condition the model.

Stage two — relighting. A second model, initialized from Wan2.1-1.3B, takes that delighting output plus one active lighting condition and generates the relit video jointly across space and time, inheriting temporal consistency from the underlying video prior.

How each lighting control is injected. Envmap sequences are encoded as three per-frame buffers — an LDR panorama, a log-intensity map, and a directional encoding — pushed through a VAE into lighting latents and projected into envmap tokens; attention for video tokens at time index t is restricted to envmap tokens from the same index so frame correspondence is kept. Lighting text is encoded by T5 and injected through text cross-attention. A lighting reference video is fused by channel-wise concatenation at the latent input layer.

Where the training data comes from. Static OLAT Data is made by linearly combining OLAT basis images from FaceOLAT (60 subjects, 2–4 expressions each) with 600 synthetic environment-map sequences and 200 real Poly Haven environment maps, augmented by rotation and temporal transformations, giving 24,000 paired videos. Dynamic Gen Data is generated with Wan2.2 and filtered by a VLM, providing over 10,000 paired videos with moving subjects and time-varying light; lighting text comes from Qwen3.5-27B, and the reference condition is derived from the relit video via strong Gaussian blur, spatial shifting, and intensity scaling to suppress identity leakage.

Stage-specific data construction. The two stages swap inputs and targets. For delighting, the relit video is the input and the illumination-neutral video is ground truth; in the generated data the neutral video is produced first and the relit version is created by hard-coded brightness, moving-pattern, and local-intensity transforms. For relighting, the neutral video is the input and the relit video is ground truth, with the neutral version produced by running the trained delighting model. Only one lighting branch is active per sample; the others are masked.

Training. Both models use three-stage schedules (OLAT, then Gen, then OLAT or Hybrid), 8 H20 GPUs, batch size 8, and LoRA (rank/alpha 16 for delighting, 32 for relighting), with the lighting projector and new envmap cross-attention layers trained fully. Inference uses 50 sampling steps with a shift of 5.0.

Why This Matters

Impact on research. The paper reframes dynamic portrait relighting as controllable video generation under time-varying illumination rather than frame-wise image editing, and shows that a two-stage delighting/relighting split removes much of the ambiguity of unknown source lighting. The hybrid data recipe is arguably the more transferable idea: it demonstrates that generated dynamic-lighting video can substitute for expensive dynamic OLAT capture while still being anchored by physically grounded OLAT supervision.

Real-world applications:

• Post-production and VFX, where directors want strobing, sweeping, or color-shifting lighting on a performance without reshooting on a light stage.
• Short-form video and social content creation, where a creator can restyle the lighting of a clip using a text description or by borrowing the lighting from another video.
• Virtual production and video conferencing, where illumination can be normalized or harmonized so a subject matches a target scene or a synthetic environment.
• Avatars and game/AR characters, where a captured performance needs to respond to an in-scene lighting environment map sequence.

Industry relevance. The work comes from Zhejiang University with vivo BlueImage Lab and vivo Mobile Communication, and code is released at github.com/vivoCameraResearch/FlashRelight. The reported inference cost (185.9 s for a 49-frame clip at 832×480 on one H20, 15.23 GiB peak) and the 1.3B-parameter base model indicate a deployment path toward on-device or near-real-time mobile camera features, though the reported times are not real-time.

Future Directions

• Making the delighting stage physically meaningful. The paper deliberately does not require an exact intrinsic decomposition. Whether a more physically grounded appearance representation would improve hard cases — heavy colored light, strong shadows — is left open.
• Quantitative evaluation of cross-video lighting transfer. Reference-video relighting is currently judged only qualitatively because paired ground truth is unavailable; a benchmark or proxy metric for this setting would make progress measurable.
• Scaling resolution, length, and speed. Experiments use 49-frame clips at 832×480 with 50 sampling steps per stage; longer videos, higher resolution, and fewer steps are natural next targets given the two-stage cost.
• Broadening the text control and the training distribution. Text-conditioned results are described as demonstrated "for the evaluated lighting descriptions" on a reserved set of 50 generated plus 50 in-the-wild Pexels videos, leaving robustness to more complex or compositional lighting prompts, and to multi-person or non-portrait subjects, as open questions.

Target Audience

Researchers and graduate students working on relighting, portrait and human video synthesis, and controllable video diffusion will find the core contribution; applied engineers building camera, editing, or virtual-production pipelines will find the data strategy and the inference cost table practically useful. It is not an introductory read — familiarity with diffusion models, latent video generation, and relighting metrics (LE, LPIPS, PSNR, tLPIPS, FID, ID similarity) is assumed throughout.

Authors’ abstract

Portrait video relighting aims to modify the illumination of a portrait video while preserving photorealism and temporal stability. The task becomes particularly challenging under dynamic lighting, where illumination direction, intensity, and color vary over time. To tackle this problem, we introduce FlashRelight, a novel two-stage framework for dynamic portrait video relighting. FlashRelight first suppresses the source illumination through delighting, and then synthesizes the target dynamic lighting on an illumination-neutral portrait video. FlashRelight supports three complementary control modalities: envmap sequences for explicit frame-level lighting control, dynamic lighting text for semantic illumination editing, and lighting reference videos for visual lighting transfer. Training follows a low-cost hybrid strategy that combines static one-light-at-a-time (OLAT) data with generated dynamic-lighting portrait videos, covering both controlled illumination variation and realistic portrait motion. Extensive experiments show that FlashRelight achieves temporally coherent and photorealistic dynamic relighting while enabling flexible cross-video lighting transfer. Code is available at https://github.com/vivoCameraResearch/FlashRelight