3DPR: Single Image 3D Portrait Relight using Generative Priors

AI summary

Overview

Research area: Computer vision and computer graphics, specifically image-based relighting, 3D-aware generative models, and neural rendering of human heads and faces.

Technical level: Advanced. The paper assumes familiarity with GAN inversion, triplane neural representations, volume rendering, One-Light-at-a-Time (OLAT) capture, and HDRI-based relighting.

Scope: The paper introduces 3DPR, a method that turns a single monocular portrait into a relightable, re-viewable 3D representation by predicting OLAT images with generative priors, and releases FaceOLAT, a large multi-view lightstage dataset of 139 subjects.

Note: the provided paper content is truncated partway through the ablation study section, so some results and discussion (including any limitations section) are not available here.

What This Paper Is About

Rendering a human head under new lighting from only one photograph is an underconstrained problem: there is not enough information in a single image to recover the geometry, skin material, and illumination that produced it. Prior approaches either decompose the image with explicit physical models (which rely on approximations) or train 2D/volumetric generative models (which often bake in the input lighting and lose identity or view consistency). 3DPR instead learns from lightstage One-Light-at-a-Time images what a face looks like under each individual light direction, then combines those predicted OLAT images linearly according to a target HDRI environment map to produce physically grounded relighting from novel viewpoints.

Key Contributions

1. An image-based 3D portrait relighting method that combines a pretrained 3D generative prior (EG3D, built on StyleGAN2) with a reflectance model trained on OLAT data, enabling physically accurate editing of both illumination and viewpoint from a monocular input image.
2. FaceOLAT, described as the first large-scale, publicly available multi-view HDR OLAT face dataset: 139 subjects, 331 dense OLAT lighting conditions, 40 viewpoints, 4K resolution, covering full-head reflectance including hair.
3. State-of-the-art quantitative and qualitative results on the WeyrichOLAT benchmark and on FaceOLAT, particularly in identity preservation and in capturing specularities, self-shadows, and subsurface scattering.
4. Public release of code and pretrained checkpoints at https://vcai.mpi-inf.mpg.de/projects/3dpr/.

Main Findings

• Dataset scale comparison: FaceOLAT provides 331 illuminations, 139 subjects, 40 views at 4K with image-based relighting support. The paper compares this against ICT-3DRFE (3 illuminations, 23 subjects, 2 views, 1K), Ultrastage (3, 100, 32, 8K), RGCA (460, 4, 110, 4K, using grouped OLATs intended for dynamic capture), and Dynamic OLAT (114, 4, 1, 1K). The authors state FaceOLAT surpasses all publicly available datasets as well as the non-public OLAT dataset of Weyrich et al. (2006).

• WeyrichOLAT benchmark (Table 2): 3DPR_w achieves SSIM 0.87, LPIPS 0.1828, RMSE 0.1332, DISTS 0.1689, PSNR 28.69, ID 0.942. For comparison, Lite2Relight reaches 0.83 / 0.2492 / 0.1841 / 0.1719 / 28.27 / 0.936; NeRFFaceLighting 0.79 / 0.2171 / 0.2393 / 0.2107 / 27.24 / 0.892; VoRF 0.69 / 0.3253 / 0.1967 / 0.1934 / 20.21 / 0.860; PhotoApp 0.72 / 0.4163 / 0.1988 / 0.2031 / 29.13 / 0.853; and NeLF 0.75 / 0.4874 / 0.2466 / 0.2212 / 19.72 / 0.798.

• FaceOLAT benchmark (Table 3): 3DPR_o reaches SSIM 0.83, LPIPS 0.1996, RMSE 0.1801, DISTS 0.1751, PSNR 21.02, ID 0.943, versus NeRFFaceLighting at 0.77 / 0.2385 / 0.2926 / 0.2193 / 16.97 / 0.906 and Lite2Relight retrained on the dataset at 0.79 / 0.2506 / 0.2619 / 0.20861 / 16.72 / 0.910.

• OLAT synthesis quality: The paper reports SSIM 0.88, LPIPS 0.1753, and PSNR 28.70 for its OLAT renderings, against VoRF at SSIM 0.71, LPIPS 0.3148, PSNR 20.43.

• Robustness to sparse and colored lighting: Both NeRFFaceLighting and Lite2Relight degrade as lighting becomes sparser. The authors hypothesize the conditions fall outside EG3D's training distribution of natural in-the-wild illumination. NeRFFaceLighting is limited by its spherical-harmonics representation and inaccurate lighting-albedo disentanglement, and breaks down under colored light. 3DPR remains robust because it explicitly models reflectance via OLAT prediction rather than interpreting the environment map with a network.

• Runtime: On an NVIDIA 3090 GPU, 3DPR synthesizes the full set of 331 OLAT images in approximately 30.49 s; reducing to 150 OLATs causes minimal degradation and lowers runtime to about 13.8 s. On an H100 GPU, all 331 OLATs take 7.74 s. Once OLATs exist for a subject and viewpoint, relighting under a novel environment map takes 0.24 s. The continuous reflectance field also supports upsampling lighting resolution, e.g. 1324 OLAT images in 34.64 s on an H100. The authors state the method is not as fast as Lite2Relight.

• Ablation findings (partially reported): The paper states that introducing the feature fusion module E_SR improves quality by regularizing the super-resolution network with identity information from the inversion stage. The ID-MRF loss is reported to recover high-frequency details more effectively than perceptual losses such as LPIPS. The text ends mid-sentence in this section, so the full ablation is not available in the provided content.

Methodology in Plain English

The pipeline works in two conceptual stages.

Stage one: put the photo into a 3D-aware latent space. A single input portrait is passed through a pretrained encoder-based GAN inversion network, which maps it into the latent space of EG3D. EG3D converts a noise vector into an intermediate code, then a StyleGAN2 generator produces triplane features that encode geometry and appearance. These triplane features can be volume-rendered from any camera viewpoint. Rendering them yields both a low-resolution RGB image and a higher-dimensional "high-frequency" feature image that carries identity detail.

Stage two: predict what the face looks like under each individual light. The triplane features are concatenated with a specific light direction and passed through a reflectance encoder (a ResNet-based network with 96 channels, a depth the authors say is critical for modeling specularities, hard shadows, and subsurface scattering). The result is reflectance-aware triplane features. A lightweight single-layer MLP decoder takes these plus a view direction and volume-renders a low-resolution OLAT image and a high-frequency reflectance feature map.

Preventing overfitting. The lightstage dataset has limited subject diversity, so the super-resolution network could overfit. To counter this, a feature fusion module combines the high-frequency reflectance features with the high-frequency identity features from the inversion stage. Because the super-resolution network was pretrained to consume those identity features, it learns to forward them through, and is forced to rely on the reflectance features only for lighting. The final super-resolution network produces a 512×512 OLAT image.

Training and losses. Only the reflectance encoder, reflectance decoder, the fusion module, and the OLAT super-resolution network are trained; the StyleGAN and EG3D decoders are frozen. Supervision uses an L1 reconstruction loss against ground-truth lightstage OLATs plus an ID-MRF loss, weighted as L = L_O + 0.3·L_MRF. The ID-MRF loss minimizes patch-wise nearest-neighbor distances in VGG19 feature space (conv3_2 and conv4_2 layers). The authors note adversarial losses were avoided because the limited number of subjects risks discriminator overfitting and unstable training.

Relighting at inference. Because light transport is additive, the predicted OLAT images are linearly combined with the weights of a target HDRI environment map. The paper's equation is C ≈ Σ f_l · O(l), where l ranges over OLAT lighting directions, O(l) is the OLAT image for direction l, and f_l are the environment map weights. This means a single forward pass replaces the test-time optimization used by VoRF and NeRFFaceLighting.

Data handling. FaceOLAT capture lasts 7 s per subject, so small involuntary movements occur; the authors interleave fully lit reference frames every 21 OLAT captures and apply optical-flow-based alignment. Preprocessing also includes calibration, detailed 3D reconstruction, and background segmentation using BGMv2 and RMBGv2. The dataset is split into 129 training subjects and 10 evaluation subjects. The paper's loss discussion separately refers to 130 subjects in the dataset.

Why This Matters

Impact on research. The paper argues that the field has lacked a large, publicly available multi-view lightstage OLAT dataset with dense illumination and wide subject diversity, which has limited how well reflectance models can generalize. FaceOLAT addresses that gap directly, and the method shows a way to combine a generative 3D prior learned from in-the-wild images with physical light transport learned from lightstage captures. The authors frame their results on WeyrichOLAT as arising from this combination rather than from data alone, since all methods there are trained and evaluated on the same split.

Real-world applications:

• Augmented and virtual reality: The paper explicitly motivates seamless blending of real and synthetic assets, noting that faces are of particular importance and require converting few-shot or single monocular images into 3D assets renderable under novel environments and viewpoints.
• Telepresence and video communication: Relightable avatars from a single portrait could be placed into a shared virtual lighting environment, though the paper does not benchmark video-rate performance.
• Film, VFX, and post-production: The paper highlights that explicit OLAT-based relighting provides control over artistic, sparse, or non-natural illumination, which it identifies as common in cinematic and indoor environments.
• Interactive avatar and game content creation: The single-forward-pass inference (0.24 s per novel environment map once OLATs exist) is presented as practical for applications where per-subject test-time optimization is considered impractical.

Industry relevance. Two of the authors are affiliated with Google, and the method builds on EG3D and StyleGAN2, so the work sits close to deployed generative avatar pipelines. The release of code and pretrained checkpoints, plus a public dataset, lowers the barrier for other groups to reproduce and extend the approach. The paper also contrasts its approach with competitors by name (PhotoApp, VoRF, NeRFFaceLighting, Lite2Relight, NeLF), making the comparison directly actionable for teams choosing a relighting backbone.

Future Directions

• Reducing the OLAT budget. The authors already show that 150 OLATs give minimal degradation versus 331 (13.8 s versus 30.49 s on an NVIDIA 3090), leaving open how far this can be pushed without quality loss.
• Higher lighting resolution. Because the method models a continuous reflectance field, it supports flexible upsampling of lighting resolution; the paper demonstrates 1324 OLAT images in 34.64 s on an H100 but does not characterize the quality limits of this scaling.
• Dynamic and video capture. The dataset comparison table notes that RGCA's grouped OLATs are intended for dynamic capture while FaceOLAT is a static multi-view capture under expressions. Extending to temporally consistent relighting of moving subjects is not reported here.
• Closing the identity/geometry gap under extreme lighting. The paper attributes Lite2Relight's failures to out-of-distribution target illuminations relative to EG3D's natural-light training data, so improving out-of-distribution robustness in the generative prior itself remains an open question.

Target Audience

This paper is most useful for computer graphics and computer vision researchers working on relighting, neural rendering, 3D-aware generative models, and inverse rendering of human heads. It also suits graphics engineers and technical artists building avatar or virtual production pipelines who need relightable 3D head assets from single images, and dataset curators interested in lightstage capture protocols, alignment strategies, and public benchmark design. Readers should already be comfortable with triplane representations, volume rendering, GAN inversion, and image-based relighting theory.

Authors’ abstract

Rendering novel, relit views of a human head, given a monocular portrait image as input, is an inherently underconstrained problem. The traditional graphics solution is to explicitly decompose the input image into geometry, material and lighting via differentiable rendering; but this is constrained by the multiple assumptions and approximations of the underlying models and parameterizations of these scene components. We propose 3DPR, an image-based relighting model that leverages generative priors learnt from multi-view One-Light-at-A-Time (OLAT) images captured in a light stage. We introduce a new diverse and large-scale multi-view 4K OLAT dataset of 139 subjects to learn a high-quality prior over the distribution of high-frequency face reflectance. We leverage the latent space of a pre-trained generative head model that provides a rich prior over face geometry learnt from in-the-wild image datasets. The input portrait is first embedded in the latent manifold of such a model through an encoder-based inversion process. Then a novel triplane-based reflectance network trained on our lightstage data is used to synthesize high-fidelity OLAT images to enable image-based relighting. Our reflectance network operates in the latent space of the generative head model, crucially enabling a relatively small number of lightstage images to train the reflectance model. Combining the generated OLATs according to a given HDRI environment maps yields physically accurate environmental relighting results. Through quantitative and qualitative evaluations, we demonstrate that 3DPR outperforms previous methods, particularly in preserving identity and in capturing lighting effects such as specularities, self-shadows, and subsurface scattering. Project Page: https://vcai.mpi-inf.mpg.de/projects/3dpr/