WildCap: Facial Albedo Capture in the Wild via Hybrid Inverse Rendering

AI summary

Overview

Research area: Computer vision — inverse rendering and facial appearance capture (facial reflectance / albedo reconstruction).

Technical level: Advanced. The paper assumes familiarity with physically based rendering, Spherical Harmonics lighting, differentiable rendering, diffusion models, and diffusion posterior sampling.

Scope: The paper presents a single method — WildCap — that reconstructs a high-quality facial diffuse albedo map from an ordinary smartphone video recorded under unconstrained, unknown lighting, and then derives specular and normal maps from it.

What This Paper Is About

Capturing a person's facial albedo (their skin's base color, independent of lighting) has traditionally required either a professional Light Stage or at least a controllable light source such as a smartphone flashlight. Both raise capture cost and restrict who can be scanned. This paper asks whether the same quality can be obtained from a ~30-second smartphone video shot in the wild, with no assumptions about the scene lighting at all — and answers it with a hybrid pipeline that combines a data-driven network with model-based optimization. It appears in the Computer Vision category (arXiv:2512.11237v2, cs.CV, 17 Mar 2026).

Key Contributions

1. A hybrid inverse rendering method for facial albedo capture that combines a data-driven network with model-based optimization, closing the quality gap between in-the-wild and controllable-lighting recordings by a large margin.

2. A texel grid lighting model — a non-physical but highly expressive lighting representation that explains artifacts baked into network-predicted images (such as shadow-baking) as local physical lighting rather than as albedo errors.

3. A scheme for jointly optimizing the texel grid lighting model and sampling a diffusion prior, which resolves the scale ambiguity between local lights and albedo and keeps the solver inside the distribution of valid high-quality albedo maps.

4. Released code, intended to let the method serve as a "handheld Light Stage" for scanning everyday users into the digital world.

Main Findings

• Hybrid framework beats raw-input inverse rendering: The "w/o hybrid" baseline, which runs the method directly on the raw captured frames rather than on network-predicted albedo images, struggles to disentangle reflectance from complex lighting such as specularity and shadows. Feeding the data-driven predictions in first provides a good initialization and yields significantly better results.

• Texel grid lighting is necessary: The "w/o TGL" baseline, using only a global spherical harmonics lighting model, cannot explain shadow-baking artifacts as lighting. Grid sizes of g = 1 and g = 24 were found less expressive; g = 384 over-explains facial detail as lighting and produces over-smooth texture. The chosen g = 96 balances artifact removal against facial detail.

• The diffusion prior is necessary: The "w/o prior" baseline, which optimizes each texel directly with Adam, produces severe artifacts because nothing guarantees convergence to a valid reflectance map. Solving albedo within the diffusion prior's distribution addresses this ill-posedness.

• Best scores on face reconstruction against in-the-wild baselines: Averaged over 6 subjects on the same cropped facial skin region, WildCap reached PSNR 28.79, SSIM 0.9520 and LPIPS 0.0610, versus FLARE* at 27.81 / 0.9411 / 0.0929 and DeFace* at 22.20 / 0.9279 / 0.1192.

• Higher-quality lighting model than the competitors: FLARE* uses a split-sum lighting model that cannot represent non-physical effects in network predictions, and DeFace* is limited to segmenting the face into a small number of regions (the paper cites 5 or 10 as typical) because each region corresponds to a trainable network. That limits their expressiveness relative to the texel grid.

• Comparable quality to controllable-lighting capture at far lower cost: Against DoRA, which reconstructs from a co-located smartphone-and-flashlight video of the same subject, WildCap's in-the-wild reconstruction was of comparable quality while significantly reducing capture cost, and better preserved person-specific traits such as nevus thanks to the LPIPS-loss texture-building step (DoRA's L2 photometric loss tends to average these out).

• Large efficiency gain over DoRA: WildCap completes in 8 minutes on a single 24 GB NVIDIA 4090, while DoRA requires 508 minutes, because WildCap samples at 1K resolution and then applies super-resolution rather than sampling the 4K map directly from the diffusion model.

• Generalizes from very little training data: The diffusion prior is trained on only 48 Light Stage scans, yet the method generalizes to unseen people across indoor and outdoor captures. The paper attributes this to the patch-level design and to the steering power of diffusion posterior sampling.

• Automatic shadow masking is nearly as good as manual: The automatic mask, built with DiFaReli and DiFaReli++, produces diffuse albedo maps of similar quality to the manually drawn mask, though it also marks regions around the eyes and can miss baked ambient occlusion on the side of the nose. The manual method is used by default.

Methodology in Plain English

The pipeline has four stages.

Capture and processing. A smartphone video is recorded around the subject in about 30 seconds with no lighting assumptions. 300 frames are uniformly sampled and resized to 960×720. Camera parameters come from COLMAP, a detailed mesh from 2DGS, and the ICT template is registered to that mesh with Wrap3D to serve as the geometry. From the 300 frames, V = 16 are selected by sharpness for reflectance estimation. The subject's skin tone is assumed to be provided, either manually or automatically, to resolve the ambiguity between albedo and lighting color.

Data-driven first pass. SwitchLight (chosen as the best publicly available method) is run on the raw frames to predict a diffuse albedo image per view. This converts the messy in-the-wild lighting into something far more constrained.

Model-based second pass. Those predicted albedo images are treated as if they were real photographs taken under simpler lighting, and are factored into lighting and a clean albedo map in UV space by minimizing a photometric loss. Because the network is not perfect, some shadow is "baked" into its predictions. The paper's key idea is to explain that baked shadow as a dark local light rather than as albedo: the texel grid lighting model stores Spherical Harmonics parameters on a 2D grid in UV space (2-order SH, 27 coefficients, bilinear interpolation), modulated by a mask marking artifact regions. A global SH term handles the rest of the face. Regularization keeps the local lighting smooth and negative (i.e. dark).

Keeping the solution valid. Because this lighting model is so expressive, the problem becomes under-constrained. The authors train a patch-level diffusion prior at 64×64 resolution on 48 Light Stage scans, then use diffusion posterior sampling to steer the albedo toward the prior distribution while still fitting the observations. Sampling starts from a noised reference scan whose skin tone best matches the subject, using 0.6·T noise steps out of T = 1000. Specular albedo and detailed normal are carried along in the same sampling process and are implicitly pulled into alignment with the albedo. The resulting 1K maps are finally upsampled to 4K by an RCAN super-resolution network.

Why This Matters

Impact on research. The paper reframes a common failure mode of neural reflectance prediction — baked-in, non-physical artifacts — as a modeling opportunity rather than an unavoidable error. Treating a network's shadow-baking as "clean albedo lit by a local dark light" is a general trick that could transfer to other inverse rendering problems where a learned front-end produces non-physical outputs. It also demonstrates that a patch-level diffusion prior trained on a small dataset can regularize a full-resolution inverse rendering problem.

Real-world applications.

• Consumer avatar creation: Scanning everyday users into digital characters with only a phone, rather than booking studio time.
• Games and virtual production: Exporting the reconstructed albedo, specular and normal maps directly into graphics engines for photo-realistic rendering, as the paper's Figure 1 illustrates.
• Telepresence and VR/AR: Building personalized digital humans from ordinary handheld footage.
• Appearance-driven commerce and media: Reproducing skin appearance faithfully enough to preserve person-specific marks such as nevus.

Industry relevance. Removing the controllable-lighting requirement removes the main cost barrier to facial appearance capture. The method runs on a single 24 GB consumer GPU in 8 minutes, which puts it within reach of consumer or prosumer pipelines rather than studio-only workflows. A caveat for industry adoption is the dependency on SwitchLight, which the paper describes as a proprietary model available only through an API.

Future Directions

• Replacing the fragile dependencies. The method relies on SwitchLight for preprocessing and on DiFaReli for automatic shadow detection, both of which are external. DiFaReli in particular is described as slow due to iterative diffusion sampling.
• Training a dedicated delighting network with shadow-confidence estimation. The authors propose using the recently released FaceOLAT dataset to train a replacement for both SwitchLight and DiFaReli — an explicit "important future direction."
• Improving automatic shadow masking. The automatic mask currently over-marks regions around the eyes and misses baked ambient occlusion on the side of the nose, which is why the manual mask remains the default. Better shadow removal confidence is the suggested route.
• Open question — how far the expressiveness can be pushed. The grid-size ablation shows a clear trade-off between removing artifacts and reconstructing facial detail, so whether a learned or adaptive lighting representation could escape that trade-off is left open.

Target Audience

Researchers and practitioners in computer graphics and computer vision working on inverse rendering, relightable avatars, and facial appearance capture, plus graphics engineers who need to produce production-ready reflectance maps from inexpensive capture. The paper is written at an advanced level; readers without grounding in physically based rendering, spherical harmonics, and diffusion posterior sampling will need background reading before the method section. Readers interested primarily in the data-driven front end (SwitchLight) or in the diffusion prior's architecture will find those details in the supplementary material rather than the main text.

Authors’ abstract

Existing methods achieve high-quality facial albedo capture under controllable lighting, which increases capture cost and limits usability. We propose WildCap, a novel method for high-quality facial albedo capture from a smartphone video recorded in the wild. To disentangle high-quality albedo from complex lighting effects in in-the-wild captures, we propose a novel hybrid inverse rendering framework. We first apply a data-driven method, i.e., SwitchLight, to convert the captured images into more constrained conditions and then adopt model-based inverse rendering. However, unavoidable local artifacts in network predictions, such as shadow-baking, are non-physical and thus hinder accurate inverse rendering of lighting and material. To address this, we propose a novel texel grid lighting model to explain non-physical effects as clean albedo illuminated by local physical lighting. During optimization, we jointly sample a diffusion prior for the albedo map and optimize the lighting, effectively resolving scale ambiguity between local lights and albedo. Other reflectance maps are then predicted from the albedo. Our method achieves significantly better results than prior arts in the same capture setup, closing the quality gap between in-the-wild and controllable recordings by a large margin.