VENI: Variational Encoder for Natural Illumination

AI summary

Overview

Research area: Computer vision, specifically inverse rendering and illumination modeling, with connections to equivariant deep learning and generative models on the sphere.

Technical level: Intermediate. Readers should be comfortable with variational autoencoders, transformers, and the general idea of rotation-equivariance, but the paper is accessible to anyone familiar with modern deep learning.

One-sentence scope: The paper introduces VENI, a rotation-equivariant variational autoencoder that learns a prior over 360-degree natural illumination environments, producing both higher-quality reconstructions and a better-structured latent space than the previous state-of-the-art (RENI++).

What This Paper Is About

Inverse rendering—recovering shape, material, and lighting from a single image—is fundamentally ambiguous because many combinations of these factors produce the same picture. A strong prior over realistic illumination helps resolve this ambiguity. The paper addresses the weakness of existing neural illumination priors: they either ignore the fact that lighting lives on a sphere and rotates naturally around a vertical axis, or they use an "autodecoder" design in which each image gets its own randomly initialized latent code, producing a latent space where the same lighting can be represented in many different ways. VENI replaces that with a proper encoder–decoder architecture that maps an environment map to a unique latent code and back, while respecting the rotational symmetry of outdoor lighting.

Key Contributions

1. A rotation-equivariant panoramic vision transformer encoder (VN-ViT). The encoder operates directly on the 3D sphere rather than a distorted 2D projection, using patches sampled as vertical stripes and Vector Neuron counterparts of standard transformer components.

2. A novel SO(2)-equivariant fully connected layer. This extends the Vector Neurons framework, which normally provides SO(3)-equivariance, so that only rotations about the vertical up-axis are treated as equivariant while the zenith direction and color channels remain invariant—exactly matching how real outdoor illumination behaves.

3. A rotation-equivariant natural illumination prior with a well-behaved, unique latent space. Because the model is a variational autoencoder, similar images map to similar latent codes, eliminating the per-image optimization required by RENI++.

4. New evaluation methodology for latent space quality. The authors introduce two intuitive metrics—uniqueness and reconstruction consistency—to quantify how well the latent space is structured, and demonstrate improvements in downstream inverse rendering optimization.

Main Findings

• Reconstruction quality improves, most dramatically at small latent sizes. At latent dimension D=27, VENI reaches 20.33 PSNR in LDR space with latent optimization versus 18.02 for RENI++; at D=300 the gap narrows but VENI still leads (22.68 vs 22.10). Even the pure autoencoder pass, which requires no per-image optimization, beats RENI++ at D=27.

• Latent space uniqueness is substantially better. Uniqueness (lower is better) drops from 1.46 to 0.04 at D=27, from 1.11 to 0.43 at D=147, and from 1.03 to 0.57 at D=300. This means interpolating between two latent codes that both fit the same image no longer produces spurious intermediate illuminations.

• Reconstruction consistency is much higher. The Spearman correlation between latent-space distances and image-space distances rises from 0.17 to 0.50 at D=27, and remains higher at all tested dimensions, indicating that latent proximity actually corresponds to perceptual similarity.

• The model scales to large datasets where the baseline degrades. Going from 1,500 to 43,260 training images, RENI++ PSNR falls from 20.11 to 17.14, while VENI's autoencoder pass actually improves (16.00 to 16.77) and its optimized variant only drops modestly (20.99 to 19.90).

• Interpolations are artifact-free. Visual comparisons show VENI smoothly transitioning between snow-forest, lake-field, and mountain-road illuminations, while RENI++ inserts phantom suns and noise into intermediate frames.

• The entire encoder does not need to be SO(2)-equivariant. Ablations show that applying the new SO(2)-equivariant layer only in the input and output projections gives the best results (18.78/19.40/19.47 PSNR across latent sizes), outperforming both full SO(2) encoders and full Vector Neuron encoders.

• Losses and pretraining both matter. Removing the MAGE and scale-invariant losses drops PSNR by roughly 1.0–1.5 dB, and removing StreetLearn pretraining causes a similar drop. Removing the SO(2) linear projection (falling back to standard layers) costs about 2 dB.

• Downstream inverse rendering benefits. Qualitative optimization results show VENI's latent space is easier to optimize over for inverse rendering than RENI++'s.

Methodology in Plain English

The core idea is to build an encoder–decoder that respects the rotational symmetry of outdoor lighting. In an outdoor scene, rotating the environment around the vertical axis produces another perfectly valid lighting environment; rotating about any other axis does not. So the model is designed to be equivariant only to rotations about the up-axis.

Encoding. A 360-degree spherical environment map is split into vertical stripes (64 of them). Rather than using positional encodings—which would destroy equivariance—each pixel is represented by concatenating its color with the 3D direction vector it was sampled from. This tells the transformer where every pixel lives on the sphere without any 2D projection, which also eliminates the distortion problems that plague equirectangular images. The stripes are then processed by a Vector Neuron Vision Transformer, a version of the standard ViT in which linear layers, attention, and non-linearities are all replaced by their rotation-equivariant counterparts.

The SO(2) extension. Standard Vector Neurons enforce SO(3)-equivariance, treating all three spatial axes equally. The authors instead design a fully connected layer that splits its input into an equivariant part (the horizontal x and y components) and an invariant part (the vertical z component plus the RGB color values). The equivariant output is a bilinear combination of the equivariant input and the invariant input; the invariant output depends on the invariant input plus the length of the equivariant vectors—which is rotation-invariant. This lets the two halves exchange information without breaking equivariance.

Latent space. The transformer's output token is projected to a mean vector and a variance, and a latent code is drawn via the reparameterization trick. Because the latent space is 3D, the sampling uses an isotropic spherical normal distribution rather than three independent 1D Gaussians, since the latter would be axis-aligned and therefore not rotation-equivariant.

Decoding. The latent code is decoded by the RENI++ conditional neural field, which takes a query direction and the latent vector and outputs a color at that direction. Equivariance is achieved by computing the direction relative to the latent code (a dot product) and applying a Vector Neuron invariant layer to the latent code itself.

Training. The model is trained in log space (matching human brightness perception and handling the unknown exposure of HDR captures) with three scale-invariant losses borrowed from depth estimation—a multi-scale gradient loss (MAGE), a scale-invariant error, and a cosine similarity loss on RGB directions—plus a KLD term adapted to the 3D isotropic prior. Equirectangular pixels are weighted by sin of the polar angle to correct for irregular sampling near the poles. The training curriculum first pretrains on 43,310 HDR-converted Google Street View images from Manhattan (for scale), then fine-tunes on the smaller, higher-quality RENI++ dataset of 1,694 outdoor HDR panoramas.

Why This Matters

Impact on research. This paper shows that a proper encoder-based variational formulation can outperform the autodecoder design that has dominated recent neural illumination priors, without sacrificing rotation-equivariance. It also provides a general-purpose tool—the SO(2)-equivariant fully connected layer—that could be reused in any spherical or panoramic vision model where only one rotation axis is meaningful, including climate data, robotics, and omnidirectional perception.

Real-world applications:

• Augmented and virtual reality: placing virtual objects into real scenes with lighting that matches the environment, including smoothly animated sun movement as time of day changes.
• Visual effects and film compositing: inserting CG elements into live-action plates with plausible, editable lighting.
• Autonomous driving and robotics simulation: generating diverse, physically plausible lighting conditions for training perception systems.
• E-commerce and product photography: relighting product shots consistently and cheaply without a physical studio setup.

Industry relevance. Game engines, content creation pipelines, and any AR platform need fast, controllable illumination representations. A latent space that supports smooth, artifact-free interpolation and efficient optimization is directly useful for interactive editing tools, where an artist wants to move a virtual sun or shift color temperature continuously rather than snap between discrete presets.

Future Directions

• Resolution and detail. The model is trained on 128×64 environment maps. Extending to higher resolutions—especially capturing sharp sun disks and fine cloud structure—remains an open problem.

• Beyond outdoor illumination. The SO(2) restriction assumes a canonical up-axis, which holds outdoors. Indoor scenes with light from multiple directions may require a different or relaxed equivariance structure.

• Tighter integration with full inverse rendering. The paper shows preliminary evidence that the latent space is easier to optimize over, but a full joint optimization of shape, materials, and lighting using VENI as the prior is a natural next step.

• Better HDR training data. The reliance on converted LDR Street View data introduces noise and biased scene statistics. Larger native HDR panoramic datasets or improved HDR reconstruction methods would likely improve results further, and the observed gap between the autoencoder pass and per-image optimization suggests the encoder still leaves some reconstruction quality on the table.

Target Audience

Researchers and graduate students working on inverse rendering, illumination estimation, intrinsic decomposition, or equivariant neural networks. It will also be valuable to practitioners in AR/VR, VFX, and computational photography who need structured, editable lighting representations, and to anyone interested in applying Vector Neurons or transformer architectures to spherical or panoramic data.

Authors’ abstract

Inverse rendering is an ill-posed problem, but priors such as illumination priors can help simplify it. Existing work either disregards the spherical and rotation-equivariant nature of illumination environments or does not provide a well-behaved latent space. We propose a rotation-equivariant variational autoencoder that models natural illumination on the sphere without relying on 2D projections. To preserve the SO(2)-equivariance of environment maps, we use a novel Vector Neuron Vision Transformer (VN-ViT) as encoder and a rotation-equivariant conditional neural field as decoder. In the encoder, we reduce the equivariance from SO(3) to SO(2) using a novel SO(2)-equivariant fully connected layer, an extension of Vector Neurons. We show that our SO(2)-equivariant fully connected layer outperforms standard Vector Neurons when used in our SO(2)-equivariant model. Compared to previous methods, our variational autoencoder enables smoother interpolation in latent space and offers a more well-behaved latent space.