NeAR: Coupled Neural Asset-Renderer Stack

AI summary

Overview

• Research area: Computer vision and neural computer graphics — specifically single-image relightable 3D asset generation, neural rendering, and 3D Gaussian Splatting.
• Technical level: Advanced. The paper assumes familiarity with 3D Gaussian Splatting, structured 3D latents (SLAT), rectified flow / diffusion models, HDR environment maps, and PBR material decomposition.
• Scope: The paper proposes NeAR, a coupled neural asset–renderer stack that "homogenizes" illumination into a canonical 3D latent (LH-SLAT) and then decodes it with a lighting-aware neural renderer to produce relightable 3D Gaussian splats from a single casually lit image, evaluated on four relighting/reconstruction tasks.

What This Paper Is About

Neural asset generation and neural rendering have traditionally been developed separately: one system creates a 3D asset assuming a fixed graphics pipeline, and another maps conventional assets to images. Because rendering is nonlinear, small errors in asset decomposition (wrong albedo, wrong normals) compound into baked-in shadows and lighting inconsistencies when the asset is re-rendered. The paper's goal is to co-design the asset representation and the renderer as a single coupled stack so that a casually lit single image can be lifted into an illumination-invariant latent space and then re-rendered under arbitrary HDR lighting and novel viewpoints in real time.

Key Contributions

1. Coupled neural asset–renderer stack (NeAR): A learnable graphics stack in which the neural asset representation and neural renderer are co-designed for single-image relightable 3D asset generation.
2. Lighting-Homogenized Structured 3D Latent (LH-SLAT): A compact, view-agnostic 3D latent that suppresses baked-in shadows and unstable highlights while preserving geometry-consistent diffuse cues, built by fine-tuning a pre-trained SLAT rectified-flow model with LoRA in sparse voxel space.
3. Lighting tokenizer and lighting-aware neural 3D Gaussian renderer: A lighting tokenizer encodes HDR environment maps into lighting condition tokens, and a decoder (Intrinsic Aware Decoder plus Lighting Aware Decoder) maps LH-SLAT, environment illumination, and view embeddings into a relightable 3D Gaussian field rendered by differentiable Gaussian splatting.
4. Extensive evaluation and real-time performance: Experiments across four tasks and multiple datasets (including Aria Digital Twin, Digital Twin Catalog, Objaverse, and a Glossy Synthetic dataset) show state-of-the-art or improved quality with real-time feed-forward inference and no per-object optimization.

Main Findings

• G-buffer forward rendering: NeAR outperforms DiffusionRenderer on all four datasets. Examples: on ADT, PSNR 29.15 vs 24.41, SSIM 0.9484 vs 0.9172, LPIPS 0.0488 vs 0.0802; on Objaverse, PSNR 32.23 vs 27.09, SSIM 0.9627 vs 0.9288.
• Random-lit single-image reconstruction: NeAR reports the best metrics against RGB↔X, DiLightNet, and DiffusionRenderer. On Objaverse, PSNR 26.53 vs 24.81 for DiffusionRenderer; on the Glossy Synthetic dataset, LPIPS 0.0368 vs 0.0547.
• Unknown-lit single-image relighting: NeAR leads against DiLightNet, NeuralGrafferer, and DiffusionRenderer. On the Glossy Synthetic dataset, LPIPS 0.0465 vs 0.0632 for DiffusionRenderer; on Objaverse, PSNR 24.38 vs 23.75.
• Novel-view relighting: Against 3DTopia-XL, Stable-Fast-3D, MeshGen, and Hunyuan3D-2.1, NeAR reports the best figures, e.g. ADT LPIPS 0.0693 vs 0.0727 for Hunyuan3D-2.1, and Objaverse SSIM 0.9392 vs 0.9328.
• Highest PSNR across all four tasks: The paper states NeAR achieves the highest PSNR on the Glossy Synthetic dataset across all four sub-tasks.
• LH-SLAT beats other latent inputs: Ablation shows PSNR 32.02 / SSIM 0.9631 / LPIPS 0.0494 for LH-SLAT alone, versus 28.95 / 0.9281 / 0.0813 for shaded SLAT and 30.38 / 0.9541 / 0.0564 for base-color SLAT; the combination LH + base color is best at 32.54 / 0.9649 / 0.0442.
• Renderer depth trade-off: Ablation on decoder blocks shows 12 + 6 blocks at PSNR 32.54, SSIM 0.9649, LPIPS 0.0442, 59.8M parameters, and 30 FPS; 12 + 9 gives 32.56 / 0.9645 / 0.0439 at 23 FPS, while 0 + 18 (LAD only) collapses to 29.43 / 0.9245 / 0.0624 at 10 FPS with 173.25M parameters. The authors select 6 layers to balance efficiency and performance.
• View information injected before lighting baking helps: Architecture comparison shows "c + d + g (ours)" at 32.54 / 0.9649 / 0.0442, versus 29.82 / 0.9472 / 0.0642 for "a + e + f".
• PBR material estimation: Hunyuan3D-2.1 is reported to misclassify wood as metal (producing erroneous metallic artifacts on eggs) and to blur edges on a wooden cup, while NeAR recovers material properties and preserves 3D consistency.
• Qualitative advantages: DiffusionRenderer and RGB↔X misestimate materials and DiLightNet shows color shifts in reconstruction; NeAR better handles specular highlights, shadow detail, and transparent materials (e.g. the helmet case where intermediate PBR decomposition struggles under standard rendering but the full neural renderer resolves it).
• Real-time inference: The method runs feed-forward at real-time frame rates without per-object optimization (reported FPS values in the ablation are 48, 38, 30, 23, and 10 depending on decoder depth).

Methodology in Plain English

NeAR follows a "homogenize-then-synthesize" strategy with two stages.

Stage 1 — Light homogenization. The authors define a "homogenized" illumination as a uniform, white ambient environment. They use a pre-trained rectified-flow SLAT model to produce an initial shaded SLAT from the input image, then fine-tune it with LoRA inside the sparse voxel latent space so that it instead outputs a Lighting-Homogenized SLAT (LH-SLAT). Training pairs are built offline: for each 3D asset, the ground-truth homogenized latent is created by rendering views under homogenized lighting, extracting DINOv2 features, back-projecting them into a sparse voxel grid, and compressing with a pre-trained SLAT VAE; the corresponding input images come from rendering the same asset under diverse random lighting and camera poses. For highly reflective materials, an optional base-color SLAT is concatenated with the LH-SLAT.

Stage 2 — Relightable neural 3D Gaussian synthesis. A feed-forward decoder takes LH-SLAT plus a target viewpoint and a target illumination (encoded by a lighting tokenizer) and produces a relightable Gaussian field, which is rendered by a differentiable Gaussian splatting rasterizer. The decoder has two modules. The Intrinsic Aware Decoder uses a Transformer with self-shifted window attention and 16 learnable register tokens injected via register cross-attention, producing view-independent, illumination-invariant features. The Lighting Aware Decoder injects view embeddings (ray direction with NeRF positional encoding and per-voxel learnable distance encoding) and lighting condition tokens via cross-attention to produce lighting-dependent features. Spherical harmonics are abandoned so lighting is not baked into the representation; intrinsics (position offset, base color, roughness, metallic, scale, rotation, opacity) come from the intrinsic features, while a 48-dim color feature, lighting-specific scale, and shadow come from the lighting-dependent features. A shallow MLP with ReLU intermediate activations and an ELU final activation predicts radiance, and the rasterizer outputs HDR images plus base color, roughness, metallic, and shadow maps. Training uses an HDR reconstruction loss (L1, LPIPS, D-SSIM, and regularization, with logarithmic transformation of HDR values) plus auxiliary L1 supervision on PBR maps and shadows.

Data and setup. Training uses 87K 3D assets with PBR textures curated from Objaverse-XL, illuminated by 2K HDRIs at 4K resolution. Stage 1 renders 150 viewpoints under normalized lighting; input images use camera poses with yaw sampled within ±45 degrees and pitch from -10 to 45 degrees, with unknown illumination modeled as six area lights uniformly distributed on a sphere, 1–3 area lights randomly sampled in the camera hemisphere, or a random Z-axis-rotated environment map, with area light intensities sampled uniformly between 300 and 700 and distances between 5 and 8 units. Stage 2 uses a fixed 40-degree FOV and samples 12 camera viewpoints on a sphere of radius 2.0, each rendered under 16 illumination conditions. All data generation uses the Blender EEVEE Next engine with raytracing enabled. Training is done on four NVIDIA H100 80GB HBM3 GPUs. A test set of 800 unseen objects is used, plus out-of-domain datasets ADT, DTC, the Glossy Synthetic dataset, and BlenderKit assets modified to use the Principled BSDF shader. Metrics are PSNR, SSIM, and LPIPS.

Why This Matters

• Research impact: The paper reframes neural asset generation and neural rendering as co-designed components rather than independent modules, arguing that this co-design creates a robust "contract" for generation. It offers an alternative to both ill-posed PBR inversion and black-box 2D diffusion relighting, and shows that a homogenized latent plus a lighting-aware decoder can deliver multi-view-consistent relighting feed-forward.
• Real-world applications (as illustrated or implied by the paper):
  • Relighting consumer photos and internet images under new illumination (demonstrated for internet images and real mobile photos).
  • Texture style transfer onto arbitrary geometry with photorealistic neural relighting.
  • Interactive content creation where relightable 3D assets are needed without per-object optimization.
  • Digital twin and e-commerce asset pipelines, given evaluation on the Aria Digital Twin and Digital Twin Catalog datasets with sub-millimeter-accurate models.
• Industry relevance: Real-time feed-forward inference without per-object optimization, explicit PBR outputs (base color, metallic, roughness, shadow maps), and compatibility with HDR environment maps make the approach relevant to graphics pipelines, game and film asset workflows, and virtual production where fast relighting and consistent materials across views matter.

Future Directions

• Extending the framework from controlled captures to in-the-wild and dynamic scenes, which the conclusion explicitly names as a limitation and direction.
• Improving efficiency and generalization of the stack, also named in the conclusion.
• Reducing the computational cost of the decoder: the ablation shows quality saturates as Lighting Aware Decoder depth grows while FPS drops (32.56 PSNR at 23 FPS for 12 + 9 vs 32.54 at 30 FPS for 12 + 6, and 29.43 PSNR at 10 FPS for the 173.25M-parameter LAD-only variant).
• Addressing residual failure modes noted in the paper: the intermediate PBR decomposition struggles with complex effects such as transparency (the helmet example) under standard rendering, and the intermediate PBR output is not identical to the final neural renderer result. (Note: the provided content is truncated in the Supplementary Material section, so additional details beyond this point are not reported here.)

Target Audience

Researchers and practitioners in neural rendering, inverse rendering, 3D generative modeling, and computer graphics who are interested in relightable 3D asset generation from single images. It is most useful for readers already comfortable with 3D Gaussian Splatting, SLAT-style sparse latent representations, rectified flow models, and PBR material estimation; newcomers would need significant background before engaging with the method.

Authors’ abstract

Neural asset authoring and neural rendering have traditionally evolved as disjoint paradigms: one generates digital assets for fixed graphics pipelines, while the other maps conventional assets to images. However, treating them as independent entities limits the potential for end-to-end optimization in fidelity and consistency. In this paper, we bridge this gap with NeAR, a Coupled Neural Asset--Renderer Stack. We argue that co-designing the asset representation and the renderer creates a robust "contract" for superior generation. On the asset side, we introduce the Lighting-Homogenized SLAT (LH-SLAT). Leveraging a rectified-flow model, NeAR lifts casually lit single images into a canonical, illumination-invariant latent space, effectively suppressing baked-in shadows and highlights. On the renderer side, we design a lighting-aware neural decoder tailored to interpret these homogenized latents. Conditioned on HDR environment maps and camera views, it synthesizes relightable 3D Gaussian splats in real-time without per-object optimization. We validate NeAR on four tasks: (1) G-buffer-based forward rendering, (2) random-lit reconstruction, (3) unknown-lit relighting, and (4) novel-view relighting. Extensive experiments demonstrate that our coupled stack outperforms state-of-the-art baselines in both quantitative metrics and perceptual quality. We hope this coupled asset-renderer perspective inspires future graphics stacks that view neural assets and renderers as co-designed components instead of independent entities.