Luce: Relightable Gaussians for 3D Asset Generation

AI summary

Overview

Research area: Computer vision / 3D generative modeling — specifically single-image-to-3D asset generation with physically based rendering (PBR) materials and relighting support.

Technical level: Advanced. The paper assumes familiarity with 3D Gaussian Splatting, latent diffusion/flow matching, variational autoencoders, and PBR shading terminology (albedo, metallic-roughness, normals, Cook–Torrance BRDF, split-sum image-based lighting).

Scope: The paper introduces a 3D representation — voxelized clouds of per-modality PBR Gaussians — plus a generative pipeline (SLatVAE + SLatFlow) that turns a single input image into a relightable, materially complete 3D asset, evaluated on Toys4K and a new 130-image AI-generated benchmark.

What This Paper Is About

Existing image-to-3D generators either bake lighting into the representation (so assets cannot be relit) or model geometry without materials. Luce's goal is a 3D representation that captures fine geometry and a full PBR material description at once, in a form that compresses into a latent a generative model can diffuse and that drops directly into standard rendering pipelines. The paper also targets high-frequency surface detail — legible text, logos, and inscriptions — that prior methods tend to smooth away.

Key Contributions

1. A multimodal PBR Gaussian representation. Each occupied voxel holds three geometrically independent Gaussian sets — albedo (G^alb), metallic-roughness (G^mr), and normals (G^nor) — each with its own positions, scales, rotations, and opacities. Because normals are stored explicitly rather than derived from surface shape, they can carry detail finer than the geometry. The paper calls the modality set M = {alb, mr, nor}.
2. A unified latent for joint geometry and material generation. The SLatVAE compresses the raw PBR Gaussian cloud into a compact, diffusible per-voxel latent, deliberately trading spatial resolution for per-voxel density (encoder downsamples 128³ → 64³; decoder emits K′=32 Gaussians per voxel per modality versus K=8 at input). A single latent decodes both to relightable PBR Gaussians and, optionally, to a textured mesh.
3. Multi-layer image conditioning. DINOv2 features from layers 6, 12, 18, and 24 are concatenated and projected (4×1024 → 1024) and injected via cross-attention, preserving both semantic context and fine spatial detail.
4. Tangent-space normal map transfer. Decoded normal Gaussians are baked onto the extracted mesh as tangent-space normal maps, adding sub-voxel surface detail (engravings, fabric weave, embossed text) at zero polygon-count cost. The final mesh carries four texture maps: diffuse (albedo), metallic, roughness, and tangent-space normal.

Main Findings

• State-of-the-art generation FID on Toys4K. Luce GS reaches FID 20.99 on N = 412 Toys4K assets, versus TRELLIS 2 at 29.22, LiTo at 29.76, TRELLIS GS at 30.75, TRELLIS mesh at 32.39, and 3DTopia-XL at 83.23. The abstract states this is a 28% improvement over the strongest baseline; the introduction describes it as "more than 8 FID."
• Best CLIP alignment on the AI-generated-image benchmark. On N = 130 AI-generated images (from Gemini 3.1 Flash Image), Luce GS scores CLIP 0.8519 versus the best baseline TRELLIS GS at 0.8299. Luce GS also leads SigLIP2 at 0.8508 versus TRELLIS GS 0.8339.
• Mesh variants lead the mesh-only metrics. Luce mesh with tangent normal leads ULIP at 0.1679 and Uni3D-L at 0.3771 on Toys4K; ULIP and Uni3D-L are mesh-only metrics (marked "—" for Gaussian-only methods).
• Normal map transfer improves normals and color. Baking decoded normals raises mesh normal PSNR from 29.5 to 33.0 dB, raises color PSNR from 31.1 to 32.5, and moves Toys4K FID from 25.04 (mesh without tangent normal) to 21.10 (with).
• Best reconstruction on color and normals. On the 338-asset PBR subset of Toys4K, Luce GS leads color rendering at 36.1 dB PSNR (SSIM 0.967, LPIPS 0.059) and normals at 34.6 dB (SSIM 0.961, LPIPS 0.038). TRELLIS 2 leads albedo (40.7 PSNR, 0.051 LPIPS vs. Luce's 38.6 / 0.055) and metallic-roughness (42.7 vs. Luce's 39.1).
• Multi-layer conditioning beats single-layer. Multi-layer DINOv2 improves Toys4K FID (20.99 vs. 25.21) and CLIP on the AI-generated benchmark (0.8519 vs. 0.8081).
• Faster than several larger baselines. Mean inference time on a single H100: Luce GS 42.20 s at 4.48B parameters, versus TRELLIS 2 at 176.72 s and 7.48B parameters. Luce mesh variants take 148.13 s (without tangent normal) and 159.16 s (with) at 4.58B parameters; LiTo is faster on Toys4K reconstruction at 1.21 s.
• Training scale. Roughly 500K PBR-filtered assets from Objaverse and Objaverse-XL plus a 158K-asset PBR subset of TexVerse. Preprocessing renders each asset from 150 cameras at 1024×1024, fitted onto a 128³ sparse grid with K=8 Gaussians per voxel per modality. Both SLatVAE and SLatFlow train on 64 H100 GPUs for 500K steps, about 14 days each.

Methodology in Plain English

• Representation first. Instead of storing one blob of appearance per point, the authors give every occupied voxel three separate Gaussian sets — one for color, one for how metallic and rough the surface is, one for which way it faces. Each set places its own points independently, so a shiny wood surface can spend its density on color detail while brushed metal spends it on scratches and normals.
• Rendering without a mesh. Each modality is splatted independently with standard 3DGS alpha compositing, producing per-pixel albedo, metallic, roughness, normal, and opacity buffers. Those buffers are then lit analytically using split-sum image-based lighting with a Cook–Torrance microfacet BRDF. Normals are renormalized after compositing. Notably, the representation stores no spherical harmonics — view dependence comes from the physics, not baked coefficients.
• Squeeze it into a latent. A variational autoencoder (SLatVAE) compresses the Gaussian cloud into a compact per-voxel latent. The encoder runs at 128³ and downsamples to 64³ only in its final block, delaying compression so it can reason about fine per-voxel features first. The decoder compensates for the coarser grid by predicting more Gaussians per voxel.
• Generate with flow matching. Generation is two-stage (structure-then-latent, adopted from TRELLIS): the pretrained sparse-structure VAE and flow from TRELLIS 2 predict which voxels are occupied, then SLatFlow — a 30-block DiT with RoPE on 3D voxel coordinates, variable-length flash attention, and adaLN timestep modulation — generates the PBR latent at each active voxel, conditioned on the image through cross-attention.
• Condition on many encoder layers. Fine detail like logos and text lives in shallow DINOv2 layers; semantics live in deep ones. The authors concatenate layers 6, 12, 18, and 24 to get both.
• Optional mesh export. The same latent feeds a FlexiCubes mesh decoder (architecture unchanged from TRELLIS, retrained on Luce's latent), and decoded normal Gaussians are expressed in the mesh's local tangent frame to produce a tangent-space normal map.
• Supervision detail. Normal Gaussians are supervised against the effective normals — the perturbed geometric normals after the asset's authored normal maps are applied — which is what lets them learn detail finer than the base geometry.

Why This Matters

Most image-to-3D systems produce assets whose lighting is frozen in. Luce's distinctive move is to make materials explicit and independent of lighting, so a generated asset can be dropped under a new environment map and shaded correctly — while still being produced by a single-image generative model rather than per-scene optimization.

Real-world applications:

• Game and VFX asset pipelines, where generated assets must accept new lighting, integrate with PBR shaders, and ship as textured meshes with normal maps.
• E-commerce and product visualization, where a single photograph could yield a relightable, materially plausible 3D product for configurable lighting.
• AR/VR and virtual try-on, where an object or surface must match the lighting of the live environment.
• Digital twins and synthetic training data, where assets need material metadata to behave correctly in downstream simulators.

Industry relevance: the mesh export path (albedo, metallic, roughness, tangent-space normal — four maps) is aimed squarely at production renderers, and the paper's cost table reports 4.48B parameters at 42.20 s per asset on an H100, which is cheaper in parameter count than TRELLIS 2 (7.48B) though not the fastest option overall.

Future Directions

1. Resolution limits. Assets whose detail is fine relative to their overall extent may be under-resolved when features span only a few voxels. The authors propose cascaded or adaptive-resolution decoding, with a coarse latent for global shape and higher-resolution stages refining local geometry, textures, and normals.
2. Richer materials. The current model covers albedo, metallic-roughness, and normals under a Cook–Torrance reflectance model. It does not explicitly model subsurface scattering, anisotropy, translucency, thin-film interference, or strongly view-dependent reflectance — additional material channels or learned view-dependent residuals are suggested.
3. Better mesh extraction. The FlexiCubes decoder comes from TRELLIS; higher-resolution or learned UV-aware extraction is left for future work.
4. Beyond objects. The pipeline targets object-centric assets; scene-level generation with coherent lighting, material consistency, and object interactions remains open.

Target Audience

Researchers and practitioners in 3D generative modeling, inverse rendering, and neural scene representations who already understand Gaussian Splatting and diffusion/flow models. It is also relevant to graphics engineers evaluating whether a generative model's output can be imported into a production PBR pipeline, and to readers interested in how multi-layer visual encoder features transfer from 2D dense prediction tasks to 3D generation.

Authors’ abstract

High-fidelity image-to-3D generation requires a 3D representation that captures both geometry and appearance. To support relighting and integration into standard rendering pipelines, the representation should include physically based rendering (PBR) modalities such as albedo, metallic-roughness, and surface normals. We propose Luce, a 3D representation that unifies geometry and PBR materials within a voxelized multimodal Gaussian cloud, using dedicated Gaussian primitives for each modality. A variational autoencoder compresses this representation into a unified material-aware latent space. A rectified-flow transformer generates this latent from a single image, conditioned on multi-layer features from a pretrained image encoder that preserve both semantic context and fine spatial detail. The latent then decodes into relightable PBR Gaussians and an optional textured mesh with a tangent-space normal map. On Toys4K, Luce achieves state-of-the-art single-image-to-3D generation, improving FID by 28% over the strongest baseline. We further introduce a benchmark of AI-generated images, on which Luce improves the CLIP image-alignment score over the best baseline (0.8519 vs. 0.8299). Luce generates relightable, geometrically accurate, and materially faithful assets that preserve fine details such as text, logos, and inscriptions.