Splatent: Splatting Diffusion Latents for Novel View Synthesis

AI summary

Overview

Research area: Computer vision — 3D scene reconstruction and novel view synthesis, specifically radiance fields operating in the compressed latent space of diffusion-model VAEs.

Technical level: Intermediate. Readers should know roughly what 3D Gaussian Splatting, VAEs, and diffusion models are, but the core insight is conceptually simple.

Scope: The paper proposes Splatent, a framework that combines latent-space 3D Gaussian Splatting with a single-step diffusion refiner that restores high-frequency detail by attending to reference input views, achieving state-of-the-art latent radiance field reconstruction and improving feed-forward 3D models.

What This Paper Is About

Modern diffusion models do not work on raw pixels; they work on a compressed latent representation produced by a variational autoencoder (VAE). Recent 3D reconstruction methods try to build radiance fields directly in that same latent space because it is faster and integrates cleanly with generative pipelines. The problem is that VAE latents are not multi-view consistent — the high-frequency parts of the latent differ between viewpoints — so when you optimize a 3D Gaussian Splatting model on them, those conflicting frequencies cancel out and the rendered result is blurry. Splatent's goal is to recover those lost details faithfully, using the original input views rather than letting a generative model invent them.

Key Contributions

1. A spectral analysis of VAE latent spaces for 3D reconstruction. The authors show across more than 45K latent images from 140 scenes that the latent spaces of popular VAEs (including Stable Diffusion's) lack multi-view consistency, and that this specifically destroys view-dependent high-frequency components during 3DGS optimization.

2. A principled latent reconstruction framework. Rather than fixing the VAE (which damages reconstruction quality) or hallucinating details with video diffusion (which is unfaithful), Splatent keeps the 3D representation in the low-frequency domain and recovers high-frequency details in 2D through multi-view self-attention over reference views.

3. State-of-the-art latent radiance field results. Splatent outperforms Feature-3DGS and Latent Radiance Fields (LRF) on PSNR, SSIM, LPIPS, and FID across dense, sparse, and cross-dataset generalization settings, with the VAE kept frozen throughout.

4. A drop-in enhancer for feed-forward 3D models. Integrated into MVSplat360, Splatent improves every metric while reducing hallucinations, demonstrating that the method composes with existing latent-based pipelines.

Main Findings

• The latent space is the bottleneck, not the 3D model. Spectral analysis shows that VAE latents encode both low- and high-frequency content, but the high-frequency portion is view-dependent and inconsistent. During 3DGS optimization these contributions average out, leaving only coarse low-frequency structure — which is exactly why decoded novel views look blurry.

• Freezing the VAE matters. Prior work (LRF) fine-tuned the VAE to make it more 3D-consistent, but this degrades reconstruction quality and breaks compatibility with pre-trained diffusion models that expect the original latent distribution. Splatent keeps the VAE frozen and preserves its generalization.

• Multi-view attention in 2D is the key mechanism. Tiling the degraded rendered latent together with reference view latents into a spatial grid and running a single-step diffusion model with self-attention lets high-frequency details propagate from references to the rendered view. This avoids any 3D-centric detail reconstruction.

• Quantitative gains are large across the board. On DL3DV-10K dense, PSNR improves from 20.19 (LRF) to 21.94, and FID drops from 75.32 to 35.60. In the sparse 5-view setting, PSNR rises from 15.34 to 17.44 and FID from 204.36 to 86.12. Improvements hold on LLFF and Mip-NeRF360, which were not in the training set.

• 3D consistency improves too. Using the MEt3R metric, Splatent scores 0.0774 dense and 0.0998 sparse versus 0.1082/0.1272 for LRF and 0.1106/0.1281 for Feature-3DGS — a 28–30% dense and 22% sparse improvement. The authors attribute this to the rendered latent providing consistent geometric grounding while diffusion only fills in high-frequency details.

• Three reference views are the sweet spot. Ablation shows 1 reference improves PSNR from 19.47 to 21.61, while 5 references give only marginal further gains (21.96) at higher memory cost.

• Feed-forward integration works. Applied to MVSplat360, PSNR rises from 16.691 to 17.976, LPIPS from 0.431 to 0.378, and FID from 13.462 to 11.097. Visual examples show details like windows and tree branches that MVSplat360 hallucinated are now faithfully recovered from input views.

Methodology in Plain English

The pipeline has two stages.

Stage one — build the 3D scene in latent space. Take a set of posed input images, encode each one into the VAE latent space of a pre-trained diffusion model (the Stable Diffusion VAE, downsampling by factor 8). Then run standard 3D Gaussian Splatting optimization, but replace the per-Gaussian RGB color with a 4-channel latent feature vector. Rendering a novel view from this latent radiance field produces a latent image — and a blurry one, because the high-frequency latent components cancelled out during optimization.

Stage two — restore the missing detail. The degraded rendered latent is placed in the top-left corner of a 2D grid. Three reference view latents (chosen as the nearest training cameras in position and orientation) fill the remaining grid cells. This grid is fed to a single-step diffusion model (Stable Diffusion Turbo, fine-tuned) at a fixed noise level. During denoising, the model's self-attention operates jointly across all grid cells, so high-frequency information from the references flows into the degraded latent. The refined top-left cell is then decoded back to image space by the VAE decoder.

Training. The refiner is supervised with three losses: L2 between the refined latent and the encoded ground-truth latent, LPIPS between the corresponding decoded images, and L2 between those decoded images. The model was fine-tuned on 400 DL3DV-10K scenes for roughly 24 hours on 8 NVIDIA H100 GPUs.

Feed-forward variant. For the MVSplat360 integration, the same refiner is applied to the latents rendered by a feed-forward network, before MVSplat360's video diffusion stage. The combined network is trained end-to-end for an additional 25K steps.

Why This Matters

Impact on research. The paper reframes a persistent problem in latent-space 3D reconstruction: instead of treating multi-view inconsistency in VAE latents as something to be engineered away in 3D, it treats detail recovery as a 2D task where attention across views can do the work. This reframing is likely to influence how future latent radiance field methods are designed, and the demonstration that a simple refiner can be bolted onto existing feed-forward models (MVSplat360) suggests broad composability.

Real-world applications:

• Sparse-view 3D capture for real estate, e-commerce, and cultural heritage, where a handful of photos must yield a navigable 3D scene.
• Immersive media pipelines (VR/AR, 360-degree video), where latent-space rendering offers memory and speed advantages.
• Autonomous driving and robotics simulation, where scene reconstruction must integrate with generative pipelines operating in latent space.
• Content creation tools and digital twins, where faithful preservation of scene detail — as opposed to plausible hallucination — is a hard requirement.

Industry relevance. All but one author is affiliated with Amazon Prime Video, and the results connect directly to production concerns: memory efficiency of latent rendering, compatibility with existing diffusion infrastructure, and reducing hallucinations that would be unacceptable in content pipelines. The ability to plug into MVSplat360 with minimal modification lowers the barrier to adoption.

Future Directions

• Extending beyond a frozen VAE. Since performance is fundamentally capped by the quality of the pre-trained VAE, a natural question is whether a lightweight adapter or a small set of fine-tuned decoder parameters could lift that ceiling without breaking diffusion compatibility.

• Generalizing the refiner across VAEs and diffusion backbones. The grid-based conditioning is deliberately architecture-agnostic, so the method could plausibly be adapted to newer VAEs and diffusion models as they appear.

• Improving the fidelity/efficiency trade-off. Ablations show quality saturates around three to five reference views, but memory grows with each view. Smarter reference selection or attention sparsification could remove this constraint.

• Closing the gap to RGB-space 3DGS. The authors concede that when RGB-space Gaussian Splatting already works well, it may be preferable. A key open question is whether latent-space methods can eventually match RGB-space quality outright, or whether latent space should be reserved for memory-constrained and generative-pipeline scenarios.

• Unified feed-forward refinement. The MVSplat360 integration shows promise, but training the whole stack end-to-end from scratch — rather than fine-tuning — could reveal larger gains.

Target Audience

Researchers and graduate students working on 3D reconstruction, novel view synthesis, and diffusion-based generative models will get the most from this paper, particularly those already familiar with 3D Gaussian Splatting and latent diffusion. Engineers building production 3D capture or immersive media pipelines will find the MVSplat360 integration and runtime discussion practically useful. Readers new to the field will need to brush up on 3DGS and VAE latent spaces first, but the central insight — recover detail in 2D from references rather than in 3D — is accessible and well motivated.

Authors’ abstract

Radiance field representations have recently been explored in the latent space of VAEs that are commonly used by diffusion models. This direction offers efficient rendering and seamless integration with diffusion-based pipelines. However, these methods face a fundamental limitation: The VAE latent space lacks multi-view consistency, leading to blurred textures and missing details during 3D reconstruction. Existing approaches attempt to address this by fine-tuning the VAE, at the cost of reconstruction quality, or by relying on pre-trained diffusion models to recover fine-grained details, at the risk of some hallucinations. We present Splatent, a diffusion-based enhancement framework designed to operate on top of 3D Gaussian Splatting (3DGS) in the latent space of VAEs. Our key insight departs from the conventional 3D-centric view: rather than reconstructing fine-grained details in 3D space, we recover them in 2D from input views through multi-view attention mechanisms. This approach preserves the reconstruction quality of pretrained VAEs while achieving faithful detail recovery. Evaluated across multiple benchmarks, Splatent establishes a new state-of-the-art for VAE latent radiance field reconstruction. We further demonstrate that integrating our method with existing feed-forward frameworks, consistently improves detail preservation, opening new possibilities for high-quality sparse-view 3D reconstruction. Code is available on our project page: https://orhir.github.io/Splatent/