Neural Texture Splatting: Expressive 3D Gaussian Splatting for View Synthesis, Geometry, and Dynamic Reconstruction

AI summary

Overview

• Research area: Computer vision and real-time rendering — specifically 3D Gaussian Splatting (3DGS) for novel view synthesis, surface (geometry) reconstruction, and dynamic 4D scene reconstruction.
• Technical level: Intermediate (requires familiarity with volume rendering, 3D Gaussian primitives, tri-planes, and photometric loss training).
• Scope (one sentence): The paper introduces Neural Texture Splatting (NTS), a plug-and-play module that attaches a global neural-field-predicted local RGBA texture to each Gaussian primitive and shows consistent gains across dense novel view synthesis, surface reconstruction, sparse-view static reconstruction, and sparse-view dynamic reconstruction.

What This Paper Is About

3D Gaussian Splatting represents a scene as a cloud of 3D Gaussian primitives, but each primitive can only describe color and opacity with a single fixed Gaussian kernel, which limits how much high-frequency detail and view-dependent variation it can express. Prior work added per-splat texture maps to each Gaussian, but the authors argue these approaches mainly help dense-view novel view synthesis with fewer primitives, and they lack view- and time-dependent variation while overfitting because each texture is optimized independently. NTS instead predicts each primitive's local texture from a single shared global neural field (a tri-plane plus a neural decoder), so the added capacity is compact, spatially consistent across neighboring splats, and can be conditioned on viewing direction and time.

Key Contributions

1. Neural Texture Splatting (NTS): a representation that augments 3DGS with neural per-primitive RGBA texture fields, described as a robust enhancement that improves 3DGS-based methods across a wide range of tasks.
2. A global tri-plane network for local textures: instead of optimizing each splat's texture independently, a global tri-plane plus neural decoder predicts localized texture fields, which the authors say both increases expressiveness and ensures robustness across tasks while reducing model size and enabling global information exchange.
3. Expressive view- and time-dependent local effects: by feeding viewing direction (and an optional timestep for dynamic scenes) into the decoder, the local texture fields can model view-dependent appearance and temporal variation, which the authors state existing per-splat texture methods fail to account for.
4. Extensive multi-task evaluation: the module is integrated into several state-of-the-art backbones and evaluated on dense and sparse-view novel view synthesis, surface reconstruction, and dynamic reconstruction, reportedly achieving state-of-the-art results.

Main Findings

• Overfitting in naive per-splat textures, and how the global field fixes it: Using GOF as backbone on DTU, a per-splat texture without the proposed global neural texture reaches training PSNR 36.05 on Scan 24 and 39.42 on Scan 105, but test Chamfer Distance is worse (0.48 and 0.70). With the global neural texture, training PSNR is lower (34.73 and 38.99) while test Chamfer Distance improves to 0.44 and 0.64 — the authors read this as reduced overfitting and better generalization.
• Dynamic reconstruction (Owlii, averaged over 4, 6, 8, and 10 training views): SplatFields4D + Ours reaches mean PSNR 29.25 versus 27.87 for SplatFields4D, 26.45 for 4DGaussians, 24.58 for Deformable3DGS, and 23.27 for 4D-GS. Mean SSIM rises to 95.86 from 95.09 (SplatFields4D), and mean LPIPS drops to 6.70 from 7.18. Per-scene PSNR for the authors' variant is 29.54 (Dancer), 28.74 (Exercise), 29.21 (Model), 29.50 (Basketball).
• Sparse-view static reconstruction (Blender, averaged over 4, 6, 8, and 10 views): SplatFields3D + Ours reaches mean PSNR 23.05, above SplatFields3D (22.58), 2DGS (21.44), LightGaussian (21.41), 3DGS (21.26), Mip-Splatting (20.75), SparseNeRF (20.88), ScaffoldGS (20.17), and SuGaR (19.43). Mean SSIM rises to 87.70 from 87.04. LPIPS is essentially flat: 13.50 versus 13.49 for SplatFields3D, with improvements on some scenes (Hotdog 9.55 → 8.22, Mic 5.62 → 5.46, Drums 13.06 → 12.92, Materials 17.44 → 17.20) and small regressions on others (Toy 13.86 → 14.03, Chair 11.63 → 11.89, Ship 28.07 → 28.75).
• Dense-view novel view synthesis: On Blender, GOF improves from 33.44 to 34.09 PSNR, and 3DGS-MCMC from 33.81 to 34.06. On MipNeRF360, GOF improves from 27.45 to 27.71 PSNR and 3DGS-MCMC from 28.14 to 28.24. Comparison rows include MipNeRF360 (30.34 and 27.69 PSNR), Instant-NGP (32.20 and 25.30), 3DGS (33.08 and 27.26), TexturedGaussians (33.24 and 27.35), and 2DGS (33.08 and 26.81).
• Surface reconstruction (DTU Chamfer Distance): GOF + Ours achieves CD 0.67 versus 0.74 for GOF and 0.80 for 2DGS. The paper notes the improvement is largest in specular regions, illustrated with an "apple" example that has fewer holes.
• Efficiency cost: On Blender with 3DGS-MCMC, PSNR improves 33.81 → 34.06 while model size grows 69MB → 87MB, training VRAM 6GB → 15GB, training time 5min → 36min, and rendering drops from 380 FPS to 85 FPS (with the same 0.29M primitives). On MipNeRF360 the same comparison goes 28.14 → 28.24 PSNR, 842MB → 860MB, 15GB → 43GB, 15min → 75min, 135 FPS → 37 FPS (3.56M primitives). For SplatFields on Blender: 22.58 → 23.05 PSNR, 74MB → 86MB, 8GB → 17GB, 80min → 135min, 430 FPS → 150 FPS (0.30M → 0.33M primitives). On Owlii: 27.87 → 29.25 PSNR, 28MB → 31MB, 7GB → 11GB, 14h → 23h, 38 FPS → 15 FPS (0.10M primitives).
• Where the runtime goes: For a single 800×800 image rendered from 0.3M primitives, rasterization takes 2.6 ms, ray-Gaussian intersection 4.2 ms, texture query 0.2 ms, and the neural network forward pass 4.7 ms.
• Qualitative gains attributed to view-dependence: The paper reports better handling of specular highlights and preservation of fine structures (for example the cable shown in the MipNeRF360 comparison), and sharper dynamic reconstructions with fewer floaters and boundary artifacts, particularly around human faces and hands.

Methodology in Plain English

The starting point is standard 3DGS: a set of 3D Gaussian "blobs," each with a position, rotation, scale, color, and opacity, composited front-to-back along each camera ray. NTS gives every blob an extra small texture — a set of three orthogonal learnable planes storing four channels (red, green, blue, alpha) — that adds extra color and extra opacity on top of what the Gaussian itself contributes.

To look up a texture value, the method takes the point where the camera ray intersects the Gaussian, converts that world-space point into the blob's own local coordinate frame (using its center, scale, and rotation), and samples the three planes at that local position with bilinear interpolation, averaging the three plane values. Query points outside a local bounding box of size 3 are discarded. The rendering equation is then modified so the texture color is added to the Gaussian color and the texture alpha is added to the Gaussian alpha, with the combined opacity clamped to [0, 0.99] to keep transmittance non-negative.

The important design choice is that these textures are not stored or optimized individually. Instead, a single global tri-plane encodes features for every blob at its center position. Those features are concatenated with the blob's center position, its viewing direction, and (for dynamic scenes) a timestep, and passed through a shallow neural network that outputs the local texture. Separate global tri-planes and decoders are used for color and opacity, and the view direction is only fed to the color network. To save computation and allow higher-resolution local planes, the network outputs 1D vectors that are combined by outer products (Canonical Polyadic decomposition) rather than predicting the full plane grids directly.

Training keeps the original 3DGS objective — an MSE/L1-style term plus a D-SSIM term with λ = 0.2 — and adds an L1 regularization on the tri-plane RGBA textures with a weight of 0.01 to encourage sparsity. When plugged into backbones such as GOF, the backbone's own auxiliary losses are retained. For dense-view experiments, methods are trained for 30k iterations: the backbone is pretrained for 5k iterations, then jointly trained with the module for another 25k, using identical hyperparameters across comparisons. Evaluation uses Blender and Owlii for sparse-view static and dynamic scenes, DTU for surface reconstruction, and Blender plus MipNeRF360 for dense novel view synthesis, with PSNR, SSIM, LPIPS, and Chamfer Distance on extracted meshes as metrics.

Why This Matters

Impact on research. The paper argues that the bottleneck in 3DGS is per-primitive representational capacity, and that the right fix is not more independent parameters per splat but a shared global field that produces them. The reported overfitting table (worse training PSNR but better test Chamfer Distance with the global texture) is a concrete example of implicit neural regularization helping generalization, echoing ideas from SplatFields. The approach is deliberately plug-and-play, demonstrated on GOF, 3DGS-MCMC, SplatFields3D, and SplatFields4D, so it can be layered onto future 3DGS variants rather than replacing them.

Real-world applications (as suggested by the paper's framing):

• AR/VR content capture and playback, where novel view synthesis and surface reconstruction quality matter.
• Robotics, where a reconstructed scene must be both visually faithful and geometrically correct for downstream use.
• Gaming and 3D content creation, a use case the introduction explicitly cites alongside AR/VR and robotics.
• Multi-camera capture of dynamic subjects (the Owlii setup uses a rotating camera trajectory with 100 training and 100 test frames), relevant to volumetric video and telepresence.
• Drone or handheld reconstruction of specular objects, which the paper highlights as a case where view-dependent modeling reduces holes in the recovered surface.

Industry relevance. The results come with an explicit efficiency price that matters for deployment: roughly 2–8x more training time, 1.5–3x more training VRAM, and 2.5–4.5x lower rendering throughput than the corresponding backbones, and rendering that in the reported setups is no longer comfortably real-time (15–150 FPS depending on dataset and backbone, versus 38–430 FPS for baselines). The inference breakdown shows the added cost is dominated by the ray-Gaussian intersection (4.2 ms) and the neural network forward pass (4.7 ms) rather than the texture query itself (0.2 ms). Model size stays nearly unchanged (for example 69MB to 87MB on Blender), which the authors attribute to the global neural network compressing per-primitive storage.

Future Directions

The paper content provided is truncated before any conclusions or explicit future work section, so the following are open questions raised by the reported results rather than stated author plans.

1. Can the runtime overhead be reduced? The neural network forward pass (4.7 ms) and ray-Gaussian intersection (4.2 ms) dominate the added cost, and rendering FPS falls as low as 15 on Owlii. Faster intersection or caching of local textures appears to be the natural next problem.
2. Why does LPIPS not improve in sparse-view static reconstruction? Mean LPIPS goes from 13.49 to 13.50 on Blender sparse-view while PSNR and SSIM improve, and a few per-scene LPIPS values get worse. Understanding whether the improvement is perceptually limited, or whether the loss/regularization weighting is the cause, is unresolved.
3. Does the approach scale beyond the tested settings? The reported evaluation covers 4 to 10 input views and the listed benchmarks; behavior at very large scene counts or widely different capture conditions is not reported.
4. How sensitive is the method to its design choices? The paper mentions an L1 texture regularizer with weight 0.01, a λ of 0.2 for the D-SSIM term, a local bounding box of size 3, and a texture resolution τ, and refers to an ablation study in Table 9 that is not included in the provided content; the trade-offs among these choices, and the 2D-texture alternative described as being in the supplementary material, are therefore not visible here.

Target Audience

Researchers and graduate students working on 3D Gaussian Splatting, neural rendering, and neural fields, particularly those interested in sparse-view reconstruction, surface reconstruction, or dynamic 4D scene modeling. It is also useful for practitioners integrating novel view synthesis into AR/VR, robotics, or content-creation pipelines, though they should weigh the reported training and rendering overheads. Readers need working familiarity with volume rendering and Gaussian primitives; beginners will find the abstract and method overview accessible, but the rendering equations and tri-plane/CP-decomposition details require an intermediate background.

Authors’ abstract

3D Gaussian Splatting (3DGS) has emerged as a leading approach for high-quality novel view synthesis, with numerous variants extending its applicability to a broad spectrum of 3D and 4D scene reconstruction tasks. Despite its success, the representational capacity of 3DGS remains limited by the use of 3D Gaussian kernels to model local variations. Recent works have proposed to augment 3DGS with additional per-primitive capacity, such as per-splat textures, to enhance its expressiveness. However, these per-splat texture approaches primarily target dense novel view synthesis with a reduced number of Gaussian primitives, and their effectiveness tends to diminish when applied to more general reconstruction scenarios. In this paper, we aim to achieve concrete performance improvement over state-of-the-art 3DGS variants across a wide range of reconstruction tasks, including novel view synthesis, geometry and dynamic reconstruction, under both sparse and dense input settings. To this end, we introduce Neural Texture Splatting (NTS). At the core of our approach is a global neural field (represented as a hybrid of a tri-plane and a neural decoder) that predicts local appearance and geometric fields for each primitive. By leveraging this shared global representation that models local texture fields across primitives, we significantly reduce model size and facilitate efficient global information exchange, demonstrating strong generalization across tasks. Furthermore, our neural modeling of local texture fields introduces expressive view- and time-dependent effects, a critical aspect that existing methods fail to account for. Extensive experiments show that Neural Texture Splatting consistently improves models and achieves state-of-the-art results across multiple benchmarks.