T$^2$-Splat: Adaptive Topology Mesh Splatting with Texture Residuals

AI summary

Overview

Research area: Computer vision — 3D scene reconstruction and novel view synthesis (NVS), specifically mesh-based differentiable splatting that bridges neural rendering and conventional graphics pipelines.

Technical level: Advanced. The paper assumes familiarity with 3D Gaussian Splatting, differentiable rasterization, Delaunay triangulation, spherical harmonics, and mip-mapping.

Scope: The paper introduces T²-Splat, a two-stage framework that adapts mesh topology through edge collapse and vertex split operations and then refines appearance with residual per-face Mip-textures, evaluated on Mip-NeRF 360, Tanks & Temples, and DTU.

What This Paper Is About

Triangle-based splatting methods connect novel view synthesis to standard graphics pipelines, but they typically need extremely dense tessellation to preserve fine appearance details — because the same triangles that define geometry also carry the appearance signal. This creates a coupling problem: collapsing triangles to make the mesh compact blurs textures and introduces aliasing, while keeping detail requires redundant faces in smooth regions.

T²-Splat's goal is to break that coupling, letting the mesh adapt its topology to local scene complexity while a separate texture model recovers high-frequency appearance on the resulting coarse mesh.

Key Contributions

1. A decoupled topology-and-texture framework (T²-Splat). Geometry/topology optimization and appearance refinement are separated into two stages, so triangle allocation no longer has to be dense enough to encode texture.

2. Dynamic Topology Evolution (DTE) inside a differentiable renderer. Progressive mesh operations — QEM-based edge collapse and gradient-driven vertex split — are integrated into the differentiable pipeline so triangles can be reallocated according to scene complexity while preserving well-conditioned manifold connectivity, with Adam optimizer moments transferred across edits (optimizer state stitching).

3. Safe Restricted Delaunay Triangulation (Safe RDT). A constrained face-selection procedure that filters and ranks RDT candidates by distance to the input triangle soup, unsigned normal agreement, and triangle quality, then greedily grows a consistently oriented surface (no duplicate faces, at most two incident faces per edge, single connected face fan per vertex).

4. A scale-aware residual per-face Mip-texture model. Zero-initialized residual textures attached to triangle faces, organized in a mip pyramid with trilinear sampling driven by screen-space derivatives, decoupling high-frequency appearance from mesh density and suppressing aliasing.

Main Findings

• Rendering quality on Mip-NeRF 360: T²-Splat reaches 25.49 dB PSNR, 0.234 LPIPS, and 0.779 SSIM with 2M vertices, versus MeshSplatting at 24.78 dB, 0.310 LPIPS, 0.728 SSIM with 3M vertices — a 0.71 dB PSNR gain, as reported in the abstract and Table 1.

• Tanks & Temples: 21.12 dB PSNR, 0.213 LPIPS, 0.795 SSIM with 1M vertices, compared with MeshSplatting's 20.52 dB, 0.287 LPIPS, 0.745 SSIM and 2M vertices. The paper notes GOF and MiLo obtain higher PSNR on this dataset, while T²-Splat has the best LPIPS and SSIM among the evaluated mesh-based renderings.

• Surface reconstruction on DTU (15 scenes): A mean Chamfer distance of 0.68, described as joint second-best, with the lowest error on three scenes (63, 83, and 105). Gaussian Wrapping reports the best mean at 0.49; RaDe-GS and MiLo also report 0.68.

• OpenGL rendering efficiency (Garden scene, 1920×1080, MacBook Air M3): 236 FPS versus 126 FPS for MeshSplatting (approximately 1.87×), with storage reduced from 152 MB to 62 MB (about 59% reduction) and 2.7M faces versus 7.3M. GOF and RaDe-GS run out of memory in this configuration.

• Extreme mesh compression: On Garden, the full model achieves 25.68 dB PSNR, 0.15 LPIPS, 0.818 SSIM with 2.7M faces; the 50% budget gives 25.38 dB, 0.17 LPIPS, 0.801 SSIM with 1.4M faces; the 10% budget gives 23.93 dB, 0.30 LPIPS, 0.697 SSIM with 0.3M faces. The 50% configuration still surpasses MeshSplatting (24.70 dB, 0.22 LPIPS, 0.762 SSIM, 7.3M faces).

• Ablation — Safe RDT: Reduces average face count from 5.3M to 3.0M (about 43%) while raising PSNR from 24.66 to 24.81 dB and SSIM from 0.734 to 0.740; LPIPS rises slightly from 0.30 to 0.31.

• Ablation — DTE: Adding DTE to Safe RDT improves PSNR by 0.31 dB while reducing faces from 3.0M to 2.5M. Combined with textures, it adds 0.23 dB for 4×4 textures and 0.18 dB for 8×8 textures.

• Ablation — Mip-textures: On the DTE-refined mesh, 4×4 textures raise PSNR from 25.12 to 25.43 dB; 8×8 textures reach 25.49 dB, lower LPIPS from 0.26 to 0.23, and raise SSIM from 0.775 to 0.779 at unchanged face count, while training time grows from 72 to 83 minutes.

• Full configuration versus baseline: +0.83 dB PSNR over the baseline configuration (a), roughly 53% fewer faces, and training time increasing from 54 to 83 minutes.

Methodology in Plain English

The method runs in two sequential stages, both differentiable.

Stage I — geometry and topology. The system starts from an unstructured "triangle soup" produced by the underlying triangle-splatting pipeline. Safe RDT then rewires the connectivity: from candidate faces it keeps only those that are close to the original soup, agree in normal direction, and are well-shaped, and it grows a clean, consistently oriented surface from them. Crucially, this step changes only connectivity and winding — vertex positions, appearance, opacity, and optimizer state are preserved.

The mesh is then edited periodically by Dynamic Topology Evolution. Two reversible operations are used. Edge collapse removes triangles in flat or redundant regions: each vertex accumulates a quadric error matrix from its adjacent faces, the cost of merging two vertices is the quadric error normalized by the median error plus base-color and opacity difference terms, and collapses are rejected if they violate the simplicial link condition, join two boundary vertices through an interior edge, or create degenerate or flipped faces. Vertex split adds triangles where detail is needed: vertices are ranked by a perspective-compensated gradient score (position-gradient magnitude times camera-space depth, averaged over the views in which the vertex is visible), and high-scoring vertices are split at the midpoint of their longest adjacent edge, with attributes linearly interpolated. To avoid discarding learned momentum, Adam's first and second moments are carried across both operations — area-weighted averaging for collapses, and symmetric perturbations for splits.

Stage II — appearance. With topology and vertex positions frozen, the mesh gets a residual texture grid per face (8×8 RGB by default), initialized to zero so training starts exactly where Stage I left off. A pixel's color is the vertex-interpolated, view-dependent spherical-harmonic color plus a bilinearly sampled residual from the face's texture. A mirror mapping reflects coordinates with u+v>1 back into the triangle so the square texture domain is used efficiently without boundary artifacts. Each texture is stored as a mip pyramid built by average pooling, and the mip level is chosen from screen-space derivatives of the texture coordinates, with trilinear interpolation across adjacent levels — a dynamic low-pass filter that reduces flicker at distance while keeping detail up close. Vertex SH coefficients and textures are optimized jointly with L1, SSIM, total-variation, and L2 losses; the TV and L2 terms keep textures as a high-frequency correction rather than a replacement for base shading.

Training uses 35,000 iterations on a single NVIDIA RTX 6000 Ada GPU (48 GB): 30,000 for geometry and base appearance, 5,000 for texture refinement. Mesh budgets are controlled simply by adjusting the relative rates of vertex split and edge collapse, after which Stage II runs on the fixed mesh.

Why This Matters

Impact on research. The paper targets a concrete structural weakness in mesh-native splatting — the "topological rigidity" it attributes to RDT-based methods such as MeshSplatting, where connectivity is frozen after initialization. Showing that progressive-mesh operations can live inside a differentiable renderer, and that appearance can be decoupled from tessellation density, gives the field a template for compact, editable, graphics-native scene representations rather than purely point-based ones.

Real-world applications.

• Game engines and interactive 3D: opaque, connected, depth-bufferable meshes at 236 FPS with 62 MB storage fit asset pipelines far better than alpha-blended Gaussian clouds.
• AR/VR: compact meshes with controllable face budgets suit the tight memory and latency limits of head-mounted devices.
• Digital twins and scene capture: a single pipeline producing both renderable meshes and competitive surface geometry (Chamfer 0.68 mean on DTU) supports scanning-to-asset workflows.
• Streaming and mobile delivery: the demonstrated 50% and 10% face-budget modes (1.4M and 0.3M faces) give a direct quality-versus-bandwidth dial.

Industry relevance. The explicit export path to PLY and GLB, zero over-drawing at convergence (one triangle evaluation per pixel), and standard OpenGL rasterization compatibility make the output directly consumable by existing tooling — the gap the paper identifies between neural rendering and conventional graphics is precisely what commercial engines and authoring tools require.

Future Directions

• Unifying the two stages. The current design deliberately freezes topology before texture refinement; allowing topology edits and texture learning to interact jointly could recover further quality at the same face budget, at the cost of the stability the freeze provides.

• Sharper quality-versus-compactness control. The 10% budget drops PSNR to 23.93 dB from 25.68 dB on Garden. Budget-aware allocation policies — deciding where to spend faces under a hard global constraint rather than tuning split/collapse rates — remain open.

• Generalizing beyond the evaluated benchmarks. All results are on Mip-NeRF 360, Tanks & Temples, and a 15-scene DTU evaluation; behavior on large-scale outdoor captures, dynamic scenes, and transparent or thin structures is not reported in the provided content.

• Reducing optimization cost. Training rises from 54 to 83 minutes for the full configuration, and to 83 minutes with 8×8 textures versus 72 with 4×4 — efficiency gains in the texture stage and topology scheduling would widen practical applicability.

Target Audience

Researchers and graduate students working on differentiable rendering, novel view synthesis, and mesh-based scene reconstruction — particularly those already familiar with 3DGS, MeshSplatting, or Triangle Splatting who want compact, graphics-pipeline-ready representations. Graphics engineers evaluating whether mesh-native neural reconstruction can meet real-time and memory constraints will find the FPS, storage, and face-budget results the most directly useful part. Beginners can follow the high-level motivation but will need background in differentiable rasterization and mesh simplification to engage with the method.

Authors’ abstract

Triangle-based splatting connects novel view synthesis with conventional graphics pipelines, but often requires excessive tessellation to preserve appearance details. We present T$^2$-Splat, which decouples topology adaptation from appearance refinement. Safe Restricted Delaunay Triangulation (Safe RDT) initializes mesh connectivity. Dynamic Topology Evolution (DTE) then adapts triangle allocation through edge collapse and vertex split operations. Residual per-face Mip-textures then refine appearance on the fixed mesh using scale-adaptive filtering. Experiments on Mip-NeRF 360, Tanks & Temples, and DTU demonstrate competitive results with compact meshes. T$^2$-Splat improves PSNR by 0.71dB over MeshSplatting on Mip-NeRF 360. Additional compression experiments demonstrate flexible control of mesh complexity, supporting face budgets as low as 10% of the original face count. Project page: https://t2-splat.github.io/.