Direct Conditional Parameterization for N-Dimensional Splatting

AI summary

Overview

Research area: Computer vision and neural rendering, specifically splatting-based novel-view synthesis (3D Gaussian Splatting and its N-dimensional, view- and time-conditioned extensions).

Technical level: Advanced. The paper works with joint Gaussian distributions, Schur complements, Cholesky factorizations, and custom CUDA kernel design; the math is central rather than incidental.

Scope: The paper replaces the covariance-derived "conditional slicing" step shared by N-dimensional Gaussian Splatting (N-DGS) and Universal Beta Splatting (UBS) with a direct parameterization of the rendered quantities, yielding two new methods, d GS and d BS, evaluated on five static and dynamic benchmarks. (arXiv:2610.22778v1 [cs.CV], 16 Oct 2026; United Imaging Intelligence, Boston, MA, USA.)

What This Paper Is About

Splatting methods such as 6DGS, 7DGS (collectively N-DGS) and UBS lift each primitive into a joint distribution over 3D position and "query" variables (view direction, or view plus time), then conditionally slice that distribution at every rendering query. That slicing requires a per-primitive matrix inversion, a regression-matrix construction, and a covariance-correction product in every forward pass, and it entangles the optimization of opacity, position, and covariance into a single set of ambient coordinates.

The paper's goal is to keep the same conditioning behaviour while removing those operations: instead of deriving the conditional mean, opacity, and covariance from a joint covariance, the authors store the rendering-relevant quantities directly as explicit per-primitive parameters (a Cholesky precision factor, a scale-normalized spatial coupling factor, and a learned per-dimension coupling vector).

Key Contributions

1. Direct conditional parameterization for N-dimensional splatting. A kernel-agnostic replacement for covariance-derived conditioning that explicitly parameterizes opacity precision (a Cholesky factor L) and a factorized query-to-position regression operator with learnable per-dimension coupling Λ. A proof in Appendix C establishes that the unrestricted operational tuple forms a smooth, globally bijective coordinate chart for joint Gaussian covariances, with block-triangular reconstruction, an explicit inverse, and a nonvanishing coordinate-volume Jacobian throughout the SPD interior.

2. Two kernel instantiations with matched-pair evaluation. The formulation is developed for the Gaussian kernel (d GS, the direct counterpart of N-DGS) and extended to the Beta kernel (d BS, the direct counterpart of UBS). Pairing each direct method with its covariance-derived counterpart isolates the effect of direct conditioning from the choice of kernel.

3. An optimized CUDA implementation, including a new baseline. Fused, register-level kernels for direct conditioning and for the N-DGS/UBS baselines. The N-DGS baseline was previously only available as a PyTorch implementation; the paper provides an optimized CUDA comparison and reports 6.9–7.7× faster production slicing than on-the-fly covariance-derived conditioning.

4. Structural priors as an explicit design lever. A spatial-scale prior on the coupling factor (‖V_pq‖_F = s̄, the mean spatial scale) and a bounded coupling vector Λ ∈ [0,1]^C, which interpolates between an opacity-only variant (d GS-O at Λ = 0) and full coupling through the opacity precision (Λ = 1).

Main Findings

• Faster conditioning step: production CUDA slicing at 1M primitives on an A100 takes 0.10 ms for d GS (7.1×) and 0.10 ms for d BS (6.9×) at C = 3, versus 0.71 ms (N-DGS) and 0.67 ms (UBS); at C = 4 it takes 0.12 ms for d GS (7.7×) and 0.13 ms for d BS (7.2×), versus 0.96 ms and 1.34 ms.

• Complete-render gains are smaller than slicing gains: profiling four trained scenes gives 1.25–1.91× complete-render speedups, because slicing occupies 42–64% of baseline frame time and shared rasterizer code still processes different learned footprints and tile workloads.

• Gaussian-kernel pair quality: under MCMC densification, d GS reaches 34.34 dB on NeRF Synthetic (+0.30 dB over N-DGS, with FPS_CUDA 593.94 vs. 348.95), 27.67 dB on Mip-NeRF 360 (+0.79 dB, with 2,574K versus 2,852K primitives, roughly 10% fewer, and FPS_CUDA 200.56 vs. 75.31), 40.34 dB on 6DGS-PBR (+0.28 dB), 34.64 dB on D-NeRF (+0.01 dB, quality parity, FPS_CUDA 746.65 vs. 501.52), and 29.90 dB on 7DGS-PBR (+0.70 dB, FPS_CUDA 577.38 vs. 511.12).

• Beta-kernel pair quality: d BS reaches 34.53 dB on NeRF Synthetic (+0.05 dB over UBS, training in 5.48 vs. 6.80 minutes, 415.66 vs. 306.33 FPS_CUDA), 28.70 dB on Mip-NeRF 360 (+0.20 dB, 157.45 vs. 76.18 FPS_CUDA), 33.55 dB on D-NeRF (+0.64 dB, 463.09 vs. 371.81), and 31.87 dB on 7DGS-PBR (+1.26 dB, the largest reported gap, with per-scene PSNR gains reaching +2.06 dB on heart).

• Learned coupling beats both fixed extremes: on NeRF Synthetic, learned Λ gives 33.91 dB vs. 33.84 (Λ = 0) and 33.78 (Λ = 1); on 6DGS-PBR, 37.75 dB vs. 36.54 (Λ = 0, −1.21 dB) and 37.31 (Λ = 1, −0.44 dB). Diffuse scenes prefer weak coupling, specular scenes prefer strong coupling, and per-Gaussian learning improves on both.

• The spatial-scale prior is critical: removing s̄ degrades PSNR by 2.65 dB on NeRF Synthetic (33.91 → 31.26) and 2.97 dB on 6DGS-PBR (37.75 → 34.78). An unscaled coupling factor is worse than no position regression at all (31.26/34.78 versus the Λ = 0 baseline at 33.84/36.54).

• Structure beats a bare direct readout: an unrestricted direct-regression control reaches 32.91/36.44 dB versus d GS at 33.91/37.75 dB, indicating the factorization and its priors matter beyond simply avoiding inversion.

• Against a broader SOTA field (Appendix A): d BS achieves the best PSNR on Mip-NeRF 360 (28.70 dB) and ties for best PSNR on NeRF Synthetic (34.53 dB), leading UBS on PSNR and SSIM across both benchmarks and on LPIPS for NeRF Synthetic.

• Fused CUDA helps both families: on Mip-NeRF 360, moving from PyTorch to fused CUDA slicing raises N-DGS mean FPS by 2.11× and d GS mean FPS by 4.72×.

Methodology in Plain English

Think of each splat as a small blob that also carries extra "query" dimensions, such as which direction you are looking from or what moment in time it is. To render, the method must ask: given this particular view or timestamp, what does this blob look like right now — where has its centre moved to, how transparent is it, and what shape does it have? In prior N-dimensional splatting, these answers are computed on the fly from a larger joint covariance: you invert the query block, build a regression matrix, and apply a correction term. That is the expensive part, and it happens once per primitive on every frame.

The authors' move is to store the answers' ingredients instead of deriving them. Three learned numbers per primitive do the work: a triangular Cholesky factor L, whose product L Lᵀ is the query precision matrix by construction (so no inversion is ever needed, only a small triangular product Lᵀδ); a spatial coupling factor V_pq, written as the primitive's mean spatial scale s̄ times a unit-norm learned direction; and a coupling vector Λ, one logit per query type, that dials how strongly the position shift follows the same precision the opacity uses. Setting Λ = 0 gives position-independent, opacity-only conditioning (d GS-O); Λ = 1 shares the full opacity precision, mirroring N-DGS; intermediate values are learned.

The spatial covariance actually rendered is kept identical to 3DGS and stays query-independent, so existing densification strategies carry over. Because only the conditioning step changes, and any opacity decay that consumes Lᵀδ can be plugged in, the same recipe instantiates with a Beta kernel as d BS, using the bandwidth for coupling (Λ_β = min(β/4, 1)) rather than a separate learned Λ.

The engineering half matters as much as the math: the fused CUDA kernels are templated on C (3 or 4) with compile-time unrolling, one thread per primitive, and the tiny matrices held entirely in registers, so no intermediate tensors or shared memory are touched. The same treatment is applied to the baselines (cofactor inverse for C = 3, register-local Gauss–Jordan for C = 4) so the comparison is apples-to-apples.

Evaluation uses PSNR, SSIM, and LPIPS under the 3DGS protocol (joint-RGB PSNR, white background on NeRF Synthetic) across five datasets — NeRF Synthetic (8 scenes), Mip-NeRF 360 (9 scenes), 6DGS-PBR (7 scenes), D-NeRF (8 sequences), and 7DGS-PBR (6 scenes) — on a single NVIDIA A100 with the TC-GS rasterizer, Adam, 30K iterations. UBS is re-evaluated under the same protocol because its released evaluator averages channel-wise PSNR and uses a black background on NeRF Synthetic.

Why This Matters

Research impact. The conditional-slicing step is inherited identically by N-DGS and UBS, so it is a shared bottleneck across two kernel families rather than a quirk of one codebase. Replacing it with an explicit, provably equivalent coordinate chart gives the N-dimensional splatting line a cheaper and more controllable foundation, and it separates two design axes that were previously conflated: which kernel you use, and how you condition on the query.

Real-world applications:

• Real-time free-viewpoint and light-field playback, where view-dependent specular highlights must respond to head or camera motion without dropping frame rate.
• Dynamic medical and physiological volume rendering, the authors' own institutional context (United Imaging Intelligence), where time-conditioned primitives capture motion such as a beating heart.
• Interactive product visualization and e-commerce, where physically-based view-dependent materials (the 6DGS-PBR and 7DGS-PBR settings) need to be inspected from arbitrary angles.
• Time-varying capture for film, sports, or telepresence, where the 7DGS-PBR setting of dynamics combined with view-dependent effects is the hardest case tested.

Industry relevance. Splatting is deployed in latency-sensitive pipelines, and the reported cost is per-primitive, so it scales with the million-scale primitive counts in large scenes. A 6.9–7.7× cut in the slicing stage — with only 1.25–1.91× end-to-end — tells practitioners exactly where the remaining budget sits. The paper also ships an optimized CUDA baseline where only PyTorch existed before, which matters for anyone benchmarking against these methods.

Future Directions

• Extending direct conditioning to other primitive attributes, other conditioning queries, and other kernel families beyond Gaussian and Beta, as the authors themselves propose.
• Validating memory use and visibility handling for city-scale scenes and kernels not covered here — the stated limitation is that the five-benchmark evaluation does not establish behaviour at that scale.
• Closing the gap between slicing speedups (6.9–7.7×) and complete-render speedups (1.25–1.91×), since 36–58% of baseline frame time sits downstream in projection, binning, sorting, and rasterization.
• Understanding the coupling geometry better: learned Λ beat both fixed extremes and scene preferences diverged (weak coupling on diffuse NeRF Synthetic, strong on specular 6DGS-PBR), suggesting per-primitive coupling could be analysed or initialized more principledly.

Target Audience

Researchers and engineers already working with 3D Gaussian Splatting or N-dimensional splatting who want to understand where the per-primitive cost concentrates and how to restructure it. It is most useful to readers comfortable with multivariate Gaussians, Schur complements, and GPU kernel optimization; the matched-pair design (d GS vs. N-DGS, d BS vs. UBS) also makes it valuable for anyone who needs a fair kernel-family comparison. Readers new to splatting will need to start from 3DGS first, since the paper assumes familiarity with the joint-conditioning formulation it is replacing.

Authors’ abstract

N-dimensional splatting extends 3D Gaussian Splatting with view direction and time to model view-dependent and dynamic effects. Each primitive is lifted into a joint distribution over position and conditioning variables, then conditionally sliced at each rendering query. N-DGS and UBS share this step across Gaussian and Beta kernels: recovering the conditional mean, opacity, and covariance requires per-primitive matrix inversions and regression products in every forward pass. We introduce direct conditional parameterization, replacing this covariance-derived readout with explicit per-effect parameters: a Cholesky precision factor for opacity and a factorized query-to-position regression operator with learnable per-dimension coupling. Unrestricted conditional coordinates recover a valid joint covariance; our factorization adds spatial-scale and coupling priors. Applied to both kernels, the parameterization yields dGS (Gaussian) and dBS (Beta). Across five static and dynamic benchmarks, they match or exceed their covariance-derived counterparts on quality, with gains up to +1.26 dB in the main MCMC evaluation. Production slicing is 6.9-7.7 times faster than on-the-fly covariance-derived conditioning; complete-render profiling gives 1.25-1.91 times speedups on four trained scenes. Direct conditioning provides a structured, kernel-agnostic alternative for N-dimensional splatting. Project page: https://gaozhongpai.github.io/dgs/.