Mobile-4DGS: Unified Static-Dynamic Real-time Mobile Gaussian Splatting

AI summary

Overview

Research area: Computer Vision — real-time 3D/4D Gaussian Splatting on resource-constrained mobile devices.

Technical level: Advanced. The paper assumes familiarity with 3D Gaussian Splatting, spherical harmonics, differentiable rendering and mobile GPU rasterization pipelines.

Scope: A single deployment-oriented framework that unifies static and dynamic Gaussian Splatting on mobile hardware by compressing appearance to first-order spherical harmonics, controlling primitive count with multi-view densification/pruning, and reusing cached depth orders during 4D playback.

What This Paper Is About

Getting 3D Gaussian Splatting to run in real time on phones is hard because millions of Gaussians carry heavy per-primitive storage (notably third-order spherical harmonics), accumulate redundant points during training, and — for dynamic scenes — require per-frame neural deformation and a full re-sort of depth order. Mobile-4DGS attacks all four bottlenecks at once, producing one pipeline that handles static and dynamic novel view rendering with a compact explicit representation.

Key Contributions

1. Unified mobile static-dynamic representation. The static case is treated as the zero-motion special case of the same explicit representation, so both modes share appearance, compression, density-control and rendering code. Each dynamic Gaussian needs only eight additional temporal scalars, with second-order Gaussian motion, learnable temporal support and a binary static-dynamic partition — continuous-time modeling without a runtime deformation network.

2. Monte Carlo Specular Energy Aggregator. High-order radiance residuals are compressed into first-order spherical harmonics by sampling directional residuals on a uniform sphere and extracting first-order directional moments (energy magnitude and dominant direction), avoiding offline distillation or pretraining. An Attribute-Conditioned SH Enhancement module predicts residual offsets from intrinsic Gaussian attributes; because it is camera-independent, those offsets are baked into the explicit parameters before inference, adding no runtime cost.

3. Multi-view alpha-based densification and pruning. Stratified camera sampling selects angularly dispersed views, and alpha-weighted error accumulation across those views yields importance and pruning scores that gate cloning, splitting and pruning, keeping primitive counts compact while preserving multi-view structural consistency.

4. Depth-Order Certificate (DOC) for 4D playback. Because the learned static-dynamic partition is exactly binary, the static subset's depth order depends only on the camera and is cached for the whole clip; a closed-form bound on the travel of the animated subset decides per frame whether the committed draw list is still valid. Certified frames skip re-projection, sorting, merging and index-buffer upload.

Main Findings

• Per-Gaussian memory compression: Figure 2 reports that Mobile-4DGS achieves 61% and 26% reductions in per-Gaussian memory footprint relative to 3DGS variants, by optimizing spherical harmonic coefficients and decoupling SH into base and view-independent components.
• First-order SH fidelity: The paper reports that Mobile-4DGS with only first-order SH renders high-fidelity, high-frequency detail comparable to 3DGS in the qualitative comparison of Figure 2(b).
• Speed and primitive count: Figure 1 reports rendering quality comparable to both 3DGS and Mobile-GS while using fewer Gaussian primitives and achieving significantly higher FPS on a mobile device with a Snapdragon 8 Gen 3 GPU.
• Four bottlenecks identified: Third-order SH overhead (48 floating-point parameters per Gaussian primitive, i.e. 16 coefficients × 3 channels); redundant primitive densification from single-view gradient criteria; MLP-based dynamic representation requiring per-frame deformation inference; and per-frame depth ordering under motion invalidating cached orders.
• Compression ratio of SH storage: Constraining appearance to first-order SH requires only 4 coefficients per channel, versus 16 coefficients × 3 channels for third order.
• Zero runtime overhead for SH enhancement: The SH offsets and the decoded first-order coefficients require a one-time decoding step before inference, introducing no per-frame decoding overhead.
• Exactness of the certificate: The paper states the certificate leaves the rendered image untouched on every frame it reuses, and the budget branch confines admitted reordering to primitives whose footprints already overlap.
• Aggregate experimental claim: Extensive experiments on static and dynamic scenes show substantial reductions in storage and rendering overhead with competitive visual quality — specific PSNR, SSIM, LPIPS, FPS or dataset-size numbers are not reported in the provided excerpt.
• Code availability: Code is released at https://xiaobiaodu.github.io/mobile-4dgs-project/.

Methodology in Plain English

The pipeline starts by training normally with full third-order spherical harmonics for the first 3,000 iterations, which establishes a high-frequency appearance. After that, the model performs a one-time conversion: it samples many directions on a sphere, measures the residual between the third-order and second-order appearance, and keeps only the dominant direction and magnitude of that residual. Those compact statistics, plus the base (zeroth-order) color and a normalized Gaussian position, are fed to a small MLP that produces first-order SH coefficients. A second lightweight MLP looks at each Gaussian's own properties — position, scale, rotation, opacity and its current SH — and predicts a small correction, but since this correction does not depend on the camera, it is computed once and baked into the stored parameters before deployment. Static and dynamic scenes are then treated identically: dynamic Gaussians simply add a small number of temporal parameters and motion terms, while static Gaussians are the zero-motion case.

For point count, instead of deciding which Gaussians to clone, split or delete from a single camera view, the method picks a spread-out subset of cameras (stratified angular binning around an estimated scene center), renders them, and flags pixels as badly or well reconstructed using error thresholds on an L1 plus DSSIM loss. Each Gaussian's alpha contributions to those flagged pixels are summed across views to give an importance score and a pruning score; cloning and splitting only happen for Gaussians that are both flagged by the standard gradient rule and above an importance quantile, and pruning only removes Gaussians that fall below the opacity threshold and above a pruning-score quantile. Finally, for 4D playback, the binary separation of static and moving Gaussians means the static depth order is cached once per camera, while a mathematical bound on how far the moving Gaussians can travel tells the renderer whether the existing draw list is still correctly sorted — if it is, all the per-frame sorting work is skipped.

Why This Matters

Research impact: The paper reframes mobile Gaussian Splatting as a joint representation-plus-renderer co-design problem rather than a pure compression problem. Two ideas are likely to be reused independently: compressing high-order radiance into low-order SH by preserving directional statistics rather than absolute residuals, and deriving the cache-invalidation rule for the renderer directly from the representation (the binary static-dynamic partition) instead of adding a separate engineering layer.

Real-world applications:

• On-device augmented reality navigation and wayfinding overlays that must run during a phone's normal power budget.
• Immersive virtual reality and telepresence, where dynamic 4D avatars must be rendered at frame rate inside a headset or phone.
• Autonomous driving and robotics visualisation, where reconstructed 4D scenes of a drive are played back for inspection on tablets or in vehicles.
• Cross-platform web viewing of captured scenes, since the framework is reported to use WebGL for both static and dynamic novel view rendering.

Industry relevance: Storage footprint and memory bandwidth, not floating-point throughput alone, are the practical limits on shipping Gaussian Splatting in mobile apps. A per-Gaussian memory reduction reported at 61% and 26% against 3DGS variants, combined with fewer primitives and no per-frame neural decoding, moves the technique closer to viable deployment on commodity phone hardware, and a unified static/dynamic path reduces the engineering cost of shipping one renderer instead of two.

Future Directions

• Quantitative benchmarking on standard datasets: The abstract and figures claim competitive quality, but per-scene PSNR, SSIM and LPIPS on recognised static and dynamic benchmarks, along with measured on-device latency and memory, are not reported in the provided excerpt and are the obvious next evaluation step.
• Extending temporal modeling beyond second-order motion: The representation uses second-order Gaussian motion with learnable temporal support; how it handles fast, non-smooth or topologically changing motion (for example large deformations or objects entering and leaving) is an open question.
• Generalising the depth-order certificate: The certificate relies on an exactly binary static-dynamic partition and a closed-form travel bound; relaxing either assumption (soft partitioning, very large motion bounds) may widen the gap to the certified fast path.
• Storage and quality trade-offs beyond first-order SH: It is not yet established how far the Monte Carlo first-order SH approach scales to scenes with extreme specular content, or how it interacts with existing vector-quantisation and entropy-coding compression pipelines.

Target Audience

Researchers and engineers working on Gaussian Splatting, real-time rendering and mobile/edge graphics; practitioners building AR, VR or web-based scene viewers who need to know what compression and rendering shortcuts survive on a phone GPU; and readers already familiar with Mobile-GS and Flux-GS who want the journal-level unification of those two lines of work, including the newly added 4D extension.

Authors’ abstract

Recent advances in 3D Gaussian Splatting (3DGS) have achieved remarkable performance in novel view synthesis, yet deploying both static and dynamic Gaussian representations on resource-constrained mobile devices remains challenging due to heavy storage, redundant primitives, and costly per-frame computation. We present Mobile-4DGS, a unified lightweight framework for high-fidelity real-time static and dynamic Gaussian rendering on mobile platforms. For compact appearance modeling, we introduce a Monte Carlo Specular Energy Aggregator that compresses high-order radiance residuals into the first-order Spherical Harmonics (SH), together with an Attribute-Conditioned SH Enhancement module whose predicted offsets are pre-baked before inference. We further propose a Multi-View Alpha-Based Densification and Pruning strategy to suppress redundant primitives while maintaining multi-view consistency. For dynamic scenes, we develop a compact explicit 4D representation by constructing second-order Gaussian motion, learnable temporal support, and a binary static-dynamic partition, enabling continuous-time modeling without runtime deformation networks. Based on this partition, a Depth-Order Certificate selectively reuses previously committed depth orders to reduce re-projection, sorting, merging, and index-buffer updates during playback. Extensive experiments on static and dynamic scenes demonstrate that Mobile-4DGS substantially reduces storage and rendering overhead while maintaining competitive visual quality, enabling real-time 3D and 4D Gaussian Splatting on mobile devices. \textcolor{magenta}{\href{https://xiaobiaodu.github.io/mobile-4dgs-project/}{Code has been released: https://xiaobiaodu.github.io/mobile-4dgs-project/}}.