MoE-GS: Mixture of Experts for Dynamic Gaussian Splatting

AI summary

Overview

Research area: Dynamic novel view synthesis and 4D scene reconstruction using 3D Gaussian Splatting (3DGS), combined with Mixture-of-Experts (MoE) architectures.

Technical level: Intermediate. Readers need a basic grasp of NeRF/3DGS rendering, deformation fields, and the general MoE idea (experts plus a gating router), but the paper explains its own terminology.

Scope in one sentence: The paper introduces MoE-GS, the first framework to combine several heterogeneous dynamic Gaussian splatting models inside a Mixture-of-Experts architecture with a "Volume-aware Pixel Router," plus efficiency techniques (single-pass rendering, gate-aware pruning) and a distillation strategy, evaluated on the N3V and Technicolor datasets.

What This Paper Is About

Existing dynamic 3D Gaussian Splatting methods each encode a different motion assumption (deformation networks, polynomial trajectories, keyframe interpolation), and the authors' empirical analysis shows that no single method wins across all scenes, all spatial regions, or all frames of a video. MoE-GS addresses this by treating several such methods as "experts" and learning to blend their rendered images adaptively, per pixel, based on volumetric Gaussian cues. The goal is higher reconstruction quality rather than lower FLOPs: unlike sparsity-driven MoE in large language models, capacity is increased on purpose, with separate techniques to keep the added cost practical.

Key Contributions

1. MoE-GS: the first dynamic Gaussian splatting framework built on a Mixture-of-Experts architecture, blending multiple Gaussian-based dynamic reconstruction models for adaptive reconstruction across diverse scenes.
2. Volume-aware Pixel Router: a router that projects Gaussian-level weights into pixel space through differentiable weight splatting, producing spatially and temporally coherent per-pixel expert blending.
3. Efficiency mechanisms: single-pass multi-expert rendering (shared projection and visibility computation) and gate-aware Gaussian pruning (removing Gaussians with negligible influence on gating weights).
4. Distillation-based expert training: individual experts are retrained from scratch using ground-truth images plus router-weighted pseudo-labels from the MoE model, so a single expert can approach MoE-GS quality without architectural changes.

Main Findings

• No single prior generalizes: The authors' analysis identifies three failure modes of individual methods — scene-level variation, spatial-level inconsistency within a scene, and frame-to-frame temporal fluctuation in which method performs best.
• MoE outperforms all listed baselines on N3V: MoE-GS reaches average PSNR of 32.54 (N=2), 33.23 (N=3), and 33.27 (N=4). The strongest listed baseline averages are E-D3DGS at 32.33, SaRO-GS at 32.15, Ex4DGS at 32.10, STG (trained on a 150-frame split) at 31.92, and 4DGaussians at 31.43.
• MoE outperforms on Technicolor: MoE-GS (N=3) averages 34.55 PSNR, compared with STG 33.69, Ex4DGS 33.45, E-D3DGS 33.06, HyperReel 32.73, DyNeRF 31.80, and 4DGaussians 30.79.
• Router design matters most: With a Volume-aware Pixel Router, PSNR is 33.23 with SSIM 0.954 and LPIPS 0.021. A pure Pixel Router yields 31.12 / 0.952 / 0.022 and a Volume Router yields 32.05 / 0.951 / 0.022. The Pixel Router produces overly smooth results, while the Volume Router shows optimization instability and oversmoothing artifacts in fine-texture regions.
• Efficiency techniques interact: For N=3, removing both single-pass rendering and pruning gives 32.54 PSNR at 36 FPS and 747 memory; removing only single-pass gives 33.23 PSNR, 40 FPS, 270 memory; removing only pruning gives 32.54 PSNR, 60 FPS, 747 memory; the full model gives 33.23 PSNR, 68 FPS, 270 memory.
• Pruning trades quality for speed: For N=2 on N3V, the unpruned MoE-GS reports 32.82 PSNR at 44 FPS and 878.7 memory; 55% pruning gives 32.80 PSNR at 83 FPS and 351.2 memory; 75% pruning gives 32.45 PSNR at 101 FPS and 281.3 memory. For reference, the table lists STG at 31.92 PSNR / 88.5 FPS / 609.5 memory and Ex4DGS at 32.01 PSNR / 120 FPS / 122.8 memory.
• Experts do not need full training budgets: With N=3, MoE-GS scores 33.23 at 100% expert budget, 32.71 at 50%, 32.60 at 20%, and 32.14 at 10%. At 20% budget it still exceeds fully trained E-D3DGS (32.33), STG (31.92), and 4DGaussians (31.43). Full training times cited are approximately 4.2 hours for E-D3DGS, 2.2 hours for STG, and 1.5 hours for 4DGaussians.
• Router overhead is small: The router adds less than 5% computation relative to expert training.
• Distillation lifts every expert: On Technicolor, E-D3DGS improves from 32.88 PSNR / 0.902 SSIM / 0.111 LPIPS to 33.67 / 0.915 / 0.091; STG from 32.83 / 0.915 / 0.083 to 33.10 / 0.917 / 0.082; Ex4DGS from 33.57 / 0.918 / 0.086 to 33.91 / 0.923 / 0.079.
• Reported numbers are not fully consistent: The N3V tables list the N=2 configuration at 32.54 average PSNR (Table 1) and 32.82 (Table 3), and Ex4DGS at 32.10 (Table 1) versus 32.01 (Table 3). The paper does not explain these differences.
• Not reported: The paper does not state the number of frames or scenes in N3V and Technicolor beyond the six and five scenes it lists, does not give units for the memory column, and does not report a total end-to-end wall-clock training time for the full MoE-GS pipeline.

Methodology in Plain English

The framework proceeds in two training stages. First, each expert (Ex4DGS, STG, E-D3DGS, 4DGaussians, in fixed combinations of N = 2, 3, or 4) is trained independently with its own original recipe and hyperparameters, initialized from COLMAP point clouds, so that every expert can reconstruct the whole scene alone. All expert parameters are then frozen.

Second, the router is trained. Each Gaussian from an expert is duplicated, and its color attribute is replaced by learnable per-Gaussian weights that encode a base weight, a viewing-direction term, and a time-dependent term. These 3D weights are projected onto the image plane by the differentiable Gaussian rasterizer (weight splatting), producing pixel-aligned embeddings that a lightweight MLP refines together with the pixel's viewing direction. A softmax over the refined values yields the gating weights used to blend the experts' rendered images, and the router is optimized with standard L1 and SSIM losses.

Two performance ideas are layered on top. Single-pass multi-expert rendering merges all experts' Gaussians into one batch, tags each with a one-hot expert identity, and separates expert outputs during alpha blending so projection and visibility are computed once. Gate-aware pruning computes each Gaussian's importance as the average magnitude of the gradient of the gating weight with respect to its per-Gaussian weights across training views, then progressively removes Gaussians below a threshold. Separately, a distillation scheme retrains each expert from scratch using a combination of ground-truth and MoE-rendered pseudo-labels, each weighted by the router's confidence, with a λ term balancing the two sources.

Why This Matters

The paper argues that the field's habit of selecting one dynamic reconstruction method per scene is a structural limitation, since different motion regimes (smooth canonical deformation, fast volumetric flow, globally smooth low-curvature trajectories, irregular interpolated motion) coexist inside real footage. MoE-GS reframes model choice as a learned, per-pixel, per-frame decision, and shows that combining heterogeneous priors raises ceiling performance. It also demonstrates a practical deployment path: distillation gives single-expert models most of the gain, and pruning plus single-pass rendering keeps multi-expert inference usable.

Real-world applications the work supports:

• Immersive and volumetric video for spatial computing and extended-reality content, where camera paths and motion complexity vary shot to shot.
• Free-viewpoint replay of live events and sports, where a single scene contains both fast motion and static backgrounds.
• Virtual production and film post-production, where temporally consistent novel views are needed under varied motion.
• Training data generation and perception for embodied agents or robots that must reason about dynamic environments.

Industry relevance: the authors are affiliated with Pusan National University and the Electronics and Telecommunications Research Institute (ETRI), and the work was supported by Korean national research programs and the Busan RISE program, indicating direct alignment with immersive media and telecommunications interests. The framework is explicitly designed to stay compatible with evolving Gaussian Splatting representations, so it can adopt new experts as they appear.

Future Directions

• Scaling to more experts: the paper shows distillation becomes more valuable at N = 4 and higher, but does not establish where direct MoE inference stops being practical.
• Canonical-space fusion: the authors note that the per-Gaussian responsibilities exposed by the router — described as liftable back into 3D in a responsibility-weighted manner (Appendix D) — provide a signal for future geometric unification or refinement, which the paper leaves as post-hoc and unexplored at scale.
• Broader benchmarking: the main results cover N3V and Technicolor, with monocular evaluation on HyperNeRF relegated to the appendix; generalization to other dynamic datasets, especially ones with different capture setups, remains open.
• Efficiency beyond pruning: the paper contrasts gate-aware pruning with "naive ratio-based pruning," which it says destabilizes optimization, but the boundary between stable and unstable pruning ratios across expert counts is not characterized.

Target Audience

Researchers and graduate students working on 3D Gaussian Splatting, dynamic/4D reconstruction, and novel view synthesis, particularly those interested in ensemble or routing-based methods for graphics and vision. It is also useful for engineering teams building real-time volumetric video or XR pipelines, who will care most about the FPS, memory, and distillation results, and for readers tracking how MoE ideas migrate from language models into 3D vision.

Authors’ abstract

Recent advances in dynamic scene reconstruction have significantly benefited from 3D Gaussian Splatting, yet existing methods show inconsistent performance across diverse scenes, indicating no single approach effectively handles all dynamic challenges. To overcome these limitations, we propose Mixture of Experts for Dynamic Gaussian Splatting (MoE-GS), a unified framework integrating multiple specialized experts via a novel Volume-aware Pixel Router. Unlike sparsity-oriented MoE architectures in large language models, MoE-GS is designed to improve dynamic novel view synthesis quality by combining heterogeneous deformation priors, rather than to reduce training or inference-time FLOPs. Our router adaptively blends expert outputs by projecting volumetric Gaussian-level weights into pixel space through differentiable weight splatting, ensuring spatially and temporally coherent results. Although MoE-GS improves rendering quality, the increased model capacity and reduced FPS are inherent to the MoE architecture. To mitigate this, we explore two complementary directions: (1) single-pass multi-expert rendering and gate-aware Gaussian pruning, which improve efficiency within the MoE framework, and (2) a distillation strategy that transfers MoE performance to individual experts, enabling lightweight deployment without architectural changes. To the best of our knowledge, MoE-GS is the first approach incorporating Mixture-of-Experts techniques into dynamic Gaussian splatting. Extensive experiments on the N3V and Technicolor datasets demonstrate that MoE-GS consistently outperforms state-of-the-art methods with improved efficiency. Video demonstrations are available at cvsp-lab.github.io/MoE-GS.