Breaking the Uniformity Trap: Scaling Video Diffusion Model via SplitMoE

AI summary

Overview

Research area: Computer vision — sparse (Mixture-of-Experts) architectures for text-to-video diffusion transformers.

Technical level: Advanced. The paper assumes familiarity with diffusion transformers, Mixture-of-Experts routing, load-balancing losses, VAE feature spaces, and prototype-based representation learning.

Scope: The paper diagnoses why language-model-style MoE routing is mismatched to video tokens (the "uniformity trap") and proposes SplitMoE, a split-role expert architecture with prototype-guided routing, evaluated on VBench-2.0 and T2V-CompBench under matched activated-parameter budgets.

What This Paper Is About

Most Mixture-of-Experts designs were built for language models, where an auxiliary load-balancing loss pushes tokens to spread uniformly across experts. Video tokens behave very differently: they are densely correlated, spatially continuous, and semantically long-tailed, so forcing uniform expert usage scatters patches from the same object across unrelated experts. The paper calls this failure mode the "uniformity trap" and proposes SplitMoE, which splits the expert pool into semantic experts (for high-level structure) and generic experts (for residual appearance), guided by learnable prototypes rather than balancing constraints.

Key Contributions

1. SplitMoE, a split-role sparse architecture for video generation. The expert pool is partitioned into disjoint semantic and generic subsets, decoupling high-level semantic abstraction from low-level generic reconstruction, with the goal of mitigating spatiotemporal fragmentation and temporal flickering.

2. Prototype-Guided Semantic Routing. Learnable prototypes bridge reconstructive VAE features and DiT visual tokens, producing semantic-aware expert assignment that reduces expert homogenization and promotes meaningful specialization while preserving flexible residual modeling.

3. Pull-push prototype regularization and router alignment. A KL-divergence alignment loss matches router predictions to VAE-prototype-induced soft targets, while a pull term anchors prototypes to current and historical VAE features (via a circular token bank) and a push term separates prototypes to avoid redundancy.

4. An empirical study of Video-MoE routing dynamics. Under matched activated-parameter budgets, SplitMoE outperforms standard MoE and reports timestep analyses revealing an emergent coarse-to-fine routing pattern consistent with denoising.

Main Findings

• Standard video MoE shows weak semantic organization. Measured with semantic distinctiveness and intra-sample routing dominance, "Standard MoE has consistently low semantic distinctiveness and routing dominance across most semantic classes, indicating that tokens from the same region are often dispersed across experts." The paper argues that strict uniform allocation can amplify spatiotemporal fragmentation.

• SplitMoE quantifies above both the dense fine-tune and the standard MoE. On VBench-2, Full SplitMoE reaches Creativity 58.46%, Common Sense 64.89%, Controllability 45.72%, Human Fidelity 84.47%, and Physics 69.41%. Dense Wan2.2-FT scores 49.57%, 55.92%, 30.37%, 73.08%, and 54.66% respectively; Wan2.2-MoE scores 53.88%, 57.15%, 35.51%, 73.49%, and 63.32%.

• Compositional results on T2V-CompBench. Full SplitMoE reports Consistent Attribute 84.63%, Interaction 69.98%, and Numeracy 44.01%, versus 77.51%, 61.08%, and 36.71% for Dense Wan2.2-FT and 78.82%, 63.89%, and 38.55% for Wan2.2-MoE.

• Latest-systems comparison. Against externally trained models, the paper reports Wan2.2 at Creativ

Authors’ abstract

Mixture-of-Experts (MoE), popularized by large language models, is a promising paradigm for scaling visual generative models. However, conventional token-wise MoE routes tokens independently within a homogeneous expert pool and regularizes expert usage toward uniformity, making it poorly matched to video data that is spatiotemporally redundant and semantically long-tailed. We show that existing visual MoEs fall into a uniformity trap: semantically under-organized routing, compounded by uniform expert-usage regularization, scatters coherent patches across disparate experts, causing routing fragmentation and structural distortion. To address this, we propose SplitMoE, a split-role sparse architecture that breaks the shackles of uniformity. To accommodate the inherent semantic imbalance, we explicitly bifurcate the expert pool into semantic experts and generic experts, with semantic experts capturing high-level semantic abstraction and generic experts preserving residual visual information and flexible generative capacity. Leveraging prototype-guided routing and pull-push regularization, SplitMoE enables tokens to cluster naturally by semantic attributes rather than arbitrary balancing constraints. Extensive results show that under an equivalent activated-parameter budget, SplitMoE outperforms traditional load-balanced MoEs in convergence speed, routing coherence, and video generation quality across standard benchmarks. By revealing an emergent coarse-to-fine denoising logic, SplitMoE provides the community with a modality-aware scaling path, serving as a critical reference for building large-scale video world models.