MCMoE: Completing Missing Modalities with Mixture of Experts for Incomplete Multimodal Action Quality Assessment

AI summary

Overview

Research area: Computer vision, specifically multimodal action quality assessment (AQA) with missing modalities. Technical level: Advanced — the paper assumes familiarity with transformers, cross-attention, mixture-of-experts, and representation learning. Scope: The paper proposes MCMoE, a single-stage framework that reconstructs missing modalities with an adaptive gated generator and refines them with a mixture of experts, evaluated on three public AQA benchmarks under both complete and incomplete modality settings.

What This Paper Is About

Multimodal action quality assessment scores how well an action is performed by combining cues such as RGB video, optical flow, and audio. In practice, one or more of these modalities is often unavailable at inference time (sensor failure, environment, privacy, or upstream algorithm failure), and most existing multimodal models simply break or degrade badly when that happens. The paper's goal is a framework that stays accurate whether all modalities are present or only a subset is, without paying the cost of heavyweight generative models or two-stage training.

Key Contributions

1. A missing completion framework with mixture of experts (MCMoE) for incomplete multimodal action quality assessment, which unifies unimodal and joint representation learning in single-stage training and reduces reliance on heavyweight generative architectures. The authors state this is the first work to explore incomplete multimodal action quality assessment.
2. An Adaptive Gated Modality Generator (AGMG) that selectively completes missing modalities from available ones using iterative cross-attention plus gating layers, along with a Shared Temporal Enhancement Module (STEM) and Cross-modal Fusion Module (CFM) to bridge semantic gaps between modalities and fuse cross-modal semantics.
3. Extensive experiments and ablations on three public benchmarks that reveal the complementarity between missing modality generation and mixture of experts, and show state-of-the-art performance in both complete and incomplete multimodal scenarios.
4. A favorable performance-efficiency trade-off: on FS1000 the method uses 4.90M parameters and 1.34 G FLOPs in a single-stage pipeline, versus 5.39M–22.63M parameters and 2.56 G–1191.39 G FLOPs for the compared methods.

Main Findings

• Incomplete-modality performance: Averaged over the six incomplete combinations, MCMoE improves Spearman correlation / MSE by 17.1%/38.0%, 11.2%/21.3%, and 11.9%/11.5% on the three datasets. It outperforms all baselines on nearly all metrics and incomplete configurations.
• Complete-modality performance: Under full modalities, MCMoE gains 2.23%/15.3% on average, and compared in the complete-modality figure it shows Spearman correlation / MSE gains of 3.0%/9.7%, 0.9%/10.1%, and 2.8%/15.4% on the three datasets, achieving best or second-best results across categories.
• Existing methods degrade sharply with missing modalities: State-of-the-art multimodal AQA methods drop notably, especially on MSE, attributed to disrupted cross-modal interactions; incomplete-learning baselines imported from other domains (action recognition, emotion recognition) lack tailored modeling for action semantics and assessment patterns.
• Efficiency on FS1000: MCMoE reports 4.90M parameters and 1.34 G FLOPs, with average 0.78/15.4 and full-modality 0.88/11.5 (Spearman/MSE), against baselines ranging from 5.39M to 22.63M parameters and up to 1191.39 G FLOPs; only some baselines are single-stage.
• AGMG drives the largest single jump: In the ablation on RG and Fis-V, adding AGMG to the STEM-enhanced baseline raises average results from 0.573/19.92 to 0.647/9.45 (RG) and from 0.637/39.72 to 0.678/21.36 (Fis-V).
• MoE adds further gains: Adding MoE on top of AGMG yields average gains of 3.9%/15.9%, attributed to adaptive fusion of unimodal and cross-modal knowledge.
• Every component and loss matters: Removing AGMG, MoE, or STEM, or removing any of the reconstruction, alignment, or diversity losses, causes clear performance drops; STEM is described as critical for temporal context and cross-modal semantics.
• Feature-space evidence: t-SNE of grade patterns on FS1000 shows compact clusters with clear inter-grade boundaries; omitting AGMG produces scattered features, while omitting MoE splits quality grades into two clusters. Generated and real features mix well in t-SNE across incomplete combinations, and AGMG generation quality is assessed with four metrics — L1 Distance, Cosine Similarity, KL Divergence, and Maximum Mean Discrepancy — including on the smallest dataset, RG.

Methodology in Plain English

The system takes three input streams — RGB video, optical flow, and audio — and uses frozen pre-trained extractors to turn them into features. A shared temporal enhancement module, built from stackable Transformer encoders with shared parameters, processes each modality separately to make their representations more comparable without directly mixing them.

During training, the model randomly masks out modalities to simulate the incomplete conditions it will face at inference. Missing inputs start as zero vectors and are filled in by the adaptive gated modality generator: the available modalities act as keys and values in cross-attention, while the zero-initialized missing features act as queries that are refined iteratively over several layers. Gating layers then decide how much of the newly generated feature to trust, which limits damage from imperfect generation.

Next, a mixture of experts handles the completed feature set. Each modality has its own lightweight expert (a two-layer MLP) that learns modality-specific knowledge, and a small soft router (also a two-layer MLP followed by softmax) weights the outputs of all experts to form both unimodal and joint representations for that modality. The fused features from all modalities go through a cross-modal fusion module and then a grade-based regression network with learnable grade prototypes and a three-layer Transformer decoder; the final score is a weighted combination of grade quantifications.

Training combines four losses: a reconstruction loss (MSE) to make generated features faithful, an alignment loss (KL divergence) between the unimodal expert output and the fused representation to encourage joint unimodal-and-cross-modal learning, a diversity loss (triplet loss) to separate grade patterns, and a task loss (MSE) to fit the quality score. All of this is optimized in a single stage rather than the two-stage pipelines used by prior state-of-the-art methods.

Why This Matters

Impact on research: The paper reframes incomplete multimodal AQA as a single-stage problem, arguing that modality completion and mixture-of-experts models are complementary: completion gives experts higher-confidence features instead of zero matrices, while the experts adaptively balance real and generated features so errors from imperfect generation do not propagate. This is presented as an alternative to heavy VAE, GAN, and diffusion generators and to costly two-stage training.

Real-world applications:

• Sports judging, where the benchmarks used are figure skating (Fis-V) and rhythmic gymnastics (RG), plus the fine-grained sports dataset FS1000.
• Rehabilitation monitoring, where pose and other modalities may drop out intermittently.
• Skill determination and coaching feedback, where audio or text modalities may be unavailable in the field.
• Any deployed multimodal sensing system where cameras, microphones, or upstream pipelines fail, causing modalities to be missing at inference time.

Industry relevance: The reported 4.90M parameters and 1.34 G FLOPs, measured on a single RTX 3090 with PyTorch 1.12.0, matter for real-time or edge deployment — the introduction explicitly criticizes heavyweight generators as "unsuitable for real-time scenes." A framework that tolerates arbitrary missing modality combinations also reduces the engineering cost of shipping one model across deployments with different sensor suites.

Future Directions

• Testing the framework's claimed readiness for arbitrary modality counts beyond the three studied (RGB, flow, audio), including text in action feedback and pose in rehabilitation, which the paper identifies as important in other domains.
• Moving beyond the fixed-missing protocol used in evaluation to settings where the missing modality changes over time or varies per sample.
• Further closing the gap between generated and true features — the generation-quality analysis (L1 Distance, Cosine Similarity, KL Divergence, Maximum Mean Discrepancy) suggests room to improve fidelity while keeping the low parameter and FLOP cost.
• Extending the approach to domains beyond AQA where incomplete multimodal learning is studied, such as action recognition, emotion recognition, and medical analysis, which the authors cite as the source of their comparison baselines.

Target Audience

Researchers and graduate students working on multimodal learning, missing-modality robustness, or action quality assessment; practitioners building sports analytics, rehabilitation, or skill-assessment systems where sensor streams cannot be guaranteed; and engineers interested in mixture-of-experts designs that balance accuracy against parameter and FLOP budgets.

Authors’ abstract

Multimodal Action Quality Assessment (AQA) has recently emerged as a promising paradigm. By leveraging complementary information across shared contextual cues, it enhances the discriminative evaluation of subtle intra-class variations in highly similar action sequences. However, partial modalities are frequently unavailable at the inference stage in reality. The absence of any modality often renders existing multimodal models inoperable. Furthermore, it triggers catastrophic performance degradation due to interruptions in cross-modal interactions. To address this issue, we propose a novel Missing Completion Framework with Mixture of Experts (MCMoE) that unifies unimodal and joint representation learning in single-stage training. Specifically, we propose an adaptive gated modality generator that dynamically fuses available information to reconstruct missing modalities. We then design modality experts to learn unimodal knowledge and dynamically mix the knowledge of all experts to extract cross-modal joint representations. With a mixture of experts, missing modalities are further refined and complemented. Finally, in the training phase, we mine the complete multimodal features and unimodal expert knowledge to guide modality generation and generation-based joint representation extraction. Extensive experiments demonstrate that our MCMoE achieves state-of-the-art results in both complete and incomplete multimodal learning on three public AQA benchmarks. Code is available at https://github.com/XuHuangbiao/MCMoE.