Spatiotemporal-Untrammelled Mixture of Experts for Multi-Person Motion Prediction

AI summary

Overview

Research area: Computer vision / human motion prediction, specifically multi-person motion prediction (MPMP) using Mixture-of-Experts (MoE) and state-space models (Mamba).

Technical level: Intermediate — the paper assumes familiarity with Transformer attention, Mixture-of-Experts routing, and state-space sequence models, but its core ideas (sparse expert selection, linear-complexity sequence modeling) are explained at a conceptual level.

Scope: The paper introduces ST-MoE, a lightweight MoE framework that replaces spatiotemporal attention and positional encodings with four heterogeneous bidirectional Mamba experts, and evaluates it on four multi-person benchmark datasets.

What This Paper Is About

Multi-person motion prediction tries to forecast where several people's joints will be in the future, given a short history of their motion. Existing Transformer-based methods have two problems: they capture spatiotemporal structure through fixed positional encodings, which is inflexible, and their attention mechanisms scale quadratically, which is expensive. This paper proposes ST-MoE, which routes motion features through four specialized spatiotemporal Mamba experts that are selected adaptively, aiming to be both more flexible and much cheaper to train.

Key Contributions

1. First lightweight MoE for MPMP. The authors propose Spatiotemporal-Untrammelled Mixture of Experts (ST-MoE), described as the first framework to integrate spatiotemporal Mamba with dynamic expert routing for multi-person motion prediction, resolving what they call the efficiency-accuracy trade-off.
2. Four distinct spatiotemporal experts. The model introduces four heterogeneous bidirectional Mamba experts (labeled ST, TT, TS, SS in the ablations), each built from a different pairing/ordering of bidirectional spatial Mamba and bidirectional temporal Mamba, so different experts specialize in different spatial or temporal dependencies.
3. Efficiency and parameter economy via parameter sharing. All experts share the same bidirectional temporal and spatial Mamba parameters, which the authors say reduces model parameters while allowing each expert to differ through propagation order.
4. State-of-the-art accuracy with reduced cost. On four benchmarks the method reports better accuracy than competing methods while compressing model size by 41.38% and achieving a 3.6× training speedup relative to IAFormer.

Main Findings

• CMU-Mocap (UMPM), three-person: ST-MoE achieves average JPE of 95 mm and average APE of 65 mm, compared with IAFormer at 96 mm JPE / 66 mm APE and JRFormer at 99 mm JPE / 71 mm APE. The authors describe this as outperforming JRFormer by 4 mm JPE and 6 mm APE in average metrics.
• Mix1 (6 persons): ST-MoE reaches average JPE of 110 mm and average APE of 64 mm, which the paper reports as gains of 4 mm JPE and 1 mm APE over IAFormer (114 mm JPE, 65 mm APE).
• Mix2 (10 persons): ST-MoE reaches average JPE of 107 mm and average APE of 69 mm, described as gains of 17 mm JPE and 6 mm APE over JRFormer (124 mm JPE, 75 mm APE).
• CHI3D (real two-person capture): ST-MoE attains average JPE of 121 mm, which is 8 mm lower than IAFormer (129 mm) and 21 mm lower than TBIFormer (142 mm).
• Efficiency: Training is 3.6× faster than IAFormer and the model uses 41.38% fewer parameters. The paper attributes this to Mamba's linear time complexity versus attention's quadratic cost. All models were trained with batch size 96 for the comparison.
• Expert ablation (Table 3, CMU-Mocap UMPM): Adding any single expert type improves over the baseline (111.1 mm average JPE, 73.3 mm average APE). The TT expert alone reduces average JPE by 13.0 mm and average APE by 6.9 mm. Combining all four experts gives the best result at 95.0 mm JPE / 65.4 mm APE, outperforming any uniform expert configuration.
• Number of activated experts: Increasing k from 1 to 4 progressively improves both average JPE and APE, with optimal performance when all four experts are activated.
• Number of MoE layers: A single MoE layer performs best; the paper states that stacking more layers may lead to overfitting and makes training more difficult.
• Bidirectional scanning ablation (Table 4, CMU-Mocap UMPM): Bidirectional scanning gives an average JPE of 95.0 mm and APE of 65.4 mm, versus 99.3 mm / 67.5 mm for forward-only and 98.9 mm / 67.0 mm for backward-only scanning.
• t-SNE analysis: Visualizing 300 randomly selected samples from CHI3D and Mix2, each expert forms well-separated clusters, which the authors interpret as evidence that the four experts capture divergent spatiotemporal motion patterns.
• Gating behavior: Gating-weight visualizations on CMU-Mocap (UMPM) show TT/ST experts associated with nearly static motions (limited spatial variation) and SS/TS experts associated with spatially dynamic motions such as wide arm swings during running.
• Prediction visualization: In a case where a person moves leftward and abruptly stops, IAFormer is reported to miss the "dynamic-to-static" transition and drift, whereas ST-MoE captures the transition.

Methodology in Plain English

The pipeline starts by padding the observed sequence: the last observed frame is repeated (T − t) times and appended, so the network receives a fixed-length input covering both history and the future horizon it must fill in. The padded sequence is transformed with a Discrete Cosine Transform (DCT) and passed through a Multi-Pose Encoder (a 3-layer GCN) into a feature representation.

That feature goes two places at once. One path is a router: a one-layer MLP gating network that computes weights over the expert pool, keeping only the top-k entries (everything else is set to −∞ so softmax pushes them to approximately zero, producing sparse activation), then combining the selected experts' outputs by weighted sum. In this work the authors choose to activate all four experts, which their experiments support as the best setting.

The other path enters the expert pool. Each expert is a pairwise arrangement of two building blocks: a bidirectional spatial Mamba that scans along the pose/joint dimension, and a bidirectional temporal Mamba that scans along the time dimension. "Bidirectional" means each module scans the sequence forward and backward and adds the two results plus a residual connection, which the authors say compensates for the unidirectional modeling limitations of the original Mamba. The four experts (ST, TT, TS, SS) differ only in the order in which spatial and temporal processing are applied. Crucially, every expert shares the same temporal and spatial Mamba weights, so the parameter count does not grow with the number of experts.

After each Mamba block, Layer Normalization and a Feed Forward Network are applied with residual connections. The aggregated expert output from a MoE layer is summed with the original input feature via a residual connection, then passed to a Multi-Pose Decoder and an inverse DCT to produce the predicted future poses.

Training uses two losses: a spatial loss (a weighted mean squared position error over both the observed and future joints, with weight λ) and a temporal consistency loss borrowed from IAFormer that convolves predicted and ground-truth motion and takes the MSE between them, to reduce temporal jitter. The two are combined as L = α·L_s + β·L_t. Configurations reported: observed 50 frames (2s) predicting 25 frames (1s), pose dimension D = 45, batch size 96, Adam optimizer with initial learning rate 0.01 decayed by 0.1^(1/50) per epoch, α = 1, β = 1, λ = 0.1, trained on a single RTX 3090 GPU.

Evaluation uses two metrics: mean per Joint Position Error (JPE), which measures global joint position error including whole-body displacement, and Aligned Position Error (APE), which subtracts the root joint so that only pose-specific error is measured.

Why This Matters

Impact on research. The paper argues that the dominant paradigm — spatiotemporal attention with positional encodings — imposes what the authors call "trammelled" (constrained) spatiotemporal patterns and quadratic cost. ST-MoE shows that heterogeneous MoE experts backed by linear-complexity Mamba can match or beat that paradigm while being substantially smaller and faster to train, which suggests the MoE-plus-SSM combination is a viable direction for structured human motion modeling. The finding that each expert learns a distinct, separable spatiotemporal role also gives a mechanistic account of why the mixture helps.

Real-world applications (as listed in the paper's introduction):

• Human-robot interaction, where robots infer human intent from observed motion for more effective collaboration.
• Autonomous driving, where anticipating pedestrian and other agent motion supports safe planning.
• Surveillance systems, where understanding multi-person movement patterns matters.
• Any deployed setting involving multiple people at once, since the paper argues multi-person prediction has greater practical relevance than single-person prediction.

Industry relevance. The 41.38% parameter reduction and 3.6× training speedup target the practical bottleneck of deploying motion prediction models: attention-based systems are costly to train and to run. A model that shares weights across experts and uses linear-complexity sequence modeling is easier to fit on modest hardware (the reported setup uses a single RTX 3090 GPU), which lowers the barrier for robotics and autonomous-systems teams.

Future Directions

1. Stochastic motion prediction. The authors explicitly state their method is limited to deterministic motion and that future work will target stochastic multi-person motion prediction, exploring MoE for modeling diversified motion distributions.
2. Generalization to more people and more complex scenes. The Mix1 (6 persons) and Mix2 (10 persons) evaluations already probe scaling, but the paper's controlled datasets leave open how the approach behaves with much larger crowds or noisier real-world captures.
3. Why one MoE layer is optimal. The ablation shows single-layer MoE works best and more layers overfit or become hard to train; understanding and overcoming that limit is an open question for deepening the architecture.
4. Better use of routing sparsity. The authors note they activate all experts rather than a sparse top-k subset; whether genuinely sparse routing could cut cost further without losing accuracy is left unexplored.

Target Audience

Researchers and graduate students working on human motion prediction, motion synthesis, and sequence modeling, particularly those interested in Mixture-of-Experts architectures and Mamba/state-space models as Transformer alternatives. It is also relevant to practitioners in robotics, autonomous driving, and surveillance who need accurate multi-person motion forecasting under tight compute budgets, and to anyone studying how heterogeneous experts specialize on different aspects of structured spatiotemporal data.

Authors’ abstract

Comprehensively and flexibly capturing the complex spatio-temporal dependencies of human motion is critical for multi-person motion prediction. Existing methods grapple with two primary limitations: i) Inflexible spatiotemporal representation due to reliance on positional encodings for capturing spatiotemporal information. ii) High computational costs stemming from the quadratic time complexity of conventional attention mechanisms. To overcome these limitations, we propose the Spatiotemporal-Untrammelled Mixture of Experts (ST-MoE), which flexibly explores complex spatio-temporal dependencies in human motion and significantly reduces computational cost. To adaptively mine complex spatio-temporal patterns from human motion, our model incorporates four distinct types of spatiotemporal experts, each specializing in capturing different spatial or temporal dependencies. To reduce the potential computational overhead while integrating multiple experts, we introduce bidirectional spatiotemporal Mamba as experts, each sharing bidirectional temporal and spatial Mamba in distinct combinations to achieve model efficiency and parameter economy. Extensive experiments on four multi-person benchmark datasets demonstrate that our approach not only outperforms state-of-art in accuracy but also reduces model parameter by 41.38% and achieves a 3.6x speedup in training. The code is available at https://github.com/alanyz106/ST-MoE.