InterMoE: Individual-Specific 3D Human Interaction Generation via Dynamic Temporal-Selective MoE

AI summary

Overview

• Research area: Computer vision and generative motion synthesis — specifically text-conditioned generation of 3D human interactions (two people moving together), with a methodological contribution to Mixture-of-Experts (MoE) routing inside diffusion models.
• Technical level: Advanced. The paper assumes familiarity with diffusion models, transformer attention, Mixture-of-Experts routing, and motion capture representations (joint rotations, 6D rotations, SMPL-X/AMASS skeletons).
• Scope: A single paper proposing InterMoE, a diffusion-based framework whose central novelty is a Dynamic Temporal-Selective MoE with a text-and-motion "Synergistic Router," evaluated on two interaction datasets plus a generalization test on single-person motion.

What This Paper Is About

Text-to-motion systems can generate a single person walking, dancing, or punching from a description. Generating two people interacting is harder: the model must keep each person's identity distinct (who is attacking, who is retreating) while also matching the text and making the bodies physically coordinate. Existing methods either fuse both people's features and then squeeze them through ordinary feed-forward layers — which washes out individual traits and produces "homogenized" motion — or concatenate both people's features and jointly generate them, which causes identity confusion where characters swap roles or positions. InterMoE attacks this by routing motion features to specialized expert sub-networks, guided by both the text and the motion itself, so that different experts handle different motion patterns and identities.

Key Contributions

1. InterMoE framework: A diffusion-based Mixture-of-Experts architecture for text-driven 3D human interaction generation that improves individual-specific characteristics, semantic fidelity, and overall motion quality simultaneously.
2. Synergistic Router: A routing mechanism that fuses two sources of routing signal — high-level text semantics and low-level kinematic features — into a single set of expert-selection logits, using a batch-level temporal feature pool rather than instance-level routing so the router can perceive noise-level variation across samples.
3. Dynamic Temporal Selection: A replacement for both Token-Choice (each token picks a fixed number of experts) and Expert-Choice (each expert picks a fixed number of top-K tokens) routing. Each expert carries a learnable bias that determines how many tokens it processes, and that capacity adapts during training, allowing experts to home in on the most salient temporal features.
4. Empirical validation plus generalization: State-of-the-art results on InterHuman and InterX interaction benchmarks, detailed ablations isolating each design choice, and evidence that the MoE paradigm transfers to single-person motion generation when inserted into existing models.

Main Findings

• State-of-the-art on both interaction benchmarks: InterMoE achieves the best FID on InterHuman (4.677) and InterX (0.297), the best R-Precision top-1/2/3 on both datasets (e.g., 0.512 / 0.671 / 0.746 on InterHuman), and the best MM-Dist (3.762 on InterHuman, 3.605 on InterX).
• Reported FID reductions: The paper claims a 9% FID reduction on InterHuman and 22% on InterX relative to prior methods. The InterX figure is consistent with the table (0.385 for TIMotion versus 0.297 for InterMoE).
• Text fidelity is prioritized over raw diversity: MultiModality is slightly below some baselines, but the authors argue the combination of high R-Precision and low MM-Dist shows the model favors accurate text adherence over generating wide variation.
• Both routing components matter independently: Ablations show that removing the parallel motion/text routers, disabling batch-level routing (reverting to instance level), disabling dynamic selection (fixed top-K), or removing temporal selectivity each causes FID degradation. Instance-level routing causes a notably severe drop, which the authors use to argue that global batch information is essential.
• The proposed routing beats standard MoE paradigms: Against Token-Choice (FID 5.095) and Expert-Choice (FID 8.699), InterMoE's 4.677 is a clear margin. Expert-Choice performs worst, which the authors attribute to its inability to exploit diffusion-process context.
• Eight experts is the sweet spot: A dense baseline (no MoE) gives FID 5.918; 4 experts gives 5.114; 8 experts gives 4.677; 16 experts degrades slightly to 4.970 — suggesting redundancy or undertraining at larger sizes.
• Capacity hyperparameter C_exp = 1 is optimal: With C_exp = 1 the model achieves its best FID (4.677) and MM-Dist; C_exp = 2 marginally improves R-Precision but worsens FID; C_exp = 0.8 is too sparse and also degrades.
• Qualitative gains center on identity and fine-grained semantics: In fencing prompts, competitors like TIMotion confuse who is attacking and who is retreating; in tug-of-war, only InterMoE renders the hand-gripping-rope posture and backward lean; in a 10-second taekwondo prompt with a circular movement constraint, competitors generate stationary sparring while InterMoE both spars and moves in a circle.
• The MoE paradigm generalizes to single-person motion: Injecting it into MDM, MLD, and SALAD on HumanML3D improves all three across R-Precision, FID, and MM-Dist (e.g., SALAD FID improves from 0.076 to 0.069).

Methodology in Plain English

The pipeline has three parts.

1. Compressing motion. A Causal-Skeletal VAE encodes each person's motion. Skeletal convolutions capture how joints relate within a single frame; causal convolutions capture how poses relate across frames without "looking into the future." A pooling layer squeezes this into a compact latent, which is cheap and efficient to work with.

2. Two cooperating denoisers. The denoising network uses two shared-weight denoisers, one dedicated to each person. Each transformer block contains self-attention (a person's own temporal structure), cross-attention (conditioning on the partner's motion), and the MoE block. Denoising timestep and text condition enter through adaptive layer normalization.

3. The MoE block, which is the real novelty. Instead of every token picking a fixed number of experts, or every expert picking a fixed number of tokens, each expert gets a learnable bias. For each candidate feature, the router computes a score; a sigmoid of that score plus the bias determines whether the expert processes the feature, and a softmax supplies the blending weight. If an expert is selecting more features than its target, the bias shifts to make it pickier, and vice versa — a simple sign-based update with a small step size that converges so average selection matches the target. Two routing signals are combined: a motion router reading kinematic features, and a text router reading the semantic description, fused as a weighted sum with equal weight. Crucially, routing is done over a batch-level pool that flattens all samples together, so experts see diverse motion and can adapt to different noise levels across the diffusion process.

This matters because in a two-person interaction, not every moment is equally important. The instant of a strike, a hand grip, or a turn carries more information than a steady transition. Letting each expert dynamically decide how much and which temporal material to handle lets the model allocate capacity where it counts, rather than treating every moment the same.

Why This Matters

Impact on research: The paper reframes a quality problem (lifeless, homogenized two-person motion) as a routing problem (assigning specialized sub-networks to distinct motion patterns), and demonstrates that a well-designed MoE can beat both standard MoE routing paradigms inside diffusion models. The generalization experiment to single-person motion suggests the routing mechanism is a transferable component, not a dataset-specific trick.

Real-world applications:

• Animation and game development: Fast generation of two-character choreographed sequences (fights, dances, handoffs) from a text prompt, cutting manual keyframing.
• Virtual and augmented reality: Populating shared spaces with believable avatars that respond to one another, not just to a user.
• Robotics and human-robot collaboration: Generating physically plausible paired motions helps robots anticipate and mirror human partners.
• Sports and physical training: Text-described drills such as sparring or tug-of-war could be synthesized for coaching or simulation.

Industry relevance: The strongest pull is from entertainment (film, games, previsualization) and from embodied AI, where paired motion data is expensive to capture but paired behavior is essential. A component that drops into existing diffusion motion models as a drop-in improvement, as shown on MDM/MLD/SALAD, is attractive to teams that already have a motion stack.

Future Directions

• Scaling beyond two people: The architecture assumes exactly two cooperative denoisers. Extending to group interactions (crowds, team sports, families) requires rethinking how cross-attention and routing scale.
• Incorporating objects and scenes: Tug-of-war and paper-handoff prompts imply physical objects the model does not explicitly represent. Explicit object and scene conditioning is a natural next step.
• Longer sequences with sustained constraints: The 10-second circular-pattern taekwondo example hints at long-horizon spatial constraints, but scaling to minutes without drift remains open.
• Rethinking expert scaling and efficiency: Performance degrades at 16 experts, which suggests either redundancy or insufficient training. Understanding why, and whether better initialization or load-balancing removes the ceiling, would strengthen the paradigm.
• Measuring identity preservation directly: The paper argues qualitatively that identity confusion is fixed, but the quantitative metrics (FID, R-Precision, MM-Dist, Diversity, MultiModality) do not directly measure whether the correct person performs the described action. A dedicated identity-fidelity metric would sharpen comparisons.

Target Audience

Researchers and graduate students working on text-to-motion and human motion synthesis, particularly those interested in interaction generation. Also relevant to practitioners of diffusion models and Mixture-of-Experts architectures who want to see MoE applied outside language and image domains, and to graphics or game engineers evaluating whether recent generative motion methods are ready for production pipelines. Readers need a working knowledge of transformer attention and diffusion sampling to follow the routing mathematics, though the qualitative results and ablation logic are accessible with less background.

Authors’ abstract

Generating high-quality human interactions holds significant value for applications like virtual reality and robotics. However, existing methods often fail to preserve unique individual characteristics or fully adhere to textual descriptions. To address these challenges, we introduce InterMoE, a novel framework built on a Dynamic Temporal-Selective Mixture of Experts. The core of InterMoE is a routing mechanism that synergistically uses both high-level text semantics and low-level motion context to dispatch temporal motion features to specialized experts. This allows experts to dynamically determine the selection capacity and focus on critical temporal features, thereby preserving specific individual characteristic identities while ensuring high semantic fidelity. Extensive experiments show that InterMoE achieves state-of-the-art performance in individual-specific high-fidelity 3D human interaction generation, reducing FID scores by 9% on the InterHuman dataset and 22% on InterX.