MoFu: Scale-Aware Modulation and Fourier Fusion for Multi-Subject Video Generation

AI summary

Overview

Research area: Computer Vision — controllable video generation, specifically multi-subject video generation conditioned on text prompts and multiple reference images.

Technical level: Advanced. The paper assumes familiarity with Diffusion Transformers (DiT), latent diffusion, flow-matching losses, FFT/IFFT frequency decomposition, and vision-language model embeddings.

Scope (one sentence): MoFu is a unified framework that combines an LLM-guided Scale-Aware Modulation module, a frequency-domain Fourier Fusion strategy, and a Scale-Permutation Stability Loss to generate multi-subject videos that preserve natural subject scale and remain invariant to the ordering of reference images, supported by a new training dataset (MoFu-1M) and benchmark (MoFu-Bench).

What This Paper Is About

Multi-subject video generation takes a text prompt plus several reference images and synthesizes a video in which each subject keeps its identity, appears at a believable size, and interacts plausibly. Current methods fail in two specific ways: scale inconsistency (references captured at different zoom levels cause subjects to appear unnaturally large or small) and permutation sensitivity (processing references one by one along the frame or channel dimension makes the output depend on input order, causing distortion, disappearance, or physically implausible motion). MoFu's goal is to solve both problems inside a single framework while also providing a benchmark that tests them explicitly.

Key Contributions

1. MoFu, a unified framework that simultaneously addresses scale inconsistency and permutation sensitivity in multi-subject video generation, rather than treating them as separate problems.

2. Scale-Aware Modulation (SMO), an LLM-guided module that extracts implicit scale relationships from the prompt and injects them into the DiT backbone through adaptive scaling, shifting, and gating parameters, so subjects keep natural sizes despite scale variation in the references.

3. A Fourier Fusion strategy that segments and encodes each reference image, transforms its features into the frequency domain with the Fast Fourier Transform, separates high- and low-frequency components with a radial frequency mask, aggregates them by direct summation (justified by the near-orthogonality of high-dimensional vectors), and reconstructs a permutation-invariant representation via Inverse FFT.

4. MoFu-1M and MoFu-Bench: a high-quality training dataset of 1M unique 81-frame clips and the first benchmark designed to explicitly assess both scale consistency and permutation invariance, containing 1,000 subject-text pairs with up to three reference images each.

A supporting contribution is the Scale-Permutation Stability Loss (SPSL), a weighted sum of a scale loss (an area-ratio-reweighted mean-squared-error term) and a permutation loss (an MSE computed in the frequency domain across permutations of the references).

Main Findings

• Best overall results on MoFu-Bench: MoFu achieves the top scores on Aesthetics (0.401), FaceSim (0.396), GmeScore (0.745), ScaleScore (0.585), and SubjectSim (0.755), outperforming Phantom, SkyReels-A2, VACE, and MAGREF.
• Motion is a partial exception: MoFu scores 0.221 on Motion, below SkyReels-A2 (0.233) and Phantom (0.229), which the authors describe as a "balanced" result that maintains temporal coherence without sacrificing fidelity.
• Baseline comparison details: Phantom reaches 0.355 Aesthetics, 0.375 FaceSim, 0.706 GmeScore, 0.229 Motion, 0.536 ScaleScore, 0.748 SubjectSim; SkyReels-A2 reaches 0.286, 0.341, 0.691, 0.233, 0.527, 0.737; VACE reaches 0.392, 0.247, 0.732, 0.214, 0.547, 0.692; MAGREF reaches 0.369, 0.362, 0.717, 0.207, 0.511, 0.731.
• I2V-style methods trade off: The paper notes that image-to-video-based methods such as MAGREF do well on FaceSim and SubjectSim because of strong reference conditioning, but their scale inconsistency limits overall quality.
• Ablation on SMO: Without SMO, relative scale becomes unrealistic — the example given is a sparrow appearing disproportionately large compared with an elephant. With SMO, realistic proportions are preserved.
• Ablation on Fourier Fusion: Without it, the model is sensitive to reference order and may drop subjects (the example given is a wooden box under permutation). With it, outputs stay complete and consistent regardless of permutation.
• No ablation on SPSL: The authors explicitly state that they omit ablations of the scale loss and permutation loss, because SMO and Fourier Fusion are supervised respectively by the scale and permutation signals from that loss, so removing the loss while training the modules independently "would render the comparison meaningless."
• Benchmark composition: Reference images span human (26%), animals (17%), clothing (10%), objects (30%), cartoons (4%), and others (13%); most prompts fall between 100 and 200 words.
• Training setup: AdamW with β1 = 0.9, β2 = 0.999, weight decay 0.01, learning rate initialized at 1×10⁻⁵ with cosine annealing and periodic restarts, run on 16 NVIDIA H800 GPUs for 7 days at 480P resolution with an 81-frame sequence length.
• Data pipeline scale: 15M clips after scene detection were filtered down to 2.5M high-quality clips, from which 1M unique clips formed MoFu-1M.

Methodology in Plain English

The authors start from a Diffusion Transformer backbone, which already supports "modulation" — a way of injecting external conditioning into a network by scaling and shifting normalized features using parameters predicted by small MLPs. They attach two new modules to this backbone.

For scale, they feed the text prompt into a frozen LLM and take its embedding, then pass that embedding through a lightweight Scale Control Adapter (an MLP) that predicts three sets of parameters: a scale factor, a shift factor, and a gating factor. These are applied to the DiT block features so that scale information derived from the prompt — for example, which object should be bigger than which — directly shapes what the model generates.

For permutation sensitivity, they avoid feeding reference images in sequence. Instead, each reference is segmented with Grounded-SAM, cropped and resized, and encoded by a 3×3 CNN into a feature map. Each feature map is converted to the frequency domain with FFT, split into high- and low-frequency parts by a radial binary mask, and the parts across all references are simply summed. Because high-dimensional vectors tend to be nearly orthogonal, summing them creates a combined representation that does not depend on the order in which the references were presented. An Inverse FFT turns this back into a spatial feature map, which is concatenated with the video features to condition generation.

During training, the Scale-Permutation Stability Loss shapes both modules. Its scale term weights the standard mean-squared-error objective using subject masks resized to the video resolution and weighted by each subject's relative area, so larger subjects are reconstructed more accurately. Its permutation term compares the Fourier Fusion output for different orderings of the references against a reference ordering, penalizing any change caused by reordering.

To build training data, the authors segmented raw videos with scene detection, filtered low-quality clips by aesthetic and motion metrics, captioned them with Qwen2.5-VL, checked text-video alignment with GmeScore, extracted entities with LLM parsing, localized them with Grounded-SAM, removed clips whose subjects were too large or too small, and applied extra face detection and filtering. The benchmark was built by applying subject-centric zoom operations to create scale inconsistencies and by randomly shuffling reference order.

Why This Matters

This work reframes two failure modes of multi-subject video generation — scale and ordering — as problems that can be handled with explicit architectural mechanisms plus a matching training objective, instead of relying on the model to infer them implicitly from prompts or image layouts. It also provides the first benchmark, per the authors, that measures scale consistency and permutation invariance together, which gives the field a way to compare methods on these axes rather than only on overall visual quality.

Real-world applications:

• Advertising and e-commerce video production, where a product and a model from separate reference photos must appear at believable relative sizes and in a consistent arrangement.
• Short-form content creation and storytelling, where creators supply several character or object images and expect them to appear together regardless of the order they upload them.
• Virtual try-on and fashion video, where clothing and human references come from different sources and at different zoom levels.
• Animation and stylized content, using cartoon references (which make up 4% of MoFu-Bench categories) alongside realistic ones.

Industry relevance: The paper comes from a collaboration spanning JD.com and several universities, and the emphasis on a 1M-clip training set, a 1,000-case benchmark, 16 H800 GPUs for 7 days, and inference-efficient conditioning suggests direct relevance to production-scale video generation systems where reference ordering and scale mismatch are common inputs from end users.

Future Directions

• Better fine-grained temporal modeling: The authors state that MoFu's emphasis on robust spatial reasoning "may limit its flexibility in modeling extremely fine-grained temporal interactions in highly dynamic scenes."
• Robustness to poor references: The framework benefits from high-quality reference inputs, and performance "could degrade when references are sparse or noisy" — improving this while retaining faithful reconstruction is an open problem.
• Domain-specific priors: MoFu uses a unified architecture for general multi-subject generation, which the authors note "may overlook domain-specific priors that could further enhance performance."
• Separating the loss terms: Since the paper deliberately omits ablations of the scale and permutation components of the Scale-Permutation Stability Loss, isolating and quantifying each term's individual contribution remains an open question.

Target Audience

Researchers and engineers working on controllable video generation, diffusion transformers, and multimodal conditioning will get the most from this paper, particularly those building systems that must combine multiple identity references. It is also useful for practitioners evaluating or constructing benchmarks for subject fidelity and spatial consistency, and for readers interested in applying frequency-domain aggregation to permutation-invariant conditioning more broadly. The paper is not beginner-friendly: familiarity with DiT modulation, flow-matching objectives, and FFT-based feature processing is assumed to follow the method section.

Authors’ abstract

Multi-subject video generation aims to synthesize videos from textual prompts and multiple reference images, ensuring that each subject preserves natural scale and visual fidelity. However, current methods face two challenges: scale inconsistency, where variations in subject size lead to unnatural generation, and permutation sensitivity, where the order of reference inputs causes subject distortion. In this paper, we propose MoFu, a unified framework that tackles both challenges. For scale inconsistency, we introduce Scale-Aware Modulation (SMO), an LLM-guided module that extracts implicit scale cues from the prompt and modulates features to ensure consistent subject sizes. To address permutation sensitivity, we present a simple yet effective Fourier Fusion strategy that processes the frequency information of reference features via the Fast Fourier Transform to produce a unified representation. Besides, we design a Scale-Permutation Stability Loss to jointly encourage scale-consistent and permutation-invariant generation. To further evaluate these challenges, we establish a dedicated benchmark with controlled variations in subject scale and reference permutation. Extensive experiments demonstrate that MoFu significantly outperforms existing methods in preserving natural scale, subject fidelity, and overall visual quality.