MUG-V 10B: High-efficiency Training Pipeline for Large Video Generation Models

AI summary

Overview

Research area: Computer vision and generative AI, specifically large-scale video generation with latent diffusion/flow-matching transformers, plus the systems engineering (data pipeline, parallelism, kernel fusion) needed to train them.

Technical level: Advanced. The paper assumes familiarity with diffusion transformers (DiT), variational autoencoders, flow matching, and distributed training concepts such as tensor, pipeline, data, and sequence parallelism.

Scope: A technical report describing an end-to-end, openly released training pipeline for a 10-billion-parameter text-to-video and image-to-video diffusion transformer, evaluated on VBench and on e-commerce-oriented human evaluations.

What This Paper Is About

Training large video generation models is expensive and unstable because of the need to align text with video, the very long token sequences involved, and complex spatiotemporal dependencies. The authors (LLM Team, Shopee Pte. Ltd) build a complete training framework that optimizes data processing, model architecture, training strategy, and infrastructure together. The result is MUG-V 10B, a diffusion transformer that matches recent state-of-the-art video generators overall and is reported to surpass leading open-source baselines in human evaluations on e-commerce video tasks.

Key Contributions

1. Scalable data processing pipeline. A filtering and extraction pipeline that pulls high-quality clips from large corpora and uses a fine-tuned vision-language model (VLM) to produce structured captions for all clips, with emphasis on throughput and stage-wise accuracy.
2. High-ratio VideoVAE compression. A VideoVAE trained to achieve 8×8×8 compression along time, height, and width. Combined with non-overlapping 2×2 patchification in the DiT, this yields approximately 2048× compression relative to pixel space, while reconstruction quality remains comparable to state-of-the-art VAEs at this ratio.
3. Training-stable 10-billion-parameter DiT backbone. A transformer block configuration that trains stably, plus a new image/frame conditioning scheme that improves cross-frame consistency and unifies text-to-video and image-to-video tasks in one architecture.
4. Multi-stage training strategy and efficient infrastructure. Small-model hyperparameter validation, curriculum-based pre-training, annealed SFT, and preference optimization, combined with a Megatron-Core-based system using data, tensor, pipeline, and sequence parallelism, hand-written Triton kernels, and no activation recomputation. On 500 Nvidia H100 GPUs it achieves near-linear scaling.
5. Full-stack open-sourcing. Model weights, Megatron-Core-based large-scale training code, and inference pipelines for video generation and enhancement are released. The authors state this is the first public release of large-scale video generation training code that uses Megatron-Core for high training efficiency and near-linear multi-node scaling.

Main Findings

• VideoVAE compression ratio: The encoder downsamples by 8×8×8 along temporal, height, and width axes, achieving 512× volumetric compression; combined with 2×2 patchification this gives approximately 2048× compression relative to pixel space. Ablations found increasing the latent channel count C improves reconstruction until diminishing returns, and C = 24 was chosen as the best trade-off between quality and storage.
• Minimal encoding principle: Enforcing that each latent token derives solely from its own 8-frame chunk removes the information-density imbalance caused by causal convolutions, and yields a flexible latent interface usable for arbitrary sequence lengths and first-, middle-, or last-frame conditioning. The decoder, by contrast, is allowed to consume windows of {1, 4, 8} contiguous latents.
• Parameter expansion: A 2B-parameter DiT with hidden size 1728 and depth fixed at 56 transformer blocks was trained first, then expanded with an equi-variant hidden-size expansion using e = 2, increasing total parameters roughly 4× to reach 10B. Hyperparameters tuned on the small model transferred to the 10B model, accelerating convergence and reducing experimentation cost.
• Three-stage pre-training curriculum: Stage 1 mixes image data with low-resolution (360p) video clips, annealing the image-to-video ratio until video dominates. Stage 2 keeps 360p but extends clip length from 2s to 5s. Stage 3 switches to 5s clips at 720p curated from around 12M high-quality videos. Stages 1 and 2 allow the model to see over ten times more samples than Stage 3.
• Post-training: Annealed supervised fine-tuning used roughly 0.3M manually selected high-quality clips with a gradually decaying learning rate. A post-hoc EMA approximation via exponentially decayed model ensembling outperformed standard online EMA in this setting. Preference optimization used human-annotated absolute labels with the KTO algorithm (to fix interpenetration and deformation errors) and pairwise better/worse labels with the DPO algorithm (to improve motion quality), with the original SFT objective retained as a regularizer.
• VBench I2V leaderboard results: MUG-V (10b) scored VTCM 23.17, VISC 98.82, VIBC 99.51, SC 95.73, BC 98.52, MS 98.90, DD 57.24, AQ 61.37, IQ 68.48, I2V 95.37, Quality 81.55, and Total 88.46. At submission time it ranked third on the VBench I2V leaderboard, behind Magi-1 and the commercial system PI.
• Comparison to the table's other systems: MAGI-1 (24b) has the highest total in the table at 89.28; Wan2.1 (14b) total 86.86; HunyuanVideo (13b) 86.82; CogVideoX (5b) 86.70; STIV (8.7b) 86.73; Step-Video (30b) 88.36; Dynamic-I2V (5b) 88.45. MUG-V's VIBC (99.51) and BC (98.52) are the highest in the table, while its AQ (61.37), IQ (68.48), and VTCM (23.17) are lower than several baselines.
• Human evaluation on e-commerce tasks: Against HunyuanVideo and Wan 2.1, inputs were randomly sampled from public model showroom images, each method's default prompt generator produced prompts, and 5-second clips were generated. Clips were pooled, randomly ordered, and judged in parallel by three independent annotators with consensus (≥2 of 3). Annotators first judged whether a clip was discernibly AI-generated, then product consistency for clips deemed realistic, then whether deployable clips were "high quality." The paper reports strong results on both pass rate and high-quality rate but states the detailed results are in Appendix B.2. The authors note that residual minor artifacts and geometric distortions still limit overall quality.
• Infrastructure results: The hybrid parallel scheme combines data parallelism, tensor parallelism, pipeline parallelism, and sequence parallelism, with activation recomputation disabled and an optimal 10B-scale configuration identified through benchmarking. A two-tier kernel fusion merges linear-layer bias addition, per-pixel scale-and-shift modulation, and residual accumulation into a single hand-written Triton kernel, and folds attention-score masking into a FlashAttention-2 softmax kernel, alongside LayerNorm plus QKV projection fusion and zero-padding removal.

Methodology in Plain English

The team follows the dominant video generation recipe: a VAE squeezes pixel videos into a compact latent space, and a diffusion transformer learns to generate in that latent space using flow matching objectives.

For data, raw videos pass a licensing and content gate, then are split into scenes using PySceneDetect together with a Color-Struct SVM method to catch gradual fades. Clips are filtered on four axes: sharpness (Laplacian variance must fall in [200, 2000]), aesthetic score (clips below 4.5 are discarded), motion amplitude estimated with RAFT optical flow (clips below 1 or above 20 are dropped), and a multimodal LLM filter fine-tuned on 24k labelled videos that removes heavy post-processing, speed-altered footage, and camera shake. Captions come from a Qwen2-VL-72B captioner fine-tuned on public plus internal data, then distilled to Qwen2-VL-7B for throughput. A large language model parses captions into entity tags used for stratified sampling and deduplication, and the top approximately 10% of clips by composite score form the post-training pool.

For the model, the VAE is initialized from a public image VAE and extended to video with hybrid convolutional stacks that alternate 2D spatial convolutions with 3D temporal convolutions. The DiT uses a transformer block close to autoregressive language models, with cross-attention inserted between self-attention and the feed-forward network. The authors keep full attention rather than spatio-temporal separated attention because the high compression ratio makes it affordable, and they use 3D RoPE for positional encoding along with global signal embeddings for timesteps and frame rate.

Training proceeds in stages: first validate hyperparameters on a compact model, then expand parameters, then run the curriculum from mixed image/360p data through 5s 360p clips to 5s 720p clips, then fine-tune on curated data, then apply preference optimization with human labels. For image conditioning, the given frame's latent is placed in a masked region with its diffusion timestep set to zero, so the rest of the sequence follows the standard noisy trajectory.

On the systems side, the authors shard activations across the tensor-parallel group, partition layers with pipeline parallelism over point-to-point communication, and add data parallelism to enlarge effective batch size, with an asynchronous I/O pipeline, prefetching, caching, and dynamic balanced sampling across ranks to keep every GPU busy.

Why This Matters

Impact on research: The paper argues that the full training stack, not just the model weights, is the scarce resource in large video generation. Releasing Megatron-Core-based training code with near-linear scaling on 500 Nvidia H100 GPUs lowers the barrier for other groups to reproduce or extend billion-parameter video models. It also documents practical tricks that reduce trial-and-error compute: small-model hyperparameter transfer, output-preserving parameter expansion, minimal encoding in the VAE, and using human preference labels where reward models are considered too weak.

Real-world applications (as described by the authors):

• E-commerce product videos such as try-on showcases, still-life displays, and functional demonstrations.
• Advertising assets, where the authors note video generation offers gains in creativity and cost efficiency.
• Film, gaming, and advertising more broadly, which the paper lists as established domains for the technology.
• Video generation and enhancement pipelines, which are part of the released inference code.

Industry relevance: The work comes from an e-commerce company and explicitly targets retail requirements: generative correctness (semantically accurate content and physically plausible motion), content consistency (color, material, texture unchanged from the input image), and visual appeal. The authors note existing automatic metrics overlook fine-grained defects such as altered fabric textures or incorrect hand poses, which are exactly the failures that matter for product fidelity, which is why they add human evaluation.

Future Directions

• Closing the aesthetic and imaging quality gap. MUG-V's AQ (61.37) and IQ (68.48) trail several baselines in the comparison table, and the authors state that residual minor artifacts and geometric distortions still limit overall quality and leave substantial headroom in e-commerce applications.
• Improving camera motion controllability. VTCM of 23.17 is well below several baselines in the table, including CogVideoX (67.68), MAGI-1 (50.85), and Step-Video (49.23).
• Scaling preference optimization for video. The authors note that preference-based reinforcement learning for video remains challenging due to limited capacity of current video evaluation (reward) models and the multiplicity of optimization axes such as appearance, motion, and temporal coherence, which is why they relied on human annotations.
• Extending the curriculum and architecture to longer, higher-resolution generation. The paper describes a progression from 360p, 2s clips up to 720p, 5s clips, and the released pipeline provides a starting point for pushing these limits further.

Target Audience

Machine learning engineers and researchers working on large-scale generative models, particularly those building or fine-tuning diffusion transformer video generators who need training efficiency and distributed systems guidance. It is also relevant to applied teams in e-commerce, advertising, and media who evaluate whether generated video is deployable, and to infrastructure engineers interested in Megatron-Core parallelism, Triton kernel fusion, and near-linear multi-node scaling for long-sequence models. The paper is dense with systems and architecture specifics, so it is best suited to readers with prior exposure to diffusion models and distributed training.

Authors’ abstract

In recent years, large-scale generative models for visual content (\textit{e.g.,} images, videos, and 3D objects/scenes) have made remarkable progress. However, training large-scale video generation models remains particularly challenging and resource-intensive due to cross-modal text-video alignment, the long sequences involved, and the complex spatiotemporal dependencies. To address these challenges, we present a training framework that optimizes four pillars: (i) data processing, (ii) model architecture, (iii) training strategy, and (iv) infrastructure for large-scale video generation models. These optimizations delivered significant efficiency gains and performance improvements across all stages of data preprocessing, video compression, parameter scaling, curriculum-based pretraining, and alignment-focused post-training. Our resulting model, MUG-V 10B, matches recent state-of-the-art video generators overall and, on e-commerce-oriented video generation tasks, surpasses leading open-source baselines in human evaluations. More importantly, we open-source the complete stack, including model weights, Megatron-Core-based large-scale training code, and inference pipelines for video generation and enhancement. To our knowledge, this is the first public release of large-scale video generation training code that exploits Megatron-Core to achieve high training efficiency and near-linear multi-node scaling, details are available in https://github.com/Shopee-MUG/MUG-V.