MOVA: Towards Scalable and Synchronized Video-Audio Generation

AI summary

Overview

• Research area: Multimodal generative modeling (computer vision / audio-visual generation), specifically joint text/image-to-video-and-audio synthesis.
• Technical level: Advanced. The paper assumes familiarity with diffusion transformers, flow matching, mixture-of-experts models, VAE latent spaces, rotary positional embeddings, and parallel training infrastructure.
• Scope: This paper introduces MOVA, an open-source 32B-parameter (18B active) mixture-of-experts model that generates synchronized video and audio from text and an optional first frame, along with the data pipeline, three-phase training recipe, and released weights, code, and fine-tuning support.

What This Paper Is About

Most video generation models ignore audio, and systems that do produce synchronized sound typically use cascaded pipelines (generate video, then synthesize audio from it), which accumulate errors and prevent the two modalities from interacting. Joint audio-video models such as Veo 3 and Sora 2 demonstrate that simultaneous generation is possible, but they remain closed-source, leaving the research community without a high-quality open alternative. MOVA's goal is to build an open model that jointly generates lip-synced speech, environment-aware sound effects, and content-aligned music, and to show that this capability scales with data and model size.

Key Contributions

1. MOVA, an asymmetric dual-tower video-audio generation model. It couples a pre-trained video tower with a pre-trained audio tower through a lightweight bidirectional Bridge module, using an aligned rotary positional encoding so that video and audio tokens share the same physical-time grid.
2. A fine-grained audio-video captioning pipeline. Three-stage curation (preprocessing, audio-visual quality assessment, audio-visual captioning) produces high-quality bimodal training data at scale using multiple open-source models.
3. A progressive three-phase joint training recipe with scaling evidence. Training moves from 360p diverse data, to 360p quality-filtered data, to 720p highest-quality data, and the authors report continuous improvements in both lip-synchronization and video-audio alignment metrics as training progresses.
4. Full open release. Model weights, training code, inference code, and fine-tuning code are released, with support for efficient inference, LoRA fine-tuning, and prompt enhancement.

Main Findings

• Model scale and configuration: MOVA uses a Mixture-of-Experts architecture with 32B total parameters, of which 18B are active during inference. It supports the IT2VA (Image-Text to Video-Audio) generation task.
• Architecture composition: The system pairs a Wan2.2 I2V A14B video backbone with a 1.3B audio diffusion transformer backbone, connected by a 2.6B bidirectional Bridge module. The Wan2.1 video VAE and a DAC-style audio VAE from HunyuanVideo-Foley are kept frozen throughout training.
• Audio tower benchmark results (AudioCaps): The 1.3B audio tower at 100 NFE reaches Inception Score 10.54, CLAP score 0.463, FD openl3 72.25, and KL passt 1.47. For comparison, TangoFLUX (516M, 50 NFE) scores 80.47 / 1.02 / 0.546 / 13.28 on FD openl3 / KL passt / CLAP / IS, and AudioLDM2 (346M, 200 NFE) scores 72.04 / 1.66 / 0.409 / 7.79.
• AudioBox perceptual results: MOVA reports the best Consistency (CU = 5.56) and Perceptual Quality (PQ = 6.20), with CE = 3.41 and PC = 3.04. It does not lead on CE or PC but the scores remain competitive against AudioLDM, AudioLDM2, Make-An-Audio 2, Tango 2, and TangoFLUX.
• Data retention: Starting from raw video at 100%, the pipeline retains 84.57% after Stage 1 (speech + non-speech), 58.75% for Stage 1 speech-only, and 26.39% after Stage 2 quality filtering. Only speech segments are used for training, accounting for 69.47% of all preprocessed segments.
• Clip standardization: Clips are normalized to 720p, 24 fps, and 8.05 seconds, which corresponds to 193 video frames (1 + 8 × 24).
• Phase 2 dataset composition: OCR filtering for videos without burned-in subtitles retains about 9.5M clips; lip-sync filtering (LSE-D ≤ 9.5 and LSE-C ≥ 4.5) retains about 2.5M clips; DOVER technical quality > 0.15 retains about 2.4M clips. The combined Phase 2 dataset contains 16.8M clips (about 37,600 hours).
• Training scale: Phase 1 uses roughly 61,500 hours of diverse data (15 days), Phase 2 uses roughly 37,600 hours (7 days), and Phase 3 uses roughly 11,000 hours of the 720p highest-quality subset (20 days). The complete run spans 42 days on 1024 GPUs, totaling approximately 43,000 GPU-days at roughly 35% MFU.
• Noise schedule asymmetry: Phase 1 uses shift_v = 5.0 and shift_a = 1.0 with text dropout p_drop_text = 0.5; Phase 2 aligns audio to the video schedule (shift_a = 5.0), lowers text dropout to 0.2, and adds LUFS normalization to mitigate CFG-induced loudness explosion.
• Heterogeneous learning rates: The Bridge is trained at η_br = 2 × 10⁻⁵ versus η_b = 1 × 10⁻⁵ for the backbone towers. The authors state that a uniform learning rate either destabilizes the towers or leaves the Bridge under-trained, and that a two-stage warm-start scheme (freeze towers, then fine-tune) reached an early performance plateau.
• Independent timesteps per modality: Video and audio timesteps are sampled independently from U(0,1) with modality-specific sigma shifts, allowing each modality its own noise schedule and allowing schedule changes at inference without retraining.
• Guidance behavior: Dual CFG with NFE = 3 gives the most flexible alignment-quality trade-off. Text-only CFG (NFE = 2) yields high semantic fidelity such as ImageBind scores but weaker temporal sync (higher DeSync), while text + modality CFG (NFE = 2) produces stronger synchronization, lower DeSync, and better lip-sync.
• Evaluation scope: The paper defines two benchmarks — Verse-Bench (600 image-text prompt pairs, with GPT-5 used to unify visual and audio descriptions) and a newly constructed benchmark with six categories (multi-speaker interaction, movie-style narratives, sports competitions, game livestreams, camera motion sequences, anime-style content) — and compares against LTX-2, Ovi, and a cascaded WAN2.1 + MMAudio pipeline at 720p. The subjective study uses 732 samples (600 from Verse-Bench, 132 from the new benchmark) with ELO initialized at 1000, K-factor 4, logistic scale 400, base 10, and 1000 bootstrap iterations.
• Reported outcome metrics are not in the provided content: The supplied paper text truncates in the Baselines section before presenting the Verse-Bench, human-preference, or synchronization results, so no comparison scores for MOVA against LTX-2, Ovi, or WAN2.1 + MMAudio are reported here.

Methodology in Plain English

Rather than building a video-audio model from scratch, the team reused strong existing single-modality models and taught them to talk to each other.

• Two towers, one bridge. A large pre-trained video generator and a smaller newly trained audio generator run side by side. A separate Bridge module adds cross-attention blocks in both directions, so video information flows into the audio tower and audio information flows into the video tower at each interaction layer.
• Making time line up. Video and audio latents sit on different temporal grids — audio tokens are much denser than video tokens. The authors scale video temporal indices by the ratio of audio frame rate to video frame rate, so both modalities use the same physical-time axis in rotary positional encoding and queries match the right keys.
• Training as noise removal. Both modalities are trained with flow matching: clean latents are interpolated with Gaussian noise, and the model learns to predict the velocity that reverses that interpolation. The loss combines a video term and an audio term with balancing weights.
• Building the data. Raw videos are standardized (cropped, centered, resized to 720p, padded to 9:16 or 16:9), segmented using voice activity detection and scene-change detection into 8.05-second clips, then scored for audio quality, video quality, and audio-visual synchronization and alignment. Clips that pass are captioned: a vision-language model describes the visuals, an omni-model transcribes speech and captions non-speech sound and music, and a large language model merges everything into a single coherent caption with a consistency check.
• Teaching in stages. The audio tower is first trained alone as a text-to-audio model. Then both towers plus the Bridge are optimized end-to-end at increasing fidelity: 360p on broad data, 360p on filtered data with a tightened audio noise schedule, then 720p on the best subset.
• Serving it. At inference, a vision-language model extracts a structured description of the input frame (style, cinematography, visual elements, any OCR text), a language model rewrites the user prompt in the style of the training data, and dual classifier-free guidance lets users separately control text adherence and cross-modal alignment strength.

Why This Matters

This work is one of the few open, fully released attempts at large-scale synchronized video-audio generation, an area where the strongest systems are proprietary. It provides not just weights but the data pipeline, training recipe, and efficiency techniques, which matters because the paper argues the field's main obstacles are data curation, modality fusion, and unverified scalability rather than any single architectural trick.

Real-world applications:

• Automated dubbing and localization: generating lip-synced speech in both English and Chinese for existing footage.
• Film, animation, and game content prototyping: producing multi-speaker dialogue, environment-aware sound effects, and content-aligned music in one pass.
• Advertising and short-form vertical video creation, since the model supports both 16:9 and 9:16 aspect ratios.
• Accessibility and archival: adding plausible synchronized audio to silent or degraded footage, and scene text generation as shown in the capability overview.

Industry relevance: The release includes efficient inference, LoRA fine-tuning, and prompt enhancement, which lowers the barrier for studios and startups to adapt the model to proprietary styles. The training stack is also ported to Ascend NPUs (34.1 s/step on 8 × Ascend 910A2 with CP=4 and DP-shard=2), signaling that the recipe is meant to be reproducible outside a single GPU vendor's ecosystem. The 35% MFU figure and the alternating high-noise/low-noise optimization scheme for the A14B MoE tower give practitioners concrete engineering guidance for large joint-modal training.

Future Directions

• Close the evaluation gap. The paper sets up objective and human-preference protocols but the results section is not included in the available text; full numbers against LTX-2, Ovi, and the cascaded WAN2.1 + MMAudio baseline are needed to judge the claimed synchronization gains.
• Extend beyond IT2VA. MOVA supports the Image-Text to Video-Audio task; whether the architecture extends to longer video, video-to-video, or purely text-driven generation at the same quality is left open.
• Push the scaling question further. The paper asks whether video-audio models sustain gains with larger data and model scale, and reports improvements across three phases — but the three phases also change resolution, filtering, and noise schedule at the same time, so separating the effects of scale from those of data quality and schedule alignment remains an open question.
• Simplify the training stack. The A14B MoE video tower required an alternating high-noise/low-noise optimization strategy because standard FSDP needs a consistent computation graph; more general parallelism schemes could remove this constraint, and the authors themselves flag that end-to-end joint optimization beat the two-stage warm-start that plateaued early.

Target Audience

This paper is most useful to multimodal generative modeling researchers and engineers working on diffusion transformers, joint audio-video synthesis, or large-scale distributed training. Practitioners who want an open, fine-tunable audio-visual generator with released weights and LoRA support will find the model and pipeline directly applicable. Readers focused on data engineering will benefit from the three-stage curation design, the retention ratios, and the specific filtering thresholds reported. It is not an introductory read: it presumes comfort with flow matching, cross-attention fusion, classifier-free guidance, and sequence-parallel training infrastructure.

Authors’ abstract

Audio is indispensable for real-world video, yet generation models have largely overlooked audio components. Current approaches to producing audio-visual content often rely on cascaded pipelines, which increase cost, accumulate errors, and degrade overall quality. While systems such as Veo 3 and Sora 2 emphasize the value of simultaneous generation, joint multimodal modeling introduces unique challenges in architecture, data, and training. Moreover, the closed-source nature of existing systems limits progress in the field. In this work, we introduce MOVA (MOSS Video and Audio), an open-source model capable of generating high-quality, synchronized audio-visual content, including realistic lip-synced speech, environment-aware sound effects, and content-aligned music. MOVA employs a Mixture-of-Experts (MoE) architecture, with a total of 32B parameters, of which 18B are active during inference. It supports IT2VA (Image-Text to Video-Audio) generation task. By releasing the model weights and code, we aim to advance research and foster a vibrant community of creators. The released codebase features comprehensive support for efficient inference, LoRA fine-tuning, and prompt enhancement.