UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactions

AI summary

Overview

• Research area: Human-centric joint audio and video generation (multi-modal generative modeling, computer vision plus speech/audio).
• Technical level: Advanced. The paper assumes familiarity with Diffusion Transformers (DiTs), flow matching, cross-attention, latent VAEs, and Classifier-Free Guidance.
• Scope: UniAVGen is a single unified framework that generates synchronized human speech audio and talking-head video together, using two parallel DiT branches joined by an asymmetric cross-modal interaction mechanism.

What This Paper Is About

Most open-source audio-video generation systems use decoupled, two-stage pipelines: either they generate a silent video and then dub audio onto it, or they generate audio first and use it to drive video. This separation prevents the two modalities from influencing each other during generation, which hurts lip synchronization, semantic consistency, and emotional alignment. UniAVGen instead synthesizes audio and video simultaneously inside one model, and adds targeted mechanisms to make the two modalities interact precisely where it matters—around the human face.

Key Contributions

1. UniAVGen framework: A unified audio-video generation framework built on a dual-branch joint synthesis architecture (two parallel DiTs) with an Asymmetric Cross-Modal Interaction mechanism and modality-specific aligner designs.
2. Face-Aware Modulation (FAM): A module that dynamically predicts a soft facial-region mask with decaying supervision signals, and uses it to constrain where cross-modal interaction is applied, making cross-modal learning more efficient and better aligned.
3. Modality-Aware Classifier-Free Guidance (MA-CFG): An inference-time strategy that repurposes classifier-free guidance to explicitly amplify cross-modal correlation signals, increasing emotional intensity in audio and motion dynamics in video.
4. Multi-task unification: A single model that handles joint audio-video generation, joint generation with reference audio, joint audio-video continuation, video-to-audio dubbing, and audio-driven video synthesis, with a multi-stage training strategy where joint multi-task training improves joint generation.

Main Findings

• Fewer training samples, competitive or better results: The abstract states UniAVGen achieves overall advantages in audio-video synchronization, timbre consistency, and emotion consistency with 1.3M training samples versus 30.1M. Table 1 lists UniAVGen at 1.3M samples and Ovi at 30.7M samples.
• Audio quality: UniAVGen records PQ 7.00 and CU 6.62, versus Ovi at PQ 6.03 / CU 6.01, UniVerse-1 at 4.56 / 4.29, and JavisDiT at 5.21 / 3.93. The two-stage baselines OmniAvatar and Wan-S2V both report PQ 8.15 and CU 7.41.
• Word Error Rate: UniAVGen reports WER 0.151, which the paper says outperforms F5-TTS and beats Ovi (0.216) and UniVerse-1 (0.296). JavisDiT reports 0.986.
• Video quality: UniAVGen achieves the highest Dynamic Degree (0.410) and Imaging Quality (0.779) among the compared joint-generation methods, with Subject Consistency at 0.973. The two-stage baselines show DD of 0.000 (OmniAvatar) and 0.130 (Wan-S2V), which the paper attributes to their inability to generate actions congruent with audio-driven emotions.
• Audio-video consistency: UniAVGen reports TC 0.832 and EC 0.573, above Ovi (0.828, 0.558) and well above UniVerse-1 (0.573, 0.300). Lip synchronization is 5.95, described as on par with leading methods (Ovi 6.48, Wan-S2V 6.35, OmniAvatar 6.34).
• Interaction design ablation (Table 2): Replacing symmetric designs with the proposed asymmetric interaction improves all three consistency metrics. Results at 10k steps of Stage 2: Symmetric Global Interaction (SGI/SGI) gives LS 3.46, TC 0.667, EC 0.459; Symmetric Time-Aligned (STI/STI) gives 3.73, 0.685, 0.472; the full asymmetric (ATI/ATI) gives 4.09, 0.725, 0.504.
• Face-Aware Modulation ablation (Table 3): Without FAM, LS 3.89, TC 0.705, EC 0.489. Unsupervised FAM gives 3.92, 0.701, 0.492. FAM with fixed weight gives 4.11, 0.719, 0.497. FAM with decaying weight gives 4.09, 0.725, 0.504 — the paper concludes supervised masks help convergence and decaying weights best improve timbre and emotion consistency.
• Training strategy comparison (Figure 7): Joint generation only (JGO) has a lower performance ceiling than joint generation followed by multi-task learning (JFML); multi-task training throughout (MTO) converges more slowly. JFML achieves the best overall performance on the LC metric.
• Qualitative generalization: On an out-of-distribution anime reference image, the paper reports Ovi fails to produce aligned lip movements and motions, while UniVerse-1 stays static with noisy audio; UniAVGen produces coherent, aligned output.
• User study (Table 4, 34 participants): UniAVGen was preferred with AQ 68.90%, VQ 62.60%, AVC 74.30%, versus Ovi at 28.75%, 37.40%, 25.70% and UniVerse-1 at 2.35%, 0.00%, 0.00%.

Methodology in Plain English

UniAVGen runs two generation networks side by side — one for video and one for audio — with matching internal structure so their internal representations line up. The video branch is initialized from Wan 2.2-5B, and the audio branch uses the architectural template of Wan 2.1-1.3B, which the paper notes differs only in number of channels. Video is processed at 16 frames per second through a pre-trained VAE; audio is sampled at 24,000 Hz and converted to Mel spectrograms. Both branches are trained with flow matching. The video caption is encoded with umT5, and the speech content is encoded with ConvNeXt blocks.

The distinctive piece is how the two branches talk to each other. The A2V aligner gives each video frame a contextualized audio window (audio tokens from neighboring frames, with the context window size set to 1/2, i.e. about 2 audio segments), so a frame's visual articulation can attend to surrounding phonemes. The V2A aligner goes the other direction and uses temporal neighbor interpolation — a weighted average of two temporally adjacent video latents — so each audio token sees a smooth, time-aware visual signal rather than a single coarse frame. Output projections are zero-initialized so each modality's generative ability is not damaged early in training.

FAM adds a lightweight head inside each interaction layer that predicts a soft face mask, supervised by a ground-truth face mask whose loss weight starts at 0.1 and decays linearly to 0. During A2V, the mask gates which video regions receive audio information; during V2A, it highlights the video features being sent to the audio branch. At inference, MA-CFG runs one forward pass with both cross-modal interactions nullified to get a shared unconditional baseline, then guides both modalities with separate strengths (s_v = 3 for video, s_a = 2 for audio), using a CFG interval of [0.5, 1] and 50 Euler ODE steps with a Vocos vocoder.

Training proceeds in three stages: Stage 1 trains only the audio branch on the English subset of Emilia (batch size 256, learning rate 2e-5, AdamW with β1 = 0.9, β2 = 0.999, ε = 1e-8, 160k steps); Stage 2 performs end-to-end joint training on an internally collected real human audio-video dataset (batch size 32, learning rate 5e-6, 30k steps); Stage 3 adds multi-task learning at a 4:1:1:2:2 ratio across the five tasks for 10k steps.

Why This Matters

• Research impact: The paper argues that decoupled two-stage pipelines are inherently limited because the modalities cannot inform each other, and it shows that a carefully designed asymmetric interaction beats both global and symmetric time-aligned designs on consistency metrics. It also provides evidence that joint multi-task training can improve joint generation itself, which is a useful signal for how multi-modal generation models should be trained.
• Real-world applications:
  • Talking-head avatars and digital humans for virtual meetings, customer service, and assistants.
  • Automated dubbing and video localization, where video-to-audio dubbing generates emotion- and expression-aligned speech for existing footage.
  • Content creation for social media, advertising, and entertainment, producing a speaking character from a single reference image and a text prompt.
  • Personalized audio-video generation with a custom reference audio to control the speaker's timbre.
• Industry relevance: The work comes from a collaboration spanning Nanjing University, Tencent Hunyuan, Shanghai Jiao Tong University, Renmin University of China, Tsinghua University, and Shanghai AI Lab, and it positions itself against commercial systems such as Veo3, Sora2, and Wan2.5. Its claim of competitive results from 1.3M training samples is directly relevant to training cost and data efficiency in production settings.

Future Directions

• Extending beyond human-centric generation: The framework is explicitly tailored to human-centric audio-video generation; whether the asymmetric interaction and face-aware modulation transfer to general scenes, ambient sound, and non-human subjects is not reported.
• Broadening evaluation of consistency metrics: The paper states that no open-source methodology currently exists to quantify timbre and emotion consistency, so it relies on Gemini-2.5-Pro with a custom system prompt averaging three independent evaluations. Developing objective, reproducible cross-modal alignment metrics is an open problem the paper surfaces.
• Scaling and data strategy: With only 1.3M training samples used, the relationship between data scale, the three-stage training recipe, and final quality remains an open question — particularly whether the multi-task ratio of 4:1:1:2:2 is optimal.
• Refining the relaxation schedule: The face mask supervision weight decays linearly from 0.1 to 0, and the paper notes decaying weights "outperform fixed weights" but reports LS slightly lower (4.09 vs 4.11) with decaying weights. The design of the decay schedule and its interaction with the mask's gradual expansion to body regions is left for further exploration.

Target Audience

This paper is most useful for researchers and engineers working on multi-modal generative models, particularly those building joint audio-video generation, talking-head synthesis, or diffusion-transformer systems. It will also interest practitioners in digital avatars, dubbing, and video localization who need synchronized speech and facial motion, and readers comparing end-to-end joint generation against two-stage pipelines. A working knowledge of diffusion models, cross-attention, and classifier-free guidance is assumed.

Authors’ abstract

Due to the lack of effective cross-modal modeling, existing open-source audio-video generation methods often exhibit compromised lip synchronization and insufficient semantic consistency. To mitigate these drawbacks, we propose UniAVGen, a unified framework for joint audio and video generation. UniAVGen is anchored in a dual-branch joint synthesis architecture, incorporating two parallel Diffusion Transformers (DiTs) to build a cohesive cross-modal latent space. At its heart lies an Asymmetric Cross-Modal Interaction mechanism, which enables bidirectional, temporally aligned cross-attention, thus ensuring precise spatiotemporal synchronization and semantic consistency. Furthermore, this cross-modal interaction is augmented by a Face-Aware Modulation module, which dynamically prioritizes salient regions in the interaction process. To enhance generative fidelity during inference, we additionally introduce Modality-Aware Classifier-Free Guidance, a novel strategy that explicitly amplifies cross-modal correlation signals. Notably, UniAVGen's robust joint synthesis design enables seamless unification of pivotal audio-video tasks within a single model, such as joint audio-video generation and continuation, video-to-audio dubbing, and audio-driven video synthesis. Comprehensive experiments validate that, with far fewer training samples (1.3M vs. 30.1M), UniAVGen delivers overall advantages in audio-video synchronization, timbre consistency, and emotion consistency.