UniAVC: Unified Audio-Video Creation via Progressive Training and Asymmetric Scheduling

AI summary

Overview

Research area: Generative audio-video modeling — specifically multimodal diffusion/rectified-flow transformers for joint audio-video generation and instruction-guided editing.

Technical level: Advanced. The paper assumes familiarity with rectified flow, latent diffusion, transformer backbones, classifier-free guidance, and noise schedules.

Scope: A single-weight model that unifies text-conditioned joint audio-video generation, bidirectional cross-modal generation, and instruction-guided editing of either or both modalities, trained by a two-stage progressive recipe with per-modality ("asymmetric") noise schedules.

What This Paper Is About

Today's audio-video tools are separate specialists: joint generators, cross-modal generators (audio→video, video→audio), and instruction-guided editors each ship their own weights and data. Chaining them together — generate a clip, then edit it — breaks the audio-visual synchrony the generator worked to establish, because the editor is unaware of the soundtrack. The obvious fix, one model doing everything, is blocked by two problems: paired audio-video data (especially editing pairs) is scarce, and nearly all existing systems corrupt audio and video with a single shared noise schedule even though video needs a long high-noise phase and audio settles much earlier. UniAVC's goal is to build one model that spans all these creation modes without per-task adapters, and to exploit rather than ignore the audio/video information asymmetry.

Key Contributions

1. A unified generation-and-editing framework. One set of weights natively performs text-conditioned joint audio-video generation, bidirectional cross-modal synthesis, and instruction-guided editing of either or both modalities, with no task-specific adapters and no secondary checkpoints (Fig. 1).

2. A progressive training paradigm. Each branch is first trained to full generative and editing competence on abundant unimodal corpora; both branches are then frozen and the scarce paired corpus is used only to train lightweight cross-modal bridges and their adaptive-normalization parameters. Alignment finetuning spends under 1% of the unimodal pretraining budget (0.67M paired samples vs. 34.3M video and 54.9M audio samples).

3. Asymmetric modality-decoupled noise scheduling. Per-branch warped noise trajectories are derived from a single shared timestep index, steering training off the diagonal of the 2D noise-level space, while "clean-branch subtasks" explicitly supervise the boundary states (one stream pinned at σ = 0) that continuous warps rarely reach.

4. Two inference modes enabled by that scheduling. Intent-aware editing pins the preserved modality at σ = 0 so the sampler returns the exact source latent rather than an approximate resynthesis; post-inference self-refinement re-noises an accepted branch and regenerates only the unsatisfactory one.

5. Empirical evaluation across joint editing, joint generation, and cross-modal generation benchmarks, plus ablations isolating each design choice.

Main Findings

• Joint editing (JAVEditBench): UniAVC leads on AQ (2.59), IC (4.42) and VF (4.48). It does not lead everywhere: JAVEdit has the best AVS (0.3688 vs. UniAVC's 0.2975) and AVQ (3.88 vs. 3.85), and MiniMax-H3 has the best VQ (0.0605 vs. UniAVC's 0.0599).

• Joint editing (InsAVE-80K test split): UniAVC records the best FVD (150.86), TV-A (25.66), FAD (1.99) and AV-A (28.09). InstructAV2AV scores higher on TA-A (34.96 vs. 27.36) and MiniMax-H3 higher on PEAVS (3.75 vs. 3.51).

• Joint generation (Verse-Bench): UniAVC has the best WER (0.08) and ties for best AS (0.49). MiniMax-H3 is stronger on PQ (6.87 vs. 6.26), CS (0.36 vs. 0.30) and LSE-C (6.96 vs. 6.72); JoVA is higher on MS (0.75 vs. 0.40).

• Cross-modal generation (VGGSound test split): UniAVC leads both directions — audio-to-video FVD 103.17 (vs. AVTok 150.26 and TempoTokens 786.61), DeSync 0.773 and IB 0.291; video-to-audio FAD 10.12 (vs. MMAudio 17.09 and VinTAGe 80.06), DeSync 0.650 and IB 0.305.

• Progressive training beats single-stage training across every JAVEditBench metric, with the largest gains on cross-modal measures rather than visual quality alone (e.g. AVQ 3.85 vs. 3.53, AQ 2.59 vs. 2.27), supporting the claim that paired data is being spent on cross-modal alignment.

• The two inference modes are complementary. Intent-aware editing raises audio quality (AQ 2.53 vs. base 2.37) by conditioning on uncorrupted source latents; self-refinement gives the largest instruction-compliance gain (IC 4.39 vs. base 4.22); combining them gives peak synchronization (AVS 0.2975, AVQ 3.85).

• Schedule asymmetry matters. With (k_a, k_v) = (1, 1), i.e. no warp, cross-modal metrics drop (AQ 2.48, AVQ 3.67) while visual quality barely moves. Overshooting to (3, 12) also degrades most metrics (AVS 0.2618), indicating warping saturates beyond a moderate shift; the chosen (2, 9) is best.

Methodology in Plain English

Backbone. UniAVC builds on LTX-2.3. Video and audio each get their own transformer stack with self-attention, feed-forward blocks and adaptive normalization; both read a shared text embedding. The stacks meet only at per-block cross-attention "bridges" (video queries attending to audio, and vice versa). The key trick: each bridge's gate, scale and shift are derived from the partner branch's noise level, so the audio→video bridge is controlled by σ_a and the video→audio bridge by σ_v.

Editing as in-context conditioning. Instead of adding new input branches, reference latents are concatenated onto the target token sequence for the same modality, separated by a temporal offset in the rotary embeddings and a learned token-type embedding. Attention masks let target queries read the whole sequence but restrict reference queries to the uncorrupted reference span — including inside the cross-modal bridges — so conditioning does not drift as noise grows. Unlike prior in-context video editors, the reference tokens are not forced onto the exact spatial positions of their targets, which would block edits that relocate content.

Stage 1 — unimodal pretraining. Each branch learns generation and editing on its own data: 34.3M video samples from a 15.3M pool (Boogu-Image, Ditto, OpenVE-3M) and 54.9M audio samples from a 14.9M pool (AudioSet, VGGSound, WavCaps, Emilia, LibriTTS, AudioEdit, ISCSLP 2026 CoT-TTS Challenge). Still images are kept in the video mixture (0.33 probability per image task, 0.17 per video task) to improve frame fidelity.

Stage 2 — alignment finetuning. Both branches are frozen and only the cross-modal bridges and their normalization parameters are trained, on 0.67M paired samples drawn from a 0.74M pool (TAVGBench, JAVEdit, InsAVE-80K).

Decoupled schedules. A single step index i is drawn per sample, and the base grid is warped per modality by S_k(σ) = kσ / (1 + (k−1)σ), then rescaled to a shared terminal level. With k_v = 9 and k_a = 2, video stays noisier than audio at every interior step, so the acoustic stream resolves first and feeds progressively cleaner conditioning to the video branch.

Boundary supervision. Because continuous warps rarely land exactly on σ = 0, a fraction of samples are assigned to subtasks where one stream is pinned clean and its loss is masked — the audio-conditioned-video case and its mirror.

Training and sampling setup. 720p, 121 frames, 24 fps, 16 kHz mono audio; AdamW at learning rate 1e-5, weight decay 0.01, bf16. Text and reference dropout are both 0.1, creating separate guidance axes for instruction compliance and source fidelity; a reconstruction subtask with a null-edit instruction supervises preserving unmentioned content. Inference uses 30 denoising steps with CFG scales (3.0, 2.0) for video and (1.5, 3.0) for audio, and λ = 0.5 for self-refinement.

Why This Matters

For research: The paper reframes a data problem as a scheduling problem. Showing that a single shared noise schedule restricts training to the diagonal of a 2D noise space — and that boundary supervision is needed to reach off-diagonal states — gives a concrete, transferable explanation for why asymmetric multimodal tasks fail at inference. It also argues that paired corpora should be spent on cross-modal coupling, not on teaching a model to render motion or synthesize clean audio, which unimodal data already covers.

Real-world applications:

• Video post-production: restyle a scene while keeping the original dialogue or ambience bit-exact.
• Dubbing and localization: change speech in a clip while holding the visuals fixed, or the reverse.
• Foley and sound design: regenerate only the soundtrack of an approved shot instead of resampling the whole pair.
• Selective repair: keep the take whose audio works and regenerate just the visuals (or vice versa) via self-refinement.

Industry relevance: A single checkpoint covering generation and editing removes the cascaded-pipeline tax of multiple models, resynchronization modules and alignment losses, and removes the need for per-control adapters. The finding that alignment needs under 1% of the unimodal budget is directly relevant to teams that can afford large unpaired corpora but not large paired ones.

Future Directions

• Cover the full noise-pair space. The authors note that current coverage is confined to a 1D trajectory plus two boundary faces, not the whole (σ_a, σ_v) plane; denser or learned off-diagonal sampling is the natural extension.
• Make bridges repair, not just align. The conclusion states that cross-modal bridges align representations without repairing unimodal deficits — so a clean partner stream cannot currently compensate for a weak branch.
• Move beyond a single shared index. The related work contrasts UniAVC's phase-locked schedules with independently sampled timesteps (as in MoVA and UniDiffuser), which cover more of the noise space but break the deterministic link to the step counter; reconciling the two is an open design question.
• Broaden evaluation. Only JAVEditBench, InsAVE-80K, Verse-Bench and the VGGSound test split are reported here; generalization to other editing benchmarks and to settings beyond 720p/121 frames/24 fps is not reported.

Target Audience

Researchers and advanced engineers working on multimodal generative models — diffusion/flow-matching video and audio systems, instruction-guided editing, and unified generation architectures. It is also useful for practitioners building production audio-video pipelines who need to understand the trade-offs between cascaded specialist models and a single unified checkpoint. Readers without a background in diffusion transformers and rectified flow will find the method sections demanding.

Authors’ abstract

Unified audio-video creation requires a single model capable of text-conditioned joint synthesis, bidirectional cross-modal generation, and instruction-guided editing across both modalities. Existing methods rely heavily on paired corpora, which are scarce for joint generation and even rarer for editing. Consequently, they are spent learning single-branch generative and editing capabilities that abundant unimodal data could readily teach, rather than on the cross-modal coupling. Furthermore, prevailing systems apply a single shared noise schedule to both modalities, neglecting two critical limitations. First, due to the significant discrepancy in information density, video requires prolonged high noise to organize spatial and motion structure, whereas the sparser audio signal can settle earlier during the synthesis process, the dynamics a uniform schedule might compromise. Second, lockstep denoising leaves asymmetric workflows, such as conditioning on one modality to synthesize or edit another, untrained. We present UniAVC, which addresses both obstacles. It first trains each branch on unimodal dataset to full generative and editing competence, and subsequently dedicates paired data to exploit cross-modal correspondence. Modality-decoupled schedulers meanwhile warp the noise schedule independently from a shared timestep, while subtasks with one clean modality directly supervise asymmetric configurations. These settings respectively enable intent-aware editing, which preserves one modality intact while enlisting it to constrain the other, and post-inference self-refinement, which regenerates only an unsatisfactory branch against an accepted one. UniAVC thus spans multiple creation tasks from a single set of weights without task-specific adapters.