AV-GRPO: Modality-Anchored Decoupling Diffusion Reinforcement Learning for Joint Audio-Video Generation

AI summary

Overview

• Research area: Computer vision and generative modeling, specifically reinforcement-learning-based post-training of joint audio–video (text-to-audio-video) diffusion/flow models.
• Technical level: Advanced. The paper assumes familiarity with GRPO-style policy-gradient RL, rectified flow / diffusion SDEs, and reward-model-based post-training.
• Scope: The paper proposes a modality-anchored online RL framework (AV-GRPO) plus a five-dimensionally decoupled training set (5DAV) for post-training dual-tower audio–video generators, evaluated on JavisBench and VABench against the LTX-2.3 (22B) base model and the GDPO baseline.

What This Paper Is About

Joint audio–video generators often produce streams that are individually low quality, misaligned with the text prompt, or out of sync with each other, and scaling pretraining does not directly target these failures. The paper asks how to apply reward-guided reinforcement-learning post-training to such models when audio and video rewards are heterogeneous, the two towers are computationally expensive to update together, and synchronization difficulty depends on whichever counterpart modality was sampled. Its goal is to make reward attribution and synchronization optimization accurate while cutting the cost of training both modality towers.

Key Contributions

1. AV-GRPO framework: A modality-anchored reinforcement learning framework that alternates controlled modality-wise updates (audio-anchored video optimization and video-anchored audio optimization, switching every n training steps) to obtain clearer reward attribution and balanced audio–video improvement.
2. Three complementary mechanisms: (i) modality-anchored rollouts that disentangle reward objectives and enable controlled synchronization comparisons; (ii) trajectory locking and tower freezing that isolate modality-wise credit assignment while reducing memory overhead; and (iii) decoupled optimization hyperparameters that accommodate asymmetric audio–video learning dynamics.
3. 5DAV dataset: A five-dimensionally decoupled training set of 5,760 prompts, with dimensions for semantic hierarchy, sound source type, synchronization difficulty, temporal complexity, and instruction granularity.
4. Empirical validation: Experiments on JavisBench and VABench showing improvements over LTX-2.3 and GDPO in generation quality, text–modality alignment, and synchronization under both LoRA and full fine-tuning, plus ablations on the dataset and the alternating schedule.

Main Findings

• Gains over the base model in perceptual quality: On JavisBench, the full fine-tuned model raises Audio Quality from 5.097 (LTX-2.3) to 5.798; on VABench, Audio Aesthetics rises from 3.319 to 3.631.
• Gains in semantic alignment: Full fine-tuning raises JavisBench CLIP from 0.318 to 0.327 and CLAP from 0.408 to 0.468.
• Gains over GDPO on cross-modal measures: AV-GRPO (full) raises JavisBench AV-IB from 0.224 to 0.247 and JavisScore from 0.202 to 0.222, and reduces DeSync from 0.708 to 0.607. On VABench, Lip Sync rises from 1.439 to 1.646 and DeSync drops from 0.726 to 0.542.
• Differences between training regimes: Full fine-tuning gives the highest JavisBench AQ (5.798) and AV-IB (0.247), while LoRA reaches a slightly higher CLIP score (0.330 versus 0.327) and a lower DeSync value (0.554 versus 0.607), so improvements are broad rather than uniform.
• Documented exceptions to the gains: LoRA's VQ is 5.816, below the base model's 5.855, and the full model's VABench visual-realism score is 4.395 versus 4.399 for the base model.
• Alternating schedule matters: Ablations on JavisBench-mini compare no alternation and switch intervals of two, four, and eight steps; the four-step schedule gives the strongest overall balance across the plotted measures, although individual metrics need not peak at the same interval.
• Dataset design matters: A comparison against VGGSound under identical full fine-tuning settings, and removals of the easiest/most difficult levels of Synchronization Difficulty (D) and Instruction Granularity (C), are reported in Figure 3; the paper states the results demonstrate that decoupling and categorical partitioning contribute substantially, but does not report the numerical values in the text.
• Frozen-tower strategy enables large-model training: Full-parameter post-training of the 22B LTX-2.3 model runs on a single node of eight NVIDIA A800 80 GB GPUs.
• Theoretical support: Appendix A shows that, under a common reference joint law, a Dobrushin contraction condition (δ_v δ_a < 1), and an exact conditional-kernel oracle, the alternating conditional Gibbs kernels converge geometrically in total variation to the joint KL-regularized optimum P*.

Methodology in Plain English

The starting point is Flow-GRPO, which turns the deterministic flow sampler into an equivalent stochastic (SDE) sampler so that each denoising step can be treated as an action in a policy-gradient update, with a probability ratio and a group-standardized advantage. AV-GRPO applies this update to one tower at a time. In an audio-anchored phase, the method first generates a full audio–video pair, keeps the entire audio denoising trajectory as a fixed "anchor," and then samples a group of G video trajectories that are all conditioned on that same anchor. Only the video tower is updated; the audio tower is frozen and no anchor reward is needed, because a constant anchor reward disappears when advantages are centered within the group. The video-anchored phase is symmetric, and the two phases alternate every n steps (four in the main runs).

This anchoring has two effects: it removes the confound in which a synchronization score can be high simply because the sampled counterpart was easy, and it lets the target tower's reward variation reflect only its own sampled trajectories. Freezing the counterpart also means its backward activations and optimizer states are not needed, reducing memory cost. Because the towers have different dynamics (different pretrained capability, latent scale, reward sensitivity, and learning speed), the paper decouples hyperparameters: for LTX-2.3 it uses a_V = 0.02 and a_A = 0.8 in the noise scale σ_t = a·sqrt(t/(1−t)), and it reweights the video policy and KL terms per step (following LongCat-Video) because the video tower became over-exposed with the original objective while the audio tower keeps the original weighting. An Adaptive Noise Clipping (ANC) scheme bounds the stochastic increment's standard deviation near t = 1, scaling the stochastic term by λ_t and the drift correction by λ_t²; 15 sampling steps are used. Rewards come from VideoAlign, CLIP, and DeSync for the video tower and Audiobox Aesthetics, CLAP, and DeSync for the audio tower, composed following GDPO, with a CLAP guardrail that uses only the alignment advantage when the group's average CLAP falls below a threshold.

Why This Matters

Impact on research. The paper reframes coupled multimodal preference learning as a set of conditional unimodal subproblems, which gives a principled way to assign credit in dual-tower generators instead of updating both branches with one shared advantage. It also contributes a controlled, difficulty-labeled dataset and an accompanying convergence argument (Appendix A) that connects alternating conditional updates to a joint KL-regularized optimum, which may be reusable for other multi-stream generative models.

Real-world applications:

• Text-to-video or text-to-audio-video content creation tools where audio and video must be generated together and stay in sync.
• Film, advertising, and short-form video production, where lip sync, foley/object sounds, and music performance need frame-level or physical-causal synchronization.
• Game and interactive media asset generation, where event-driven effects (for example, pitch rising as a train approaches) must match visual action.
• Accessibility and dubbing pipelines, where text-prompt alignment plus audio–video synchronization are quality-critical.

Industry relevance. The frozen-tower strategy lets a 22B model be fully post-trained on eight NVIDIA A800 80 GB GPUs, indicating that reward-guided post-training of large joint generators is feasible on a single node rather than requiring much larger clusters. The dataset's adjustable category sampling ratios also support staged training curricula and targeted debugging of known model weaknesses.

Future Directions

• Extend the anchoring scheme beyond two modalities. The framework is formulated for an audio tower and a video tower; whether modality anchoring generalizes to more than two streams is not addressed.
• Close the remaining metric gaps. LoRA's VQ (5.816) stays below the base model's 5.855, and full fine-tuning's VABench visual realism (4.395) is marginally below the base (4.399); the paper does not resolve these tradeoffs.
• Tighten the gap between theory and practice. The Appendix A convergence result relies on an exact conditional-kernel oracle and a Dobrushin contraction condition; the paper does not report how closely the trained models approximate these conditions.
• Make the difficulty controls more automated. 5DAV relies on predefined difficulty levels for Synchronization Difficulty and Instruction Granularity; the ablation indicates these levels supply distinct signals, but the paper does not report how levels should be scheduled over training, nor the numerical magnitude of the effects.

Target Audience

Researchers and engineers working on diffusion/flow-based generative models, reinforcement learning from reward models, and multimodal generation, particularly those post-training large dual-stream audio–video systems. It is also relevant to practitioners who need controllable, difficulty-labeled training data for generative media pipelines. Readers without background in GRPO-style policy optimization and diffusion SDEs will find the method sections difficult, and the benchmark tables assume familiarity with metrics such as DeSync, AV-IB, CLIP, and CLAP.

Authors’ abstract

Recent years have witnessed major progress in joint audio-video generation. Existing models still suffer from limited per-modality fidelity, insufficient text-modality alignment and weak cross-modal synchronization. While reinforcement-learning post-training offers a promising remedy, directly adapting it to joint audio-video generation is challenging. Heterogeneous multimodal rewards entangle learning signals and complicate credit assignment. Joint optimization of two modality towers is computationally expensive given their divergent dynamics. Moreover, synchronization evaluation difficulty depends on paired samples, preventing fair reward comparisons. We propose AV-GRPO, a modality-anchored online diffusion RL framework, and 5DAV, a decoupled, difficulty-controllable training dataset. AV-GRPO includes three key modules: (1) modality-anchored rollouts to disentangle learning signals and stabilize difficulty; (2) trajectory-locked frozen-tower optimization to reduce cost and reassign credit; (3) adaptive objectives and perturbation strengths tailored to modality-specific dynamics. This converts coupled multimodal preference learning into unimodal subproblems for precise reward attribution and better synchronization. Our 5DAV dataset decouples samples across five dimensions for systematic training. Experiments on JavisBench and VABench demonstrate AV-GRPO outperforms LTX-2.3 in generation quality, semantic alignment and cross-modal synchronization under LoRA and full fine-tuning. Ablations confirm our designs. Code and data: https://github.com/zhiyuxu03/AV-GRPO