All modalities are equal, but video is more equal: Closing the Cross-Attention Gap in Joint Video Generation

AI summary

Overview

Research area: Computer Vision / multimodal generative modeling — specifically joint (single-process) generation of video together with a companion modality such as 3D human motion or audio.

Technical level: Advanced. The paper assumes familiarity with diffusion/flow-matching transformers, cross-attention, KL divergence, LoRA fine-tuning, and multimodal evaluation benchmarks.

Scope: The paper diagnoses an asymmetry between the two directions of cross-modal attention in pretrained joint video generators and proposes a lightweight KL-based regularizer, RecCAR, that closes this gap, validated on video–motion and video–audio generation.

Publication details: arXiv:2609.27901v1 [cs.CV], 23 Sep 2026, licensed CC BY 4.0. Authors are affiliated with Bar-Ilan University and NVIDIA.

What This Paper Is About

Modern joint generators produce video and a companion modality (motion, audio) inside one denoising process, with cross-attention connecting the two streams in both directions. The authors find that this bidirectional wiring is functionally one-sided: the video-to-modality attention pathway learns rich, well-localized correspondences, while the reciprocal modality-to-video pathway — the one that lets motion or audio correct the video — stays substantially weaker. The goal is to strengthen that weaker pathway using only information already present in the pretrained model, without external supervision or new architecture.

Key Contributions

1. Diagnosis of a systematic asymmetry. The authors show that nominally bidirectional joint multimodal diffusion transformers develop a strong video-to-modality correspondence but a much weaker reciprocal modality-to-video correspondence, and they formalize the disagreement as the reciprocal correspondence gap Δ_corr, computed as a KL divergence between the two directions' distributions over video tokens.

2. RecCAR (Reciprocal Cross-modal Attention Regularization). A lightweight KL regularizer that treats the well-established video-to-modality correspondence as a fixed reference (via stop-gradient) and aligns the weaker modality-to-video pathway toward it, added to the original generative loss as L_total = L_gen + λ_RecCAR · L_RecCAR.

3. A modality-agnostic recipe. The same formulation is applied unchanged to two structurally different companion modalities, with parameter-efficient adaptation: the pretrained backbone is frozen and only LoRA parameters on attention projections are trained.

4. Demonstrated generality and controlled ablations. Gains are shown on video–motion (EchoMotion) and video–audio (LTX-2, JavisDiT++), with ablations separating the effect of the alignment loss from the effect of generic fine-tuning on the same curated data.

Main Findings

• Video–motion anatomy improves. Applied to EchoMotion, RecCAR raises the VBench Human Anatomy score from 0.69 to 0.75, the largest gain among the evaluated metrics, while Motion Smoothness rises from 0.98 to 0.99, Dynamic Degree stays at 0.83, Aesthetic Quality goes from 0.56 to 0.57, Imaging Quality from 0.6785 to 0.6869, Appearance Consistency from 0.2814 to 0.2841, Background Consistency from 0.9346 to 0.9406, and Subject Consistency from 0.9108 to 0.9245. Baseline comparisons on the same 1,000-prompt test set report CoMoVi at Human Anatomy 0.67 and FlowMo at 0.65.

• The gain is not from retained motion difficulty. The authors state that Dynamic Degree is comparable to EchoMotion while anatomical correctness rises substantially, arguing the improvement comes from more accurate and coherent motion rather than slower or simpler motions.

• Video–audio synchronization improves on T2AV-Compass. Applied to LTX-2, RecCAR reduces absolute audio–video desynchronization from 0.804 to 0.752 on the 500 prompts of T2AV-Compass, the best value among evaluated methods (LTX-2 0.804, ITS-LTX-2 0.792, UniAVGen 0.866). Predicted desync drops from 0.095 to 0.068. Audio realism improves (AAS 4.180 to 4.225, MTC 3.920 to 3.975), as do Video Aesthetics (4.600 to 4.628), Technical video quality (0.9942 to 0.9955), Text-Video alignment (0.0344 to 0.0353), Text-Audio alignment (0.1595 to 0.1619) and Audio-Video alignment (0.115 to 0.124).

• The same objective helps a second audio–video backbone. On JavisDiT++, RecCAR reduces absolute AV Desync from 1.078 to 1.046 and predicted desync from 0.307 to 0.238.

• AVGen-Bench confirms the trend. On the 235-prompt AVGen-Bench, RecCAR lowers LTX-2 DeSync from 0.424 to 0.406 and JavisDiT++ DeSync from 0.518 to 0.438, while CLAP and AV-CLIP are preserved or improved (JavisDiT++ + RecCAR CLAP 0.23, AV-CLIP 0.74, versus 0.21 and 0.72 for the base model). The paper notes ITS yields only a marginal DeSync reduction for JavisDiT++ (0.518 to 0.507) and none for LTX-2 (0.424 to 0.447, i.e. worse).

• Standard fine-tuning does not explain the gains. In the video–motion ablation, fine-tuning EchoMotion on the same curated data without the alignment loss worsens N-MPJPE (0.197 to 0.208), MPJPE (0.34 to 0.35) and Human Anatomy (0.70 to 0.66), whereas RecCAR improves all three (N-MPJPE 0.148, MPJPE 0.29, Human Anatomy 0.78) while keeping Dynamic Degree at 0.85 and Motion Smoothness at 0.99.

• The audio–video ablation matches. Standard fine-tuning moves DeSync only from 0.518 to 0.498 for JavisDiT++ and from 0.424 to 0.421 for LTX-2, while RecCAR reaches 0.438 and 0.406 respectively, with CLAP and AV-CLIP unchanged.

• Qualitative evidence. For a trotting horse, the baseline LTX-2 produces hoofbeat transients that lead or lag the visible hoof contacts and can show visual artifacts such as missing hooves or multiple legs, whereas RecCAR better aligns the waveform with the gait. For motion, the authors report that a head-motion token's video-to-modality correspondence localizes the head while the reciprocal correspondence concentrates on the upper torso.

Methodology in Plain English

1. Express both attention directions the same way. For each companion-modality token, both directions are turned into a probability distribution over video tokens. The video-to-modality direction is already normalized over video tokens; the modality-to-video direction is re-normalized over video tokens (it is natively normalized over modality tokens) so the two answers can be compared directly. Both then answer: which video tokens does this modality token correspond to?

2. Measure the disagreement. The reciprocal correspondence gap averages the KL divergence between the two distributions across modality tokens, cross-attention layers, and attention heads.

3. Regularize in one direction. RecCAR freezes the stronger video-to-modality distribution with a stop-gradient and pushes the weaker modality-to-video distribution toward it, so the established correspondence becomes an internal target. No external labels, auxiliary models, or new correspondence supervision are needed.

4. Train lightly. The total loss adds the regularized gap to the original denoising/flow-matching objective, weighted by λ_RecCAR ≥ 0. Only LoRA parameters are optimized; the pretrained backbone stays frozen, and there is no additional inference-time component.

5. Curate data for the video–motion track. 1,500 human text prompts were drawn from VidProM with 1,000 more held out for testing; 16 video–motion pairs were generated per training prompt by varying the random seed, giving roughly 18,000 completed generations as candidates. On a calibration subset of 25 videos, 2D pose was estimated with MediaPipe, 3D motion was projected into the video's image coordinates using known camera parameters, and MPJPE and NMPJPE were computed; Gemini rated anatomical plausibility on a 1–10 scale (below 5 considered implausible) from RGB video only, without seeing the motion or the errors. Pairs with both MPJPE and NMPJPE below 0.15 were treated as good matches. 3D pose for training came from CameraHMR monocular estimation (camera-space SMPL), not the generated motion. The final dataset is 4,292 video–motion training pairs, to be released upon acceptance.

6. Train the two tracks. Video–motion: LoRA with rank 128 on all joint self-attention weights, 10 epochs, λ_RecCAR = 0.01, AdamW at learning rate 1e-5, effective batch size 8, roughly 48 GPU-hours on 4 H100 GPUs. Video–audio: about 4,300 randomly sampled VGGSound training clips used as provided without extra filtering, LoRA rank 128 on all cross-attention weights, 10 epochs, loss weight 0.01, everything else frozen.

7. Evaluate. Video–motion is scored with VBench on a purpose-built 1,000-prompt VidProM test set (EchoMotion's original evaluation prompts were not released) against EchoMotion, CoMoVi, and FlowMo. Video–audio is scored on T2AV-Compass (500 prompts) and AVGen-Bench (235 prompts) against LTX-2, JavisDiT++, UniAVGen, and ITS with both backbones as baselines, using AV Desync, AAS/MTC audio realism, SigLIP, video quality, AudioBox aesthetics (PQ, CU, CE, PC), NISQA speech quality, and T-V, T-A, A-V alignment.

Why This Matters

Impact on research. The paper reframes a common assumption in joint multimodal generation: bidirectional architecture does not guarantee bidirectional information flow. It offers a self-supervised way to strengthen a weak pathway using the model's own already-learned correspondences, requiring no external reference or annotations. It also connects to prior work on modality imbalance in multimodal classifiers, extending that concern into generative settings.

Real-world applications (implied by the tasks studied):

• Animation and virtual avatar pipelines where generated 3D body motion and rendered video must agree on limb position and body orientation.
• Film, game, and video dubbing where generated audio must line up with visible events such as footsteps, hoofbeats, or tapping.
• Content creation tools built on pretrained video generators that need cross-modal coherence without retraining from scratch.
• Any joint generation system where one stream must be able to correct inconsistencies in the other.

Industry relevance. The method is a fine-tuning add-on: it freezes the base model, uses LoRA, adds no inference-time cost, and improved results are reported at roughly 48 GPU-hours on 4 H100 GPUs for the video–motion track. That profile fits organizations that license or host pretrained joint generators and want better cross-modal consistency without architectural changes.

Future Directions

• Testing whether the same reciprocal-attention objective transfers to other companion modalities that joint generators already support, such as depth or flow, which are mentioned as existing joint-generation targets but are not evaluated here.
• Determining whether RecCAR can be applied during pretraining rather than only as a post-hoc fine-tuning step, and whether the asymmetry it corrects originates in pretraining data or in the loss.
• Exploring how the regularization strength λ_RecCAR (set to 0.01 in both tracks) and the set of regularized layers ℛ affect the trade-off between synchronization and other quality metrics.
• Releasing and extending the 4,292-pair curated video–motion dataset and the anatomy-filtering criterion (MPJPE and NMPJPE below 0.15, calibrated against Gemini ratings) to other data curation pipelines.
• Investigating whether the reciprocal correspondence gap can serve as a general diagnostic for other multimodal models beyond generation.

Target Audience

Researchers and engineers working on multimodal generative models, particularly joint video–motion and video–audio diffusion transformers; practitioners doing parameter-efficient fine-tuning (LoRA) of large pretrained generators; and readers interested in attention-based interpretability and control, cross-modal alignment, and evaluation of synchronization and anatomical plausibility in generated video. Some background in transformer attention and diffusion/flow-matching training is needed to follow the method section.

Authors’ abstract

Video is a rich representation of a physical event, capturing appearance, geometry, motion, and temporal evolution. Other modalities, such as 3D body motion or audio, encode narrower aspects of the same event. We find that joint multimodal diffusion transformers exhibit a corresponding asymmetry in cross-modal correspondence: companion modalities develop strong correspondences to video, but the reciprocal correspondences through which they constrain video remain substantially weaker. We express both directions as comparable correspondence distributions over video tokens and define their disagreement as the reciprocal correspondence gap. We introduce RecCAR, standing for Reciprocal Cross-modal Attention Regularization, a KL regularizer that uses the well-established video-to-modality correspondence as a fixed reference and aligns the weaker modality-to-video correspondence toward it. Across joint video-motion and video-audio generation, RecCAR improves the Human Anatomy score from 0.69 to 0.75 and reduces audio-video desynchronization from 0.804 to 0.752, while improving overall generation