SyncReward: Human-Aligned Synchronization Reward for Audio-Visual Generation

AI summary

Overview

• Research area: Computer vision, specifically joint audio–visual generation, with a focus on synchronization evaluation, human-annotated benchmarking, and reward modeling for generative post-training.
• Technical level: Advanced. The paper assumes familiarity with contrastive representation learning, cross-modal Transformers, ordinal human-rating aggregation, reward models, and reinforcement-learning post-training for diffusion/flow-matching video generators.
• Scope (one sentence): The paper builds a 80K-clip human-rated synchronization dataset, a held-out 1K-clip benchmark, and a 1.6B-parameter reward model (SyncReward) that both scores audio–visual synchronization and serves as an optimization signal for inference-time selection and post-training of an audio–visual generator.

What This Paper Is About

Audio–visual generators can now produce video with speech, music, and sound effects, but the generated audio often drifts out of time with the visible events (a voice continuing while the mouth is still, an impact sound landing before the object contact). The paper argues that progress is blocked by a lack of human-annotated synchronization data, a lack of reliable benchmarks, and the fact that existing synchronization metrics and general omni-modal LLM judges disagree substantially with human judgment on generated content. The goal is to build data, a benchmark, and a learned reward that both measures synchronization the way humans do and can be used to improve generation.

Key Contributions

1. SyncReward-Data-80K: A dataset of 80K professional synchronization ratings on generated clips from eight audio–visual generators, organized by a three-level sound-event taxonomy covering four families (Speech, Onset, Instrument, Ambient), 16 intermediate categories, and 58 fine-grained subcategories. Each clip was independently assessed by five professional annotators under a two-stage protocol.
2. SyncReward-Bench: A held-out benchmark of 1,000 generated clips, strictly disjoint from the training data and independently rated by multiple professional annotators, used to systematically compare synchronization-specific metrics, off-the-shelf omni-modal judges, and fine-tuned omni-modal models.
3. SyncReward reward model: A 1.6B-parameter model trained in two stages (semantic–temporal contrastive pretraining on real clips, then human-supervised training on generated clips) that achieves the strongest agreement with human ratings among the evaluated methods across all four sound-event families.
4. Demonstrated use as a reward: SyncReward improves synchronization through inference-time Best-of-16 selection and DiffusionNFT reinforcement-learning post-training on LTX-2.3, outperforming the evaluated alternative rewards in both settings.

Main Findings

• Human-alignment gap in existing evaluators: On SyncReward-Bench, specialized synchronization metrics and omni-modal judges show substantial misalignment with human judgments. AV-Align reaches a Spearman correlation of 4.4 and pairwise accuracy of 51.8 on Speech, and falls below 50% pairwise accuracy on Onset (47.3) and Instrument (43.2).
• Fine-tuning on the same data helps but is not enough: Qwen2.5-Omni-3B fine-tuned on SyncReward-Data-80K reaches 61.9/75.7 (Speech), 64.7/76.6 (Onset), 51.8/71.0 (Instrument), and 73.9/81.3 (Ambient) in Spearman ρ × 100 and pairwise accuracy. Qwen3-Omni-30B-A3B-FT reaches 69.5/79.4, 68.6/78.8, 64.7/77.1, and 77.2/82.9. Both remain below the smaller SyncReward model in every category.
• SyncReward is the best metric: SyncReward (1.6B parameters) achieves 79.7/85.1 on Speech, 70.5/79.9 on Onset, 68.1/79.3 on Instrument, and 82.1/85.9 on Ambient. Relative to the strongest baseline, Qwen3-Omni-30B-A3B-FT, this is an improvement of 1.9–10.2 Spearman points and 1.1–5.7 pairwise accuracy points, with the largest margin on Speech.
• SyncReward is the best reward for selection: In inference-time Best-of-16 selection with 16 LTX-2.3 candidates per prompt, SyncReward achieves human and Gemini preference scores of 63.67 and 71.94, exceeding the next-best reward by 11.1 and 17.7 points. For comparison, SynthSync reaches 52.57 and 54.20, Synchformer 38.90 and 41.16, and AV-Align 36.15 and 40.83, against a base-model reference of 32.31 and 47.97.
• SyncReward is the best reward for post-training: With DiffusionNFT post-training, SyncReward leads the next-best method by 12.1 points in human preference (65.78) and 15.2 points under the Gemini judge (67.86).
• Metric-specific wins do not imply human preference: AV-Align and Synchformer perform best on the metrics they directly target but receive much lower preference scores. AV-Align post-training raises its own metric from 0.306 to 0.420 while degrading the other three automatic metrics relative to the base model.
• Both training stages matter: The full model (Spearman 78.2, pairwise accuracy 83.6) outperforms Stage 1 only (60.5/74.6), Stage 2 only (40.9/66.3), and PE-AV initialization (26.6/60.4). The full model improves over Stage 1 alone by 17.7 Spearman points and 9.0 pairwise accuracy points.
• Temporal negatives matter more than semantic negatives: In the Stage 1 ablation, removing temporal negatives causes a larger correlation drop (7.6 points) than removing semantic negatives (4.1 points). Removing the ranking loss in Stage 2 lowers pairwise accuracy by 3.8 points, versus 1.6 points when removing the regression loss.
• Representations encode synchronization more linearly: SyncReward's first principal component has an absolute Spearman correlation of 0.47 with human ratings, versus 0.29 for Synchformer and 0.04 for Qwen3-Omni. A linear probe gives 0.41 for SyncReward versus 0.28 and 0.25 for the other two. The authors note Qwen3-Omni's low PC1 correlation does not rule out such information elsewhere in its representation.
• Post-training can trade off prompt fidelity: In a case study, SyncReward-guided post-training improves visible lip articulation (score 0.17 to 1.84) and vocal correspondence in a singing example (0.62 to 1.63), but the post-trained singing output omits the instrument named in the prompt.

Methodology in Plain English

The researchers first needed a way to define what "synchronized" means for generated audio–video, so they used Qwen2.5-Omni-7B to caption real clips from AudioSet and VGGSound, then organized those descriptions into a three-level taxonomy of on-screen sound events. Off-screen narration and background music were excluded, so every target sound must be grounded in something visible.

Because the original caption pool was highly imbalanced, they mapped captions to taxonomy categories and resampled according to predefined quotas, producing three disjoint sets of 10K captions. An LLM rewrote each sampled caption into a generation prompt that explicitly describes the temporal correspondence between a visible sound source, its action, and the associated sound.

They then generated clips with eight proprietary and open-source generators — Kling 3.0, Seedance 2.0, Veo 3, LTX-2.3, daVinci, MOVA, NAVA, and Ovi — with a pool of roughly 10K clips per model. Five professional annotators rated each clip with hidden generator identity under a two-stage protocol: first decide whether the audio is related to the visible scene, then assign related clips an ordinal score of 0 (clearly desynchronized), 1 (partially or uncertainly synchronized), or 2 (well synchronized). Annotations were aggregated into bias-corrected consensus targets, and splits were constructed so no (uid, model) pair is shared between training and validation. A separate 1,000-clip benchmark was annotated independently.

The reward model itself trains in two stages. Stage 1 splits real audio–visual clips into T temporally corresponding segments and learns encoders with a symmetric contrastive objective in which same-clip-same-time pairs are positives, different time positions within the same clip are temporal negatives, and segments from other clips are semantic negatives. Stage 2 freezes both encoders and trains a cross-modal Transformer plus a scalar score head on the human-rated generated clips, using a reward token and modality separator, with a SmoothL1 regression loss combined with a pairwise ranking loss. The sigmoid output is scaled to the [0, 2] human rating range.

For evaluation, they report Spearman's rank correlation and pairwise ranking accuracy against human ratings. For the reward setting, they use SyncReward for Best-of-16 selection and for DiffusionNFT post-training of LTX-2.3, measuring results with human and Gemini pairwise preferences plus four automatic synchronization metrics (SyncR., DeSync, AV-Align, JavisScore). The paper notes that the preference scores in the reward experiment are provisional estimates from base-referenced comparisons and will be replaced by complete all-pairs evaluation.

Hardware and training details: Stage 1 uses AdamW with a learning rate of 1×10⁻⁵ on 4×8 NVIDIA H20 GPUs with per-GPU batch size 2, with each video represented by 18 overlapping segments of 0.5 s and a stride of 0.25 s. Stage 2 uses 2×8 NVIDIA H20 GPUs with per-GPU batch size 8. The base model is the Perception Encoder Audio–Visual checkpoint (pe-av-large); Stage 1 uses AudioSet and VGGSound, totaling 2.2M natural audio–visual clips.

The paper states that the data, benchmark, model, and code will be released at https://github.com/WeChatCV/SyncReward.

Why This Matters

Impact on research. The paper shifts audio–visual synchronization from proxy objectives (temporal offsets, cross-modal embedding similarity) toward human-aligned supervision on generated content specifically. It shows that metrics trained on real video transfer less reliably to generated failures, which can include semantic ambiguity, missing events, and hallucinated events rather than simple global shifts. It also provides an open dataset and benchmark for a problem where the paper states human-annotated data and reliable benchmarks were previously lacking.

Real-world applications.

• Automated quality control for AI video tools that generate talking-head, music, or sound-effect content, catching lip-sync drift and impact-sound misplacement before a user sees them.
• Better ranking and sampling in consumer generation products, using a synchronization reward to pick the best of several candidate outputs.
• Post-training or fine-tuning of generative video models to reduce synchronization failures without redesigning the generator architecture.
• Content moderation and QC pipelines for film, advertising, and short-video platforms that mix generated video with speech, music, and effects.

Industry relevance. The work is a collaboration between the Gaoling School of Artificial Intelligence at Renmin University of China and WeChat, Tencent Inc. (Kaisi Guan's contribution was done during an internship at Tencent). The evaluated generators include major commercial systems (Kling 3.0, Seedance 2.0, Veo 3), and the reward is demonstrated on a real open generator (LTX-2.3), which makes the recipe directly relevant to teams shipping audio–visual generation at scale.

Future Directions

• Combining synchronization with complementary objectives. The case study shows that optimizing synchronization alone can drop prompt fidelity (a specified instrument missing after post-training). The authors explicitly motivate combining synchronization rewards with objectives that preserve requested scene content.
• Closing the metric gap for specialized methods. AV-Align and Synchformer win on their own targeted automatic metrics but score poorly on human and Gemini preference, suggesting a need to understand which measurable quantity actually predicts human satisfaction.
• Extending the benchmark and annotation protocol. The Instrument subset is smaller because music-dominant clips were removed from the evaluation set, and the reward experiment's preference scores are described as provisional. A complete all-pairs evaluation and broader category coverage are natural next steps.
• Going beyond real-clip pretraining. Methods trained mainly on real video transfer less reliably to generated content, and Stage 2 alone performs far worse than the two-stage pipeline, raising the question of how much further generated-content supervision (rather than real-clip pretraining) can be pushed.

Target Audience

Researchers and engineers working on audio–visual generation, multimodal reward modeling, and generative model evaluation will benefit most. It is also relevant to practitioners building evaluation suites for generated media, to teams doing reinforcement-learning or preference-based post-training for video generators, and to readers interested in human-annotation methodology for subjective, temporally fine-grained qualities. Because the technical level is advanced, readers should already be comfortable with contrastive learning, ordinal rating aggregation, and post-training algorithms such as DiffusionNFT.

Authors’ abstract

Recent years have witnessed rapid progress in joint audio--visual generation. However, audio--visual synchronization remains a challenge, as generated speech and sound often temporally misalign with visual events. A key bottleneck is the lack of human-annotated data, reliable evaluation benchmarks, and synchronization-aware evaluation models for generated audio--visual outputs. To address these limitations, we conduct a comprehensive study spanning data, benchmarking, and reward modeling. We first introduce SyncReward-Data-80K, a dataset containing 80K professional synchronization ratings for clips from eight audio--visual generators. A taxonomy-guided construction pipeline covers four sound-event families: Speech, Onset, Instrument, and Ambient. We further construct \textbf{SyncReward-Bench}, a held-out benchmark of 1K clips, each independently rated by multiple professionals. Using this benchmark, we systematically evaluate specialized synchronization metrics and omni-modal models, revealing their substantial misalignment with human judgments. Finally, we develop SyncReward, a reward model that combines semantic--temporal contrastive pretraining on real clips with human supervision on generated clips. As an evaluation metric, SyncReward achieves the strongest agreement with human ratings among the evaluated methods. As a reward, it improves synchronization through inference-time best-of-$N$ selection and reinforcement learning based post-training. We will release the data, benchmark, model, and code at https://github.com/WeChatCV/SyncReward.