AVC-Reward: Event-Centric Audio-Video Consistency Evaluation

AI summary

Overview

Research area: Computer vision and multimodal generation — specifically, automatic evaluation of audio-video (AV) consistency produced by joint audio-video generation models, plus reward-model training and reinforcement-learning-based generator improvement.

Technical level: Advanced. The paper assumes familiarity with multimodal large language models (MLLMs), reward modeling, supervised fine-tuning, and group relative policy optimization (GRPO), as well as prior AV synchronization metrics.

Scope (one sentence): The paper proposes an event-centric, three-level audio-video consistency metric, instantiates it as a trained reward model called AVC-Reward built on Qwen3-Omni-Thinking, and validates it both as a human-aligned automatic evaluator on a new 706-sample benchmark (AVC-Benchmark) and as an online RL reward that improves the LTX-2 joint generation model.

What This Paper Is About

Joint audio-video generation models such as Veo 3, MiniMax H3, Seedance 2.0, Ovi, JavisDiT, UniVerse-1, and LTX-2 can now produce scenes containing multiple simultaneous events, audio tracks that mix physical sounds with voice-over and background music, and cross-modal relations that are not one-to-one (for example, a valid off-screen sound with no visible source). Existing consistency metrics were designed for earlier, simpler tasks in which one modality was fixed — lip-sync checking (SyncNet), motion-versus-sound-peak matching (AV-Align), temporal-offset prediction (DeSync), or whole-clip embedding similarity (ImageBind, AVHScore, JavisScore) — and they treat a clip as a single unit with one mixed audio layer and a predefined relation. The paper's goal is a general evaluator that first disentangles audio and visual events, judges each event and its cross-modal relation separately, and aggregates those judgments into a coarse-to-fine score that can also serve as a training signal for generators.

Key Contributions

1. An event-centric, three-level consistency metric. The authors formalize audio-video consistency across generation paradigms using three separate scores — global narrative harmony (S_mac), event-level temporal synchronization (S_temp), and fine-grained attribute alignment (S_attr) — each in [0, 1], combined as S_AVC = (S_mac + S_temp + S_attr) / 3 when a single scalar is needed.

2. A trainable reward model, AVC-Reward. The metric is implemented as a five-stage structured reasoning chain (audio-only analysis, video-only analysis, audio-to-video verification, video-to-audio verification, final judgment) on top of Qwen3-Omni-Thinking, trained with SFT on roughly 8k structured reasoning samples and then GRPO on a 10k rollout set built from roughly 10k human-calibrated samples.

3. AVC-Benchmark. To the authors' knowledge, the first audio-video consistency benchmark primarily built from diverse samples generated by various types of audio-video generation models (A2V, V2A, and JAVG), with each sample independently annotated by three annotators, plus a small set of controlled negatives.

4. Dual validation of the reward. AVC-Reward is shown both as an interpretable automatic evaluator with state-of-the-art human alignment among automatic methods, and as an effective online RL signal that measurably improves AV consistency when substituted for DeSync inside the OmniNFT training framework for LTX-2.

Main Findings

• Best human alignment among automatic evaluators (Table 2, 706 samples). AVC-Reward reaches Pairwise 0.586, Δ-Pairwise (Δ > 0.2) 0.728, PLCC 0.494, and SRCC 0.244. For comparison, the strongest baselines are AGAV-Rater (Δ-Pairwise 0.646, PLCC 0.261, SRCC 0.226) and AV-Align (Δ-Pairwise 0.674, PLCC 0.301, SRCC 0.215); the zero-shot base model Qwen3-Omni-Thinking scores Δ-Pairwise 0.605 with PLCC 0.347.

• A substantial gap to the human ceiling remains. The inter-annotator upper bound is Pairwise 0.756, Δ-Pairwise 0.920, PLCC 0.781, and SRCC 0.497 — roughly 0.19 higher on Δ-Pairwise and 0.29 higher on PLCC than AVC-Reward.

• Accuracy rises as human score differences grow. Across varying Δ thresholds, pairwise accuracy improves as the threshold increases, which the authors interpret as expected behavior for a human-aligned evaluator, since clearer human differences yield more reliable pairwise judgments.

• Generation-side gains generalize to metrics the reward never optimized. Using AVC-Reward inside OmniNFT improves IB-AV by 20.65–28.35%, AV-Align by 18.78–59.47%, JavisScore by 20.47–28.74%, and AVHScore by 22.00–26.80% across AV-Phys-Bench, VABench, and AVGenBench. DeSync gains remain comparable to OmniNFT's even though OmniNFT optimizes DeSync directly and AVC-Reward does not.

• The trained reward also lifts the model's own score. AVC-Score improves from 0.756 to 0.819 on AV-Phys-Bench (+8.35%), from 0.764 to 0.850 on VABench (+11.17%), and from 0.786 to 0.849 on AVGenBench (+8.02%), versus OmniNFT's +3.26%, +3.54%, and +2.76%.

• CAVP shows little separation. It changes by less than 1.7% for both methods, which the authors attribute to limited discriminative power rather than to generation quality.

• Text alignment can dip slightly. Because text-audio, text-video, and audio-video objectives are not perfectly aligned, stronger AV optimization slightly reduces some text-alignment scores (for example, IB-TV falls from 0.324 to 0.308 on AV-Phys-Bench and CLIP from 0.287 to 0.272), while single-modality audio and video quality (AQ, VQ) remain at or above the OmniNFT level — AQ on AVGenBench rises from 4.882 to 5.265.

• Human raters prefer the AVC-Reward-optimized outputs. In a blinded, randomized study over 100 prompts sampled from AVGenBench, AV-Phys-Bench, and VABench, with 10 annotators holding undergraduate degrees or higher, the authors' outputs received the highest preference for both audio-video consistency and overall quality.

• Both training stages are necessary. On AVC-Benchmark, the base model scores Δ-Pairwise 0.605, GRPO-only 0.630, SFT-only 0.679, and SFT+GRPO 0.728 (PLCC 0.494), showing that the five-stage reasoning supervision cold-starts the process and GRPO supplies human-preference alignment.

• Averaging the three levels beats any single level. S_mac only scores 0.688 Δ-Pairwise, S_temp only 0.717, S_attr only 0.715, while the mean (S_AVC) reaches 0.728, supporting the claim that the three dimensions are complementary.

Methodology in Plain English

The authors start from three properties of modern generated scenes: multiple events can co-occur; the soundtrack can mix physical sounds with narrative audio such as voice-over or music; and audio and visual events need not correspond one-to-one. These properties generate three design requirements — separate the events, distinguish the role of each audio event, and verify relations case by case — which an audio-video-capable MLLM can handle inside a single reasoning process.

The evaluation is organized as five stages. First the model listens to the audio alone and records a global description plus a set of audio events, each with a time span, category (human speech, music or rhythm, or other physical sound), a source role (physical or narrative), and acoustic attributes; physical events are additionally marked on-screen or off-screen where observable. Second it watches the video alone and records a global description plus visual events with timestamps, action category, and observable morphology, source location, and motion. Third it checks each audio event against the video, and fourth it checks each visual event against the audio; the second direction explicitly allows "reasonable silence" for naturally silent actions and treats a missing sound for a sound-producing action as an existence failure. Either side of a relation record can be empty, which is how asymmetric cases such as off-screen sounds are represented. The fifth stage converts these records into the three scores, moving from global semantics to event presence and timing, then to attributes only for pairs that passed temporal verification.

Training data comes from existing joint audio-video, audio-driven video, and video-to-audio models. Structured reasoning targets are produced by Gemini 2.5 Pro and Qwen3-VL-Max acting as complementary annotators, with a silence detector providing auxiliary temporal cues; the five stage records are concatenated and rewritten by Qwen3-235B into one SFT target, and after filtering approximately 8k samples are retained. For human alignment, annotators apply a 39-label taxonomy over the three dimensions; a multi-annotator subset with independent dimension scores is used to have Qwen3-235B map label records to the three scores plus a diagnostic conclusion, yielding approximately 10k human-calibrated samples.

Training runs in two stages: full fine-tuning on the structured reasoning data as an SFT cold start, then GRPO on a 10k rollout set. The GRPO reward combines a structure reward that enforces the five-stage format with a score distillation reward r_distill = (1/3) Σ exp[−w_i(pred_i − ref_i)²] over the three scores, weighted as r_total = (2 r_struct + r_distill) / 3.

For generation-side testing, the authors keep the OmniNFT framework and swap its DeSync reward for AVC-Reward's overall score, with reward-specific learning-rate and weight adjustments for comparable optimization strength. They deliberately avoid JavisBench because part of its prompt set was used to collect training samples, evaluating instead on AV-Phys-Bench, VABench-T2AV, and AVGen-Bench.

Why This Matters

Impact on research. The paper argues that evaluation methodology has not kept pace with joint generation and offers a reusable formulation rather than a task-specific metric. It also couples evaluation to optimization: the same model that scores a clip can be plugged into an online RL loop, closing what the authors call the "generate–evaluate–optimize" cycle. The accompanying benchmark, built primarily from model-generated rather than curated real data, targets reproducibility of human-alignment measurement.

Real-world applications:

• Automated quality gating and ranking for commercial text-to-video or joint AV generation services, where a scalar score alone is insufficient and

Authors’ abstract

As joint audio-video generation models become more capable, they can generate audio-visual scenes exhibiting three forms of complexity: multi-event composition, multi-layer audio mixture, and asymmetric cross-modal relations. However, current evaluations largely reuse metrics from earlier simple and unidirectional tasks. They typically evaluate an audio-video clip as a whole and rely on simple predefined relations, making them unsuitable for these increasingly complex scenes. We propose an event-centric evaluation framework that disentangles audio and visual events at the event level, analyzes each event independently, cross-validates relations across modalities, and aggregates the judgments into a coarse-to-fine, three-level consistency metric. The metric assesses global narrative harmony, event-level temporal synchronization, and fine-grained attribute alignment, providing a general formulation for audio-video consistency evaluation across diverse generation settings and cross-modal relations. We instantiate the metric as AVC-Reward, a trainable audio-video reward model. To evaluate the agreement between automatic judgments and human perception across generation paradigms and relation complexities, we construct AVC-Benchmark, which is primarily built from diverse positive and negative samples generated by various types of audio-video generation models. AVC-Reward achieves state-of-the-art alignment with human judgment among automatic methods, provides interpretable rationales that localize failures, and improves the audio-video consistency of a joint generation model when used as an online reinforcement-learning signal. Project page: https://tsuki0512.github.io/AVC-Reward.github.io/