Skip to content
AI.info

Research

EVA-Cap: Optimizing Audiovisual Video Captioning via Event-Centric Alignment

Existing audiovisual captioners are trained with caption-level supervision, treating a caption as one unstructured block of text. This means a caption can read fluently while still omitting key…

EVA-Cap: Optimizing Audiovisual Video Captioning via Event-Centric Alignment
arXiv
2610.22186
Published
2026-10-15
Authors
Jiapeng Shi, Qiuxia Hou, Jiaqi Leng, Junke Wang, Yanhao Zhang, Haonan Lu, Ziyi Ye, Zuxuan Wu

AI summary

Overview

Research area: Computer vision / multimodal video understanding — specifically audiovisual video captioning with Omni Language Models (OLMs), combined with reinforcement-learning post-training.

Technical level: Advanced. The paper assumes familiarity with supervised fine-tuning, Group Relative Policy Optimization (GRPO), reward modeling, and multimodal encoder–LLM architectures.

Scope: The paper proposes EVA-Cap, a Qwen2.5-Omni-based captioning framework that decomposes audiovisual captions into atomic events and optimizes caption generation against three event-level rewards, supported by a 63K self-curated dataset (EVA-Data).

What This Paper Is About

Existing audiovisual captioners are trained with caption-level supervision, treating a caption as one unstructured block of text. This means a caption can read fluently while still omitting key events, attributing speech or actions to the wrong person, or mismatching audio events with the visual moment they belong to. EVA-Cap reframes the task as assembling a sequence of discrete, modality-grounded events (each with a type, character, and content), and trains the model to be rewarded for getting those events, their attributions, and their temporal correspondences right.

Key Contributions

  1. An event-centric formulation of audiovisual captioning. Captions are represented as sequences of events e_k = (m_k, τ_k, c_k, d_k) — modality, event type, character, and content — so that a caption can be scored on intra-event attribution and inter-event synchronization rather than as a single holistic unit.
  2. EVA-Data, a 63K audiovisual video–caption dataset. Built from 100K source videos (each under 200 seconds) drawn from LLaVA-Video, FineVideo, and Shot2Story, using a three-stage automatic pipeline (initialization with AVoCaDO, refinement with Gemini-2.5-Pro, and event-centric quality control with Gemini-2.5-Pro event extraction plus GPT-4.1 verification, discarding samples with an event hit ratio below 0.8).
  3. A two-stage post-training recipe with three event-centric rewards. SFT on EVA-Data followed by two rounds of GRPO using a global event completeness reward (event recall), an intra-event attribution reward (character matching, restricted to events with an explicit character), and an inter-event synchronization reward (audio-to-visual retrieval scored by set-level F1).
  4. A progressive reward schedule. Round one uses only the completeness and synchronization rewards (λ_comp = λ_sync = 1, λ_attr = 0); round two adds attribution (all three coefficients set to 1), moving from coarse global alignment to fine-grained character-level attribution.

Main Findings

  • General captioning: EVA-Cap reaches a total error rate of 34.6% on the video-SALMONN-2 testset (17.6% miss, 17.0% hallucination) and an average score of 76.9 on UGC-VideoCap (76.9 audio, 79.0 visual, 74.7 detail). Among the open models in Table 1 this is the best reported total error rate, though the proprietary Gemini-2.5-Pro (31.3) and Gemini-2.5-Flash (33.3) show lower totals.
  • Lowest miss rate: EVA-Cap's 17.6% miss rate is the lowest in Table 1. The paper argues this matters because low-hallucination baselines like HumanOmniV2 achieve 12.3% hallucination only at the cost of a 49.2% miss rate.
  • QA-cascaded benchmarks: EVA-Cap scores 56.1 on Daily-Omni and 29.0 on WorldSense, which the paper reports as gains of +3.3% on Daily-Omni and +3.3% on WorldSense (the abstract cites the +3.3% figure for WorldSense).
  • Event-centric alignment: On ChronusAV, EVA-Cap reports 55.4% V2A accuracy and 56.2% A2V accuracy, with METEOR of 2.4 and 1.2 respectively; on the AV-SpeakerBench Speaker Recognition subset it reaches 40.1% accuracy. The abstract describes this as a +9.0% average gain on ChronusAV.
  • SFT alone gives the largest single jump: V2A accuracy rises from 8.4% (baseline Qwen2.5-Omni) to 44.8%, A2V from 11.9% to 46.1%, AV-Speaker from 3.1% to 31.8%, and Daily-Omni from 12.4% to 50.1%. The paper attributes the weak baseline to a modality bias in which audio content is frequently omitted.
  • Each reward contributes measurably: Adding only ℛ_comp lifts V2A accuracy by 8.5% (44.8 → 53.3), AV-Speaker by 2.1% (31.8 → 33.9), and Daily-Omni by 3.7% (50.1 → 53.8). Adding ℛ_sync then yields +2.1% on A2V (52.9 → 55.0) and +3.1% on AV-Speaker (33.9 → 37.0).
  • Order of rewards matters: Injecting ℛ_attr in the first GRPO round degrades results (V2A 47.9, A2V 50.8, AV-Speaker 35.6, Daily-Omni 52.9) relative to using only completeness and synchronization in round one. The authors hypothesize an optimization conflict between inter-event relations and intra-event accuracy, and argue the progressive schedule resolves it.
  • Gains are not just extra data: A control setting that keeps only ℛ_comp and ℛ_sync in the second round performs below the full three-reward stage-2 setting (V2A 56.5 vs 55.4, but A2V 55.5 vs 56.2, AV-Speaker 37.0 vs 40.1, Daily-Omni 55.1 vs 56.1), which the authors use to argue the improvements stem from reward design.

Methodology in Plain English

The team starts with an off-the-shelf omni model, Qwen2.5-Omni-7B, and keeps its vision and audio encoders frozen while training only the adapters and the language model.

Building the data. Because high-quality audiovisual captions with event-level fidelity are scarce, they generate their own. A capable existing captioner (AVoCaDO) writes an initial description of each video. That draft, plus the raw video, goes to Gemini-2.5-Pro, which is instructed to tighten audiovisual synchronization and fix who did or said what. Finally, Gemini-2.5-Pro extracts timestamped event tuples directly from the video, and GPT-4.1 checks whether each tuple is actually "hit" in the refined caption — meaning both the audio and visual sides are entailed with consistent timing. Samples scoring below a 0.8 hit ratio are thrown out.

Stage one: supervised fine-tuning. The model learns to produce long, detail-rich audiovisual captions from the 63K curated pairs, using standard next-token prediction. SFT: two epochs, batch size 128, learning rate 1×10⁻⁵.

Stage two: reinforcement learning with event-level rewards. A subset of 3.2K rollout videos is used for GRPO. For 6.4K videos that achieved a 100% hit ratio, the team precomputes reference structures: a set of audio and visual events, and a set of "synchronization queries" where each audio event is paired with a candidate set of 5 visual events (chosen around the true match to create hard negatives; audio events spanning more than 5 visual-event indices are dropped as too coarse).

Three rewards are then computed per generated caption, judged by Qwen3.5-27B:

  • Completeness — what fraction of reference audio and visual events the caption actually covers (event recall).
  • Attribution — among events that have an explicit character, what fraction are covered and assigned to the same character as the reference caption.
  • Synchronization — for each audio event, the judge retrieves which candidate visual events the caption says are simultaneous; predicted and reference index sets are compared with set-level F1 (both-empty counts as 1, one-empty or disjoint counts as 0).

These are combined with balancing coefficients into a single reward, and GRPO normalizes each reward within a group of 8 sampled outputs to compute advantages. GRPO: two epochs, batch size 16, learning rate 2×10⁻⁶, KL penalty 0.04, group size 8. Training runs on 32 NVIDIA H20 GPUs; video is sampled at 2 FPS with a maximum generation length of 2048.

Why This Matters

Impact on research. The paper argues that the bottleneck in audiovisual captioning is not model capacity but the representation of supervision — that caption-level rewards leave structured video semantics unmodeled. Framing rewards as event recall, event attribution, and cross-modal retrieval gives a reusable template for turning structured annotations into RL signals, and the ablation showing that reward ordering matters is a concrete finding for anyone doing multi-objective GRPO post-training.

Real-world applications:

  • Video accessibility and description — more faithful audio descriptions that do not drop dialogue or misassign speakers.
  • Video retrieval and search — captions that preserve who said what and when make event-level indexing feasible.
  • Video question answering and reasoning — the QA-cascaded results on Daily-Omni and WorldSense show caption quality directly gates downstream reasoning accuracy.
  • Content moderation and media monitoring — reliable speaker attribution and audio–visual synchronization support claims about who did what on screen.

Industry relevance. The work comes from Fudan University and OPPO AI Center, and trains a 7B model on 32 H20 GPUs — a realistic budget for a production team. Its result that a 7B event-trained captioner is competitive with much larger omni models (Qwen3-Omni-Instruct and Qwen3-Omni-Captioner at 30B-A3B) on several benchmarks is the practically interesting claim.

Future Directions

  • Fixing the attribution reward's instability. The ablation shows that adding ℛ_attr early actively hurts, and the authors offer only a hypothesis (conflict between inter-event and intra-event objectives) rather than a diagnosis. Understanding and removing that conflict is an open problem.
  • Extending beyond 200 seconds. The dataset caps source videos at under 200 seconds and averages 73.7 seconds per video; whether event-centric rewards scale to genuinely long-form content is untested here.
  • Reducing dependence on proprietary judges. Data construction relies on Gemini-2.5-Pro and GPT-4.1, and reward computation relies on Qwen3.5-27B as judge. Cost, latency, and judge-model bias in the reward signal are not analyzed.
  • Closing the gap to proprietary models. On the video-SALMONN-2 testset EVA-Cap's 34.6% total error remains above Gemini-2.5-Pro's 31.3%, and its 17.0% hallucination rate is higher than several baselines — so how event-centric training trades against hallucination is an open question.

Target Audience

Researchers and engineers working on multimodal LLMs, video captioning, and reinforcement-learning post-training for generation tasks. It is most useful to readers already comfortable with SFT/GRPO pipelines who want a concrete design for converting structured, verifiable annotations into reward functions — and to practitioners deciding whether a 7B omni model with better supervision can substitute for a much larger one.

Authors’ abstract

Recent advances in Omni Language Models (OLMs) provide a unified solution for audiovisual video captioning. Nevertheless, existing approaches predominantly treat captioning as an unstructured text generation process, neglecting the inherent semantic structure of videos and leading to suboptimal fine-grained audiovisual alignment. To bridge this gap, we propose EVA-Cap, a novel framework that decomposes captions into atomic audiovisual events and transforms unstructured text…

The full abstract on arXiv
Read the original paper