R-AVST: Empowering Video-LLMs with Fine-Grained Spatio-Temporal Reasoning in Complex Audio-Visual Scenarios

AI summary

Overview

• Research area: Multimodal video understanding — specifically audio-visual spatio-temporal reasoning with Video-LLMs (video-capable multimodal large language models).
• Technical level: Advanced. The paper assumes familiarity with reinforcement learning (GRPO/PPO), grounding metrics (tIoU, vIoU, AP), and Video-LLM architectures.
• Scope: The paper introduces R-AVST, the first video dataset with fine-grained spatio-temporal annotations in complex audio-visual scenarios, defines three reasoning tasks over it, and proposes AVST-Zero, a GRPO-trained model that improves performance on those tasks.

What This Paper Is About

Existing video datasets and models handle either audio-visual events (with temporal boundaries only) or visual spatio-temporal grounding (with temporal and spatial boxes but no audio), but not both together. R-AVST fills that gap by pairing untrimmed audio-visual videos with frame-level object boxes and event time spans, then defining three question-answering tasks — temporal, spatial, and spatio-temporal — over sounding and silent objects. The authors also train AVST-Zero to improve a Video-LLM's accuracy on those tasks using reinforcement learning with task-specific rewards rather than large-scale supervised labels.

Key Contributions

1. A new dataset. R-AVST is presented as the first video dataset with fine-grained spatio-temporal annotations in complex audio-visual scenarios. It contains 5,237 videos, 27,253 objects across 100 types of audio-visual events, and 8,166 question-answer pairs, built from untrimmed videos sourced from UnAV-100.
2. Three reasoning tasks with automatically constructed QAs. The paper defines Audio-Visual Temporal Reasoning, Audio-Visual Spatial Reasoning (covering both sounding-visible and silent-visible objects), and Audio-Visual Spatio-Temporal Reasoning, with QAs generated from LLM-derived object labels.
3. AVST-Zero. A Video-LLM fine-tuned entirely with GRPO and a multi-dimensional reward system (format, object, temporal, spatial) that requires no intermediate supervision and directly optimizes behavior.
4. Empirical validation. Experiments over the R-AVST test set plus cross-dataset tests on AVE and AVSBench-V1, showing competitive or leading results on the defined tasks.

Main Findings

• Dataset scale and composition: R-AVST contains 5,237 videos of 100 categories totaling over 220,833 seconds, averaging 42.17 seconds per video, with an average of 5.2 objects per video. Of the 27,253 annotated objects, 50.88% are labeled "visible&audible." The train/test split is 4,171/1,066 videos with 6,533/1,633 QAs.
• QA distribution: The training set holds 2,663 temporal, 2,666 spatial, and 1,204 spatio-temporal questions; the test set holds 663, 664, and 306 respectively (totals of 3,326, 3,330, and 1,510).
• Temporal reasoning win: AVST-Zero leads on the Audio-Visual Temporal Reasoning task with an m_tIoU of 47.96%, R1@0.3 of 71.13%, R1@0.5 of 51.43%, and R1@0.7 of 23.91%, ahead of VideoChat-R1 (43.17% m_tIoU) and Qwen2.5-VL (36.05% m_tIoU).
• Spatio-temporal reasoning gain: On the Audio-Visual Spatio-Temporal Reasoning task, AVST-Zero reaches 46.04% m_tIoU and 8.59% m_vIoU, against 41.81% m_tIoU and 2.15% m_vIoU for VideoChat-R1. AVST-Zero-Omni reaches 17.74% m_vIoU and 22.90% AP@0.3 on the same task.
• Trade-off between variants: AVST-Zero-Omni (built on Qwen2.5-Omni 7B) achieves higher object and spatial accuracy than AVST-Zero (object accuracy 19.48 vs. 14.34; m_vIoU 3.87 vs. 2.27 on spatial reasoning) but performs worse on the temporal dimension — attributed to the base model's strong audio-visual joint perception but weaker temporal perception.
• RL beats SFT: On a balanced test subset, supervised fine-tuning reaches 42.84% m_tIoU (temporal task), 3.42% m_vIoU (spatial task), and 38.40% m_tIoU / 4.26% m_vIoU (spatio-temporal task), while AVST-Zero reaches 48.17%, 4.62%, 46.93%, and 10.87% respectively.
• Reward components matter: Removing the temporal reward drops temporal accuracy from 48.17% to 46.67%; removing the spatial reward drops m_vIoU from 4.62% to 3.28% (spatial) and from 10.87% to 9.23% (spatio-temporal). The authors note cross-effects between reward modules due to the interdependence of the spatio-temporal dimension.
• Cross-dataset transfer: On AVE, AVST-Zero reaches 38.15 m_tIoU versus 40.81 for VideoChat-R1, 29.49 for Qwen2.5-VL, and 18.37 for Video-R1. On AVSBench-V1, AVST-Zero reaches the best m_vIoU at 6.84, ahead of Qwen2.5-VL (3.79), VideoChat-R1 (2.98), and Video-R1 (1.98).
• Human quality check: Two human raters scored test data on a 4-point scale with Quadratic Weighted Cohen's k of 0.71; 1,066 videos were filtered from an original 2,131 for the final test set. Mean score rose from 2.54 on the original test set to 2.93 on the curated one.
• Qualitative behavior: Qwen2.5-VL produces sparse and inaccurate object locations, and VideoChat-R1 correctly identifies the girl in the shown example, but AVST-Zero's predictions are closer to ground truth in object recognition and spatial localization.

Methodology in Plain English

The pipeline starts from untrimmed YouTube videos and event captions in UnAV-100. A three-step filter balances duration (grouped into 0-20s, 20-40s, and 40-60s), caps videos at three audio-visual events, and removes videos whose event-to-total duration ratio falls below 0.08.

Caption analysis then uses GPT-4o-mini as an analyzer LLM to extract noun-based objects and label each as visible-only or visible&audible (the prompt explicitly discourages assigning "audible-only" to vague environmental sounds). Object bounding boxes are produced automatically by Grounded-SAM2, frame by frame within event segments, using object names as text prompts, with a box threshold of 0.4 and a text threshold of 0.3; results are stored as structured JSON. Programs then convert these annotations into the three question types (when, where, what), and humans verify annotations and QAs, discarding bad samples.

The model side uses GRPO, which samples a group of responses per question and estimates relative advantages from rule-based rewards without a critic model. Four rewards are combined: a format reward checking tag pairs (<answer>, <object>, <when>, <where>); an object reward checking Word2Vec cosine similarity against a threshold; a temporal reward equal to the IoU of predicted and ground-truth time intervals; and a spatial reward equal to the mean 2D IoU over time steps in the overlapping interval. The total reward is a weighted sum, with the format weight fixed at 1 and the other weights set per task. Training uses Qwen2.5-VL 7B and Qwen2.5-Omni 7B as bases, four NVIDIA RTX A6000 GPUs, one epoch, batch size 1 per device, and a group generation number of 6.

Why This Matters

• Research impact: The paper argues that existing datasets either lack audio-visual content in spatio-temporal grounding benchmarks (VidSTG, HC-STVG, V-STaR) or lack spatial annotations in audio-visual benchmarks (AVE, UnAV-100, PU-VALOR). R-AVST combines both, and the authors position it as the first dataset designed for real-world audio-visual spatio-temporal reasoning, offering a new benchmark direction.
• Human-computer interaction: Localizing which sounding object is visible and where it is at a given moment supports voice-driven interfaces and assistive systems.
• Autonomous driving: Distinguishing audible and visible agents (e.g., a sounding siren versus a silent vehicle) in time and space is directly relevant to perception stacks.
• Video retrieval and surveillance: Searching untrimmed footage by "what made this sound, when, and where" instead of purely visual cues.
• Industry relevance: The work was supported by the National Key Research and Development Program of China (Grant 2024YFE0203100) and ZTE Industry-University-Institute Cooperation Funds (Grant No. IA20240906004), with authors from Southern University of Science and Technology, University of Birmingham, The University of Hong Kong, ZTE Corporation, and Spatialtemporal AI — indicating direct industrial interest in video understanding infrastructure.

Future Directions

• Extending beyond three-event videos: The current filtering caps videos at three audio-visual events, so scaling to denser, longer real-world footage remains open.
• Closing the temporal-spatial gap between model variants: AVST-Zero-Omni's stronger object and spatial accuracy but weaker temporal accuracy suggests that a base model combining both strengths has not yet been identified.
• Improving reward design: The ablation shows cross-effects between reward modules, and the paper notes that existing RL models lack reward designs specifically for spatio-temporal audio-visual reasoning — a signal that better multi-dimensional reward shaping is an open problem.
• Broadening evaluation beyond two external datasets: Cross-dataset validation is currently limited to AVE (temporal) and AVSBench-V1 (spatial), leaving other audio-visual and spatio-temporal grounding benchmarks untested.

Target Audience

Researchers and engineers working on multimodal large language models, video understanding, and audio-visual grounding; benchmark builders evaluating spatio-temporal reasoning; and practitioners in video retrieval, human-computer interaction, or autonomous perception who need models to jointly reason over sound and sight. Readers without background in reinforcement learning or video grounding metrics will find the model and evaluation sections demanding, while the dataset and task definitions are more accessible.

Authors’ abstract

Recently, rapid advancements have been made in multimodal large language models (MLLMs), especially in video understanding tasks. However, current research focuses on simple video scenarios, failing to reflect the complex and diverse nature of real-world audio-visual events in videos. To bridge this gap, we firstly introduce R-AVST, a dataset for audio-visual reasoning featuring fine-grained spatio-temporal annotations. In constructing this, we design a pipeline consisting of LLM-based key object extraction, automatic spatial annotation and manual quality inspection, resulting in over 5K untrimmed videos with 27K objects across 100 types of audio-visual events. Building on this dataset, we define three core tasks for spatio-temporal reasoning in audio-visual scenes and generate more than 8K high-quality, evenly distributed question-answer pairs to effectively benchmark model performance. To further enhance reasoning, we propose AVST-Zero, a reinforcement learning-based model that avoids intermediate supervision, directly optimizing behavior via carefully designed multi-dimensional rewards. Extensive experiments validate the effectiveness of our R-AVST in advancing audio-visual spatio-temporal reasoning, upon which AVST-Zero demonstrates competitive performance compared to existing models. To the best of our knowledge, R-AVST is the first dataset designed for real-world audio-visual spatio-temporal reasoning, and AVST-Zero offers a novel perspective for tackling future challenges in this domain.