Query-Guided Spatial-Temporal-Frequency Interaction for Music Audio-Visual Question Answering

AI summary

Overview

Research area: Multimodal machine learning — specifically Audio–Visual Question Answering (AVQA) applied to music scenes, combining computer vision, audio signal processing, and natural language processing.

Technical level: Advanced. The paper assumes familiarity with transformer attention mechanisms, cross-modal fusion, spectrogram-based audio representations, and multimodal QA benchmarks.

Scope: The paper proposes QSTar, a question-guided framework that fuses spatial, temporal, and frequency-domain cues from video and audio to answer natural-language questions about musical performances, evaluated on the MUSIC-AVQA and AVQA benchmarks.

What This Paper Is About

Audio–Visual Question Answering asks a model to watch and listen to a video, then answer a question such as "How many instruments are sounding?" or "Is the clarinet louder than the bassoon?" Existing AVQA systems lean heavily on visual processing — detecting objects and motion — and treat audio as a supporting signal, while the text of the question is usually injected only at the very end of the pipeline through a simple operation such as multiplication. That design fails when the crucial evidence is auditory (a flute player barely moves but still sounds) or when several instruments play at once and must be told apart by their timbre.

The goal of this paper is to build a model in which the question guides the extraction of both audio and visual features from the start, and in which the audio is analyzed not only over time but also across frequency bands — the spectral "fingerprint" that distinguishes a clarinet from a bassoon even when their time-domain waveforms or pitches look similar.

Key Contributions

1. QSTar framework with query guidance throughout the pipeline. A Query-Guided Multimodal Correlation (QGMC) module refines audio and visual features conditioned on the question in three stages — self-enhancing, capturing, and propagating — rather than only fusing the question at the final prediction step.

2. A spatial–temporal–frequency interaction (STFI) module. Spatial–Temporal Interaction (STI) aligns patch-level visual features with query-guided audio, while Temporal–Frequency Interaction (TFI) uses features from a pretrained Audio Spectrogram Transformer (AST) plus a frequency-wise attention mechanism to emphasize question-relevant spectral bands.

3. A Query Context Reasoning (QCR) block. Inspired by prompting, this block encodes context keywords tied to five aspects of music questions — type, duration, location, sequence, and loudness — and uses them, together with the sentence-level question embedding, to guide final multimodal fusion.

4. State-of-the-art results across AVQA benchmarks. QSTar reaches 78.98% average accuracy on the MUSIC-AVQA test set and 91.2% total accuracy on AVQA when prompting is used, with ablations isolating the contribution of each module and of query guidance at each stage.

Main Findings

• Overall performance on MUSIC-AVQA: QSTar achieves 78.98% average accuracy, outperforming TSPM and QA-TIGER by 2.19% and 1.36% respectively. For comparison, the paper reports TSPM at 76.79%, QA-TIGER at 77.62%, and APL at 74.53%.

• Per question-type breakdown: QSTar scores 80.63% on Audio QA, 84.17% on Visual QA, and 75.98% on Audio-Visual QA. It outperforms QA-TIGER by 2.05% on Audio QA and 2.24% on Audio-Visual QA, and trails QA-TIGER by only 0.97% on Visual QA, despite not using pre-trained object detectors or dedicated visual perception modules.

• Largest gains in comparative and temporal questions: The paper reports improvements exceeding 5% on comparative and temporal question types.

• Module ablations: Removing all modules and keeping only simple multimodal fusion drops performance by over 5% (to 73.29% average). Removing QGMC yields 76.80%, removing QCR yields 78.19%, removing STI yields 77.80%, removing TFI yields 77.41%, and removing the combined STFI yields 76.62%, versus 78.98% for the full model.

• Frequency reasoning matters most for audio: Eliminating the TFI module causes a 2.42% decrease in Audio QA and a 1.59% decrease in Audio-Visual QA performance. Removing STI drops overall accuracy by 1.18% and Visual QA accuracy by 1.55%.

• Query guidance helps at every stage: Removing early-stage guidance drops average accuracy by 1.05% (to 77.93%); removing the question embeddings in TFI drops it by 0.43% (to 78.55%); removing final-stage prompting drops it by 0.73% (to 78.25%).

• Frequency-aware audio features outperform VGGish-style features: Removing the AST features from the STFI module lowers average accuracy by 1.37% (to 77.61%), with the largest degradation on audio-related questions.

• Patch-level visual features are critical for spatial perception: Removing patch-level visual features (F_p) from STI causes a 1.43% drop in Visual QA and a 0.9% drop in Audio-Visual QA.

• Early question guidance beats late fusion: On the comparison of early-stage feature-processing strategies, direct audio–visual fusion scores 77.52% average, separate cross-attention scores 78.30%, dual-branch sequential cross-attention (as in QA-TIGER) scores 78.89%, and the proposed QGMC scores 78.98%. All early-guidance variants beat direct fusion by at least 0.8%, and the sequential cross-attention variant holds only a 0.08% edge over QGMC on the Audio-Visual QA type.

• Results on the AVQA dataset: QSTar achieves 90.9% total accuracy without prompting and 91.2% with prompting, compared with PSTP at 90.2%, TSPM with prompting at 90.8%, HCRN at 89.0%, and PSAC at 87.4%.

• Qualitative behavior: In the MUSIC-AVQA test examples, QSTar correctly predicts "Four" even when the cello is not consistently visible by using frequency-enhanced audio cues, and correctly distinguishes two guzhengs. The STFI module identifies that the clarinet continues playing while the bassoon stops at the middle timestamp.

Methodology in Plain English

The video is chopped into non-overlapping one-second audio and visual segments. Visual segments are encoded with a frozen CLIP-ViT-L/14 model to produce both frame-level and patch-level features; the patch features are compressed with Token Merging (ToMe). Audio is encoded two ways: a 2D CNN called VGGish (pretrained on AudioSet) gives general audio features, and a pretrained Audio Spectrogram Transformer (AST) gives time–frequency features. The question is tokenized and encoded with CLIP's text encoder into a sentence-level vector and word-level token embeddings. All features are projected to 512 dimensions.

The QGMC module first applies self-attention separately to the visual, audio, and word-level text features. Then the self-enhanced words act as the query in cross-attention over visual and audio features, capturing shared semantics; the results are summed with the text features to form a query-guided context, which is propagated back into the visual and audio streams via cross-attention and a feed-forward network with residual connections.

The STFI module then works in two directions. Spatially, patch-level visual features attend to the query-guided audio features to focus on sounding regions. Temporally, the query-guided audio and visual features are combined through a dot-product attention over time. The two outputs are concatenated and passed through a feed-forward network. On the audio side, AST features are averaged over time into a condensed frequency representation; this is combined with the question embedding through learnable projections to produce frequency-attention weights, which are broadcast over the AST features to emphasize relevant spectral bands. The attention-weighted AST features and the query-guided audio features are then fused with a two-layer convolutional block.

Finally, the QCR block builds "prompt" embeddings from context keywords covering instrument type, performance duration, location, temporal sequence, and loudness. These are concatenated with the sentence-level question embedding and passed through self-attention to form a query-context feature, which cross-attends over both the refined visual and refined audio features. The two outputs are concatenated, passed through a linear layer with tanh activation, and multiplied element-wise with the sentence embedding to predict the answer from a predefined vocabulary.

Training used the AdamW optimizer with an initial learning rate of 1e-4, decayed by a factor of 0.1 every 10 epochs, a batch size of 64, and 30 epochs on a single NVIDIA H100 GPU.

Why This Matters

Impact on research: The paper challenges the visual-centric assumption that dominates AVQA research, showing that question-aware refinement of audio from the outset — plus explicit frequency-domain modeling — produces measurable gains without object detectors or bespoke visual perception modules. The reported 1.37% average-accuracy drop when AST features are removed supports the claim that spectrogram-based representations carry information that VGGish-style features do not.

Real-world applications:

• Music education and practice tools that identify which instrument is playing, when it starts and stops, and how loud it is relative to others.
• Automatic indexing and captioning of concert or rehearsal recordings for archives and streaming platforms.
• Assistive technology for hearing-impaired users that surfaces non-speech audio events and their sources in video.
• Multimodal video assistants and interactive search systems that answer natural-language questions about audio-rich footage.

The paper also lists autonomous driving, human–computer interaction, and event localization as motivating application areas for audio–visual scene understanding more broadly.

Industry relevance: Music streaming, live-event production, and video platform companies deal with large volumes of multi-instrument recordings where machine-readable metadata (instrument type, timing, loudness) is valuable. The framework's reliance on off-the-shelf frozen encoders (CLIP, VGGish, AST) rather than task-specific detectors lowers the barrier to adoption. The authors state that code and pretrained models will be released after publication, which is not yet the case at the time of the paper.

Future Directions

• Better spatial localization of instruments. QSTar trails the previous state of the art on Visual QA by 0.97%, and the authors explicitly frame this as motivation for future work on localizing performing instruments.
• Extending beyond music to general real-world scenes. The paper evaluates on AVQA in the supplementary material and describes its scenarios as more diverse than music-related events, but its design is motivated by musical instruments; broader generalization is left open.
• Refining the prompting mechanism. The appendix notes that additional studies on prompting are provided in Supplementary Section A.2.3, and the ablation shows final-stage prompting still contributes a 0.73% gain, suggesting room for stronger context design.
• Scaling to larger and more varied benchmark data. The main experiments use MUSIC-AVQA (more than 40K QA pairs across 9,288 videos) and AVQA (over 57K QA pairs); the paper does not report results on other AVQA datasets such as Pano-AVQA, leaving cross-dataset generalization unverified.

Target Audience

Researchers and practitioners in multimodal learning and audio–visual understanding, particularly those working on AVQA, video QA, or music information retrieval. The paper is most useful to readers already comfortable with transformer attention mechanisms and spectrogram-based audio representations, since the method sections are written at an advanced technical level. It is also relevant to engineers building question-answering or indexing systems for audio-rich video, and to students studying how query conditioning can be applied at multiple stages of a multimodal pipeline rather than only at the final prediction step.

Authors’ abstract

Audio--Visual Question Answering (AVQA) is a challenging multimodal task that requires jointly reasoning over audio, visual, and textual information in a given video to answer natural language questions. Inspired by recent advances in Video QA, many existing AVQA approaches primarily focus on visual information processing, leveraging pre-trained models to extract object-level and motion-level representations. However, in those methods, the audio input is primarily treated as complementary to video analysis, and the textual question information contributes minimally to audio--visual understanding, as it is typically integrated only in the final stages of reasoning. To address these limitations, we propose a novel Query-guided Spatial--Temporal--Frequency (QSTar) interaction method, which effectively incorporates question-guided clues and exploits the distinctive frequency-domain characteristics of audio signals, alongside spatial and temporal perception, to enhance audio--visual understanding. Furthermore, we introduce a Query Context Reasoning (QCR) block inspired by prompting, which guides the model to focus more precisely on semantically relevant audio and visual features. Extensive experiments conducted on several AVQA benchmarks demonstrate the effectiveness of our proposed method, achieving significant performance improvements over existing Audio QA, Visual QA, Video QA, and AVQA approaches. The code and pretrained models will be released after publication.