Hear, Localize, and Reason: Spatially Aware Scene Understanding for Audio-Visual LLMs

AI summary

Overview

Research area: Computer Vision / audio-visual multimodal learning (spatial audio understanding with large language models).

Technical level: Advanced — assumes familiarity with multimodal LLM architectures, audio encoders, LoRA fine-tuning, and 3D acoustic simulation.

Scope: The paper introduces HLR-AVSceneQA, a simulator-built benchmark organized around a "hear–localize–reason" hierarchy, and HLR-LLM, a spatially aware audio-visual LLM that adds a binaural audio branch and chain-of-thought curriculum training to a Qwen2.5-Omni backbone.

What This Paper Is About

Most audio-visual LLMs treat audio as monaural semantic content — they know what is heard but not where it comes from, so they cannot reason about sources that are off-screen, occluded, in another room, or ambiguous among several candidates. The authors build a unified benchmark that tests hearing, egocentric and allocentric sound-source localization, and scene-level relational reasoning in one place, then build a model that uses binaural spatial cues to do all three while keeping its semantic audio-visual abilities intact.

Key Contributions

1. HLR-AVSceneQA benchmark. A 12-task benchmark spanning spatial hearing (sound event recognition, far-field ASR), spatial grounding (egocentric and allocentric direction and distance), and spatial relational reasoning (visibility, correspondence, hidden-source, loudness change, proximity, scene captioning). It contains 175,061 training and 21,808 test QA pairs, with 53,279 training and 4,869 test pairs drawn from scenes with multiple simultaneously active sound sources, plus 94,519 CoT-augmented training pairs. Localized source information is used both as a prediction target and as intermediate evidence for downstream reasoning.

2. HLR-LLM architecture. A spatially aware AV-LLM built on Qwen2.5-Omni that adds a frozen pretrained spatial audio encoder, a trainable MLP projector, and a new <spatial_audio> media token alongside the base visual and monaural audio pathways.

3. Three-stage curriculum with final-stage CoT tuning. Spatial audio alignment on 514k spatial audio-only QA samples, then spatial audio-visual instruction tuning on HLR-AVSceneQA (175k), then chain-of-thought tuning on 94.5k CoT pairs so the model produces intermediate rationales before a normalized answer.

4. Normal-field-of-view evaluation. Unlike panoramic-input prior work, the benchmark uses 90° horizontal FOV images at 1280×720, forcing models to reason about visible, occluded, and off-screen sources from partial visual evidence.

Main Findings

• Zero-shot AV-LLMs and commercial models are weak at grounding. Qwen2.5-Omni zero-shot reaches 17.33% egocentric direction and 7.10% egocentric distance (random chance 12.50% and 10.00%); Gemini-2.5-Flash-Lite reaches 19.67% and 12.23%; GPT-Realtime-Mini 12.65% and 14.55%. Sound event recognition is comparatively reasonable (e.g., 57.21% for Qwen2.5-Omni, 55.84% for Qwen3-Omni).

• Fine-tuning alone is not enough. Fine-tuned Qwen2.5-Omni rises to 32.69% egocentric direction and 30.92% egocentric distance, and fine-tuned Hear You Are LLM reaches 44.65% and 22.63%, but both stay far below HLR-LLM.

• Explicit binaural modeling beats separate left/right channels. HLR-LLM reaches 78.21% egocentric direction, 55.56% egocentric distance, 52.03% allocentric direction, and 45.08% allocentric distance, all above the dual-channel Qwen2.5-Omni baseline (64.27%, 46.59%, 42.71%, 37.84%).

• Best spatial relational reasoning across the board. HLR-LLM scores 80.51% visibility, 76.19% correspondence, 67.23% hidden source, 80.87% loudness, and 61.50% proximity — best on every one of these tasks versus all zero-shot and fine-tuned baselines (e.g., dual-channel Qwen2.5-Omni: 77.30%, 69.07%, 60.12%, 72.59%, 55.64%).

• Captioning is a mixed result. HLR-LLM has the best SPIDEr (3.109), but dual-channel Qwen2.5-Omni has the best CIDEr-D (5.414) and METEOR (0.444); HLR-LLM gets 4.461 and 0.439.

• Semantic hearing is preserved. HLR-LLM scores 81.03% sound event accuracy and 14.43% WER, comparable to fine-tuned Qwen2.5-Omni (80.57%, 15.33%) and dual-channel Qwen2.5-Omni (84.52%, 13.42%), which is slightly better on both.

• Vision and binaural audio are complementary (ablation). Removing vision drops allocentric direction from 78.21% to 21.51% and proximity from 61.50% to 34.13%, but egocentric direction only falls to 74.72%. A visual-only model collapses on grounding (31.20% egocentric direction). Removing binaural audio drops egocentric direction to 32.69% and distance to 30.92%.

• Final-stage CoT beats joint training. CoT after Stage-2 instruction tuning gives the strongest grounding (78.21/55.56/52.03/45.08) versus joint mixing of Stage-2 and CoT data (78.00/53.04/45.01/39.25), with the largest gaps on allocentric and multi-step tasks. Joint training does edge ahead on hidden-source (69.78% vs. 67.23%).

• Real-world transfer is suggested but the numbers are truncated. The paper reports a STARSS23 adaptation experiment covering sound event recognition, egocentric direction and distance, and visibility prediction, and states that HLR-AVSceneQA pretraining provides useful spatial priors — but the Table 6 values are cut off in the provided content, so specific accuracies are not reported here.

Methodology in Plain English

The authors could not collect enough real-world spatial audio with accurate annotations, so they simulated it. Using SoundSpaces 2.0 for room acoustics and HM3DSem for 3D indoor geometry (216 indoor spaces, 3,100 rooms; 145 scenes for training, 36 for testing), they placed an agent at an egocentric viewpoint that defines both a camera and a binaural microphone. They inserted 2 to 6 candidate object meshes per room from a pool of Objaverse-XL assets (roughly 81 meshes per category for training, 35 for testing), attached up to two sound sources, and rendered a synchronized 1280×720 image plus two-channel audio. Sound sources came from VGGSound (over 200,000 clips, 310+ classes, filtered for classes that cannot plausibly attach to an indoor object) for non-speech and LibriSpeech for speech.

Because the simulator knows everything — agent pose, object coordinates, source assignments, visibility, occlusion type, pairwise geometry — question-answer pairs can be generated with exact labels. Some audio-object pairs are deliberately matched in space but mismatched in semantics, and some sources are hidden, occluded, or in an adjacent room, so models cannot succeed by category matching alone. Answer formats are mostly constrained to multiple choice or discretized bins. Chain-of-thought rationales are generated deterministically from the metadata templates, with GPT-4 used only to paraphrase question wording.

For the model, binaural audio goes through a frozen spatial audio encoder from BAT and a trainable MLP projector into the LLM's embedding space, while the channel-averaged version feeds the original monaural pathway. Training is a three-stage curriculum: align the spatial projector on 514k spatial audio-only QA samples, instruction-tune on HLR-AVSceneQA, then tune on CoT answers. Only the projector and LoRA parameters (rank 16, alpha 16, dropout 0.05, learning rate 2×10⁻⁴, weight decay 0.01) are updated; all encoders stay frozen. Training runs 2 epochs on four NVIDIA A6000 GPUs with per-device batch size 4, using 32 kHz audio, window 1024, hop 320, and 128 mel bins.

Why This Matters

The paper reframes spatial audio as evidence for reasoning rather than a separate perception task. Rather than adding another narrow direction-estimation benchmark, it evaluates whether a localized source can be used — to answer whether something is hidden, which object it aligns with, or whether it gets louder as you move. It also shows that bolting on left/right channel tokens is not the same as modeling binaural space.

Real-world applications:

• Service and assistive robots that must locate a person calling from another room or a hidden alarm and navigate toward it.
• Smart home and security devices that fuse microphone arrays with cameras to identify where an anomalous sound came from.
• AR/VR and telepresence, where spatial audio drives attention cues and scene understanding.
• Accessibility tools that describe off-screen or occluded sound events to a user with limited vision.

Industry relevance: Any product combining microphones and cameras — robotics, automotive cabin monitoring, smart speakers, wearables — needs models that generalize from "what" to "where." The benchmark and the finding that pretraining on it transfers to real recordings (STARSS23) is directly relevant to teams building spatially aware assistants. The code and benchmark are released at https://hlrllm.github.io/.

Future Directions

• Fully report and expand real-world transfer. The STARSS23 adaptation is limited to sound event, egocentric direction/distance, and visibility because object- and scene-level annotations are missing; extending real-world data to support hidden-source, allocentric, and relational tasks is the obvious next step.
• Close the allocentric gap. Allocentric grounding remains the weakest spatial task (52.03% direction, 45.08% distance), far below egocentric performance — better viewpoint transformation and 3D scene representation could help.
• Recover the captioning trade-off. HLR-LLM wins on SPIDEr but loses CIDEr-D and METEOR to the simpler dual-channel baseline, suggesting open-ended description generation is not yet fully served by the spatial branch.
• Beyond the simulator. All training data is synthetic; how much simulator-specific acoustic and layout bias remains, and whether scaling to real captured binaural data or more diverse room types changes the picture, is unresolved.

Target Audience

Researchers and graduate students working on audio-visual LLMs, spatial audio, and embodied or egocentric perception; benchmark designers interested in hierarchical evaluation that links perception to reasoning; and applied engineers building robots, AR/VR systems, or smart devices that must localize sounds in a visual scene. Readers need comfort with multimodal transformer architectures, audio spectrogram pipelines, and LoRA fine-tuning to get the most from the method section.

Authors’ abstract

Audio-visual large language models (AV-LLMs) have made strong progress in multimodal understanding, yet most treat audio as monaural semantic content, limiting their ability to reason about where sounds originate and how they relate to the visual scene. Recent spatial audio-visual studies address parts of this problem, but often focus on specific abilities, such as spatial correspondence or direction and distance reasoning. In this work, we introduce HLR-AVSceneQA, a comprehensive benchmark with a hear-localize-reason hierarchy that evaluates whether models can recognize what is heard, localize sound sources in egocentric and allocentric views, and use these localized sources to reason about how they relate to the visual scene. We further introduce HLR-LLM, which augments a strong audio-visual foundation model with a binaural spatial audio branch alongside its monaural semantic audio pathway. We train HLR-LLM with a curriculum that incorporates final-stage chain-of-thought supervision for multi-step spatial reasoning. Experiments show that existing AV-LLMs remain limited in spatial grounding and relational reasoning, while HLR-LLM substantially improves spatial audio-visual scene understanding without compromising semantic audio-visual perception.