OmniEcho: Audio-Visual Spatial Understanding for Omni-Modal Embodied Agents

AI summary

Overview

Research area: Embodied AI at the intersection of spatial audio processing, omni-modal (audio-vision-language) foundation models, and vision-language navigation (VLN).

Technical level: Advanced. The paper assumes familiarity with multimodal LLM architectures, first-order ambisonics (FOA) audio encoding, spherical harmonics, and navigation benchmarks such as R2R, RxR, and VLN-CE.

Scope: The paper introduces OmniEchoBench (a real-world benchmark for spatial audio-visual perception and sound-guided navigation) and OmniEcho (an omni-modal model built on Qwen3-Omni-30B-A3B that adds a first-order ambisonics encoder and a three-stage training procedure to a frozen pretrained semantic audio pathway).

What This Paper Is About

Embodied agents today rely mainly on vision, language, and semantic audio, and largely ignore spatial audio — the directional information in sound that lets humans locate an occluded or off-screen sound source. This paper asks how to both evaluate and model spatial audio understanding in embodied settings, since collecting real-world spatial audio is expensive and existing pre-trained omni-modal models have no obvious place to plug it in. The authors respond with a real-world benchmark (OmniEchoBench), a scalable spatial-audio rendering pipeline for training data, and a model (OmniEcho) that fuses FOA spatial tokens with a frozen semantic audio tower for spatial question answering and sound-guided navigation.

Key Contributions

1. The first treatment of spatial audio understanding for omni-modal embodied agents, covering both spatial audio-visual question answering and sound-guided navigation within a single task formulation.
2. OmniEchoBench, a benchmark with human annotations in real-world scenes: 197 real-world spatial audio-visual scenes, 2,972 QA pairs, 900 navigation samples, and FOA audio collected from 30 real-world environments across six tasks.
3. A controllable spatial-audio rendering pipeline that preserves geometric consistency among sound sources, visual observations, and agent trajectories, generating large-scale synthetic training supervision (QA pairs, audio-only data, and sound-guided navigation data).
4. OmniEcho, an omni-modal foundation model that adds an FOA spatial encoder alongside a frozen pretrained semantic audio pathway, trained in three stages and reaching state-of-the-art performance on spatial audio-visual perception in the authors' evaluation.

Main Findings

• OmniEcho leads on OmniEchoBench-QA. With FOA audio plus vision, OmniEcho scores 28.5 Overall, versus 18.5 for its Qwen3-Omni-30B-A3B backbone with single-channel audio plus vision, 12.6 for Qwen2.5-Omni-7B, and 11.4 for SO-7B. Random baseline Overall is 25.0 on the subtask subset where it is defined, which the authors report alongside the model rows.

• Audio-only performance also improves with FOA. In the audio-only setting, OmniEcho reaches 21.8 Overall with FOA input, compared with 17.5 for Qwen3-Omni-30B-A3B with single-channel audio, 12.2 for GLM4-Voice-9B, and 10.1 for SO-7B with FOA audio.

• Vision matters most for camera-centric reasoning. Removing visual input leaves Source Direction, 3D Localization, and Source Motion largely unchanged (22.6 vs 22.1, 14.2 vs 14.0, 24.7 vs 24.9), but Camera Rotation drops from 41.7 to 29.1 and Cognitive Map is only reported in the audio-visual setting (47.5). The authors conclude FOA mainly captures sound-source geometry while vision remains important for camera-centric reasoning.

• SO-7B degrades on real-world data. Although SO-7B claims spatial-audio training, the authors report it scores 10.1 Overall audio-only and 11.4 with vision on their real-collected benchmark, which they attribute to a lack of visual supervision during training and a sim-to-real gap.

• Sound-guided navigation approaches text-guided VLN. On OmniEchoBench-Nav, OmniEcho achieves 16.2% SR and 11.5% SPL, above Seq2Seq (11.3 SR / 9.4 SPL) and CMA (9.8 / 8.5), and below InternVLA-N1 (17.8 / 16.7) by 1.6 points SR and 5.2 points SPL. The SoundSpaces model converted to binaural audio reaches 5.4 SR / 3.9 SPL and struggles to stop autonomously.

• On simulated VLN-CE R2R, success rate is competitive but path efficiency lags. OmniEcho with audio instruction reaches 22.2 SR, 16.3 SPL, 7.31 NE, 24.5 OS, and 5.54 TL, slightly exceeding Seq2Seq and CMA in SR (with SPL below CMA's 20.0), with the shortest reported trajectory length, but behind VLN-R1 (30.2 / 21.8), NAViLA-SAGE (38.0 / 36.0), and InternVLA-N1 (50.6 / 43.3) in SR and SPL.

• Both audio pathways are complementary. The full configuration (native audio encoder + frozen FOA encoder) scores 28.5 Overall. Removing the native audio encoder drops Overall to 26.2, and using only the native audio encoder without FOA gives 19.8. Training (rather than freezing) the FOA encoder in Stage 3 degrades every subtask.

• Feature-based input beats raw waveforms, and Stage-1 pretraining helps. The feat5+stage1 variant achieves 17.9° azimuth error / 7.70° elevation error / 1.13 distance error, versus 30.5° / 13.8° / 1.65 for raw4+stage1. Adding an FOA encoder without Stage 1 and Stage 2 training brings no clear gains.

• Simulated and real distributions are aligned deliberately. The synthetic training corpus contains 145,103 samples; comparison with the real OmniEchoBench-QA benchmark yields a JSD of 0.14 bits for sound-source type, an identical median source distance of 2.8 m with a Kolmogorov–Smirnov statistic of D = 0.20, and added non-spatial detect/count/identify tasks.

Methodology in Plain English

Data collection. For perception (OmniEchoBench-QA), the authors wrote screenplays specifying acoustic events, actor motion, and their spatio-temporal relation to the camera, deliberately emphasizing off-screen and boundary-crossing sound sources. Human actors performed each script in one synchronized take, recording first-person video, 360° panoramic video, and four-channel FOA audio. The panoramic video is used only for annotation; at test time the model sees a single-view video or still image plus FOA audio. Annotators labeled every active sound source's position at 2 Hz, and rule-based sampling produced typed questions with automatically derived answers.

For navigation (OmniEchoBench-Nav), 30 real-scanned indoor environments (10 single-room, 20 multi-room spanning two to four connected rooms) each contribute 30 navigation samples, for 900 total. A sound source at a known 3D position was captured across a dense grid of receiver positions, producing 12,900 four-channel, 48 kHz FOA recordings. Sources split into 505 non-speech environmental events and 395 spoken commands. At test time the agent receives the recording nearest its current position, directionally remapped to its orientation.

Training data synthesis. The authors built a unified rendering framework in which each source follows a time-varying 3D trajectory and the listener is co-located with the camera in a right-handed frame (azimuth 0 = straight ahead, +90 = left, -90 = right). Monophonic stems are encoded to 4-channel FOA with real spherical harmonics, with optional distance attenuation and room effects. Dynamic scenes come from Seedance, combined with trajectory-grounded sound events; existing VLN trajectories are augmented with destination-conditioned sound events. In total the model trains on 363,193 examples.

Model design. OmniEcho builds on Qwen3-Omni-30B-A3B. Stage 1 pretrains a lightweight FOA encoder (384-dimensional tokens) on a 100k-clip synthetic corpus of scenes with 1–4 sources, aligning to a frozen CLIP text encoder with a SigLIP objective. Stage 2 fine-tunes it with a cross-attention head that predicts azimuth, elevation, and distance for a queried source. Stage 3 grafts the frozen encoder into the omni backbone: the omnidirectional channel is down-mixed to mono for the frozen audio tower to preserve native semantic audio tokens, while the same clip passes through the FOA encoder to produce spatial tokens, resampled to the 7 Hz audio-token grid and inserted after the corresponding audio tokens via a trainable projector. Only the LLM parameters and the projector are updated. Stage 3 training used 32 NVIDIA A100 GPUs for approximately four days. Navigation follows the InternNav setting, predicting a trajectory over the next K = 8 time steps from T frames plus FOA audio.

Why This Matters

The work argues that spatial acoustic cues are native, underexploited evidence for embodied agents — sound can reveal events outside the camera view and indicate the direction of occluded targets. It also supplies a real-world, FOA-recorded benchmark and simulator for closed-loop evaluation, addressing a gap where prior audio-visual navigation benchmarks relied on rendered acoustics and prior spatial-audio models lacked visual supervision or navigation evaluation.

Real-world applications:

• Search and rescue or service robots that need to locate a person calling for help from another room (the benchmark's spoken-command category includes help requests and calls).
• Home assistants and care robots that must react to alarms, doorbells, animal sounds, or footsteps outside the current field of view.
• Industrial monitoring agents that localize mechanical faults or alarms by sound direction and distance in noisy environments.
• Accessibility and telepresence systems that convey spatialized audio context to remote operators or users with limited vision.

Industry relevance: the method integrates a new sensory modality into an existing omni-modal LLM checkpoint rather than training a model from scratch, and is trained on top of open VLN trajectory datasets (ScaleVLN, RxR, R2R, VLN-PE), suggesting a practical path for adding spatial audio to existing agent stacks. The authors state that code and data will be released at the project's GitHub repository.

Future Directions

• Fine-grained spatial localization and distance estimation remain open challenges, which the authors explicitly identify as key remaining problems.
• Closing the sim-to-real gap in training data. The authors acknowledge that synthetic simulation data introduces bias, and report that Appendix A.5.1 suggests the distribution could potentially be corrected using a small amount of real spatial-audio data.
• Reliable stopping and path efficiency in sound-guided navigation. The paper notes that efficient navigation and reliable stopping remain significant challenges, and that OmniEcho falls behind stronger text-guided systems especially in SPL on VLN-CE R2R.
• Better integration of spatial and semantic audio pathways, given the ablation showing that removing the native audio encoder costs 2.3 Overall points, and that naively adding an FOA encoder without Stage 1 and Stage 2 pretraining produces no clear gains.

Target Audience

Researchers and engineers working on embodied agents, multimodal foundation models, spatial audio processing, and vision-language navigation. It is most useful for readers who want either a benchmark for evaluating spatial audio understanding in realistic settings or a concrete recipe for adding a new spatial sensory modality to a pre-trained omni-modal model — and for practitioners in robotics and assistive technology considering sound as a navigation and perception signal.

Authors’ abstract

Humans can effortlessly localize the direction of a sound source and integrate it with visual cues for reasoning, yet this remains challenging for embodied agents. In particular, it is still unclear how to effectively evaluate and model spatial audio understanding in embodied settings. To address this gap, we introduce \textbf{OmniEchoBench}, a unified benchmark for spatial audio-visual perception and audio-vision-language navigation. OmniEchoBench comprises six tasks over 197 real-world spatial audio-visual scenes, 2,972 question-answer pairs, and 900 navigation samples with first-order ambisonics (FOA) audio collected from 30 real-world environments. To enable scalable training supervision, we develop a controllable rendering pipeline for spatial audio. It preserves geometric consistency among sound sources, visual observations, and agent trajectories. Building on this, we propose \textbf{OmniEcho}, a spatially aware omni-modal model. It introduces an FOA spatial encoder alongside a pretrained semantic audio pathway. Extensive experiments show that OmniEcho achieves state-of-the-art performance on spatial audio-visual perception. For our sound-guided navigation, OmniEcho reaches a performance level close to that of traditional vision-language navigation. These results demonstrate that spatial audio can serve as a valuable signal for embodied scene reasoning and navigation, while also highlighting fine-grained spatial localization and distance estimation as important open challenges. Our code and data will be available in https://github.com/PKU-VaLuE-Lab/OmniEcho/tree/main