Audio-Visual Camera Pose Estimation with Passive Scene Sounds and In-the-Wild Video

AI summary

Overview

Research area: Computer vision and multimodal (audio-visual) learning, specifically relative camera pose estimation for real-world video.

Technical level: Advanced. The paper assumes familiarity with 6-DoF pose estimation, transformer-based pose backbones, spatial audio representations (direction-of-arrival spectra, binauralization), and AUC-based pose evaluation protocols.

Scope: The paper introduces an audio-visual framework that augments a state-of-the-art vision-only relative camera pose estimator with passive spatial audio, and validates it on two large datasets (Ego-Exo4D and HM3D-SS) plus visually corrupted variants.

What This Paper Is About

Estimating how a camera moved between two frames is normally done with vision alone, which breaks down under motion blur, darkness, occlusion, or textureless scenes. This paper asks whether incidental, everyday scene sounds—captured passively by microphones that move with the camera—can supply complementary geometric cues to a pose estimation model. The authors build a system that fuses direction-of-arrival audio spectra and learned binaural audio embeddings into a vision-only pose network, and show consistent gains on real-world video, especially when vision is degraded. To their knowledge, this is the first work to successfully use audio for relative camera pose estimation in real-world videos.

Key Contributions

1. A spatial audio encoder. The paper introduces an encoder that learns spatial embeddings from two complementary cues: direction-of-arrival (DOA) spectra computed analytically via MUSIC++ with frequency normalization, and cross-view binauralization features from a learned monaural-to-binaural "lifting" model.

2. Validation on two large-scale datasets. The approach is trained and evaluated on the Ego-Exo4D dataset (132 hours of egocentric sequences with audio) and on HM3D-SS (Habitat-Matterport3D SoundSpaces), achieving state-of-the-art performance among audio-visual baselines for relative camera pose estimation.

3. Training on real-world video. Unlike prior spatial-audio work that relies on static scanned environments and simulated audio, this method trains entirely on in-the-wild video from a variety of scenarios and activities.

4. Comprehensive analysis of audio characteristics. The paper provides a systematic study of which acoustic properties help or hurt multimodal pose estimation, including decoupled motion/audio regimes, per-scenario analysis, corruption studies, and negative controls.

Main Findings

• Audio-visual gains on Ego-Exo4D. On the Ego-Exo4D validation split, the full model reaches total AUC@5/10/20 of 0.81/2.99/8.82 versus the vision-only Reloc3r baseline at 0.57/2.33/7.33, with relative gains of +42.1%/+28.3%/+20.3%. Rotation-only AUC@5/10/20 rises from 37.35/53.62/68.98 to 38.54/54.49/69.61, and translation-only from 0.73/2.63/7.77 to 1.03/3.36/9.34. All gains over baselines and ablations are reported as statistically significant (p ≤ 0.05).

• Robustness to corrupted vision. Under corrupted visual inputs, the full model reaches total AUC@5/10/20 of 0.35/1.44/4.80 versus 0.21/1.00/3.66 for vision-only, i.e. relative gains of +66.7%/+44.0%/+31.2%. The method outperforms the baseline for all forms of corruption tested (Gaussian noise, Gaussian blur, color jitter) at all corruption levels.

• Audio-only baselines are weak but non-trivial. On Ego-Exo4D, the DOA-CP audio-only model is the strongest audio-only method (rotation AUC@5/10/20 of 19.07/33.37/50.07, total 0.03/0.14/0.78), ahead of Chance (19.02/33.33/50.03 rotation; total 0.02/0.12/0.76) and the NA-CP model, indicating DOA features carry meaningful geometric cues on their own.

• Both audio components matter. Ablations show the full model beats variants without DOA (total 0.75/2.86/8.41), without binaural (0.74/2.80/8.34), and with monaural embeddings (0.74/2.82/8.37). DOA features mainly improve rotation accuracy; binaural embeddings mainly improve translation.

• HM3D-SS results. On HM3D-SS, the model achieves the lowest mean absolute rotation error of 0.06 degrees on original data and 2.46 on corrupted data, versus 0.09/3.83 for vision-only Reloc3r and 0.77 for SLfM. Relative gains over the vision-only model are +33.3% (original) and +35.8% (corrupted). The improvement over vision-only on original data is described as minimal, attributed to the controlled, visually informative nature of HM3D-SS scenes.

• Noisy audio is tolerable. With Gaussian noise added to audio, performance stays largely preserved at total AUC@5/10/20 = 0.78/2.88/8.62, well above the vision-only baseline (0.57/2.33/7.33)—consistent with the late-fusion design limiting the influence of any single corrupted modality.

• Decoupled motion and audio regimes. Gains over vision-only persist when translation exceeds 1 m with rotation under 10° (AUC@5/10/20: 3.57/9.51/20.08 vs 2.56/7.58/16.65), under small camera motion with large sound change (1.05/3.74/10.89 vs 0.71/2.95/9.15), and under large camera motion with small sound change (0.40/1.57/5.89 vs 0.30/1.36/4.99).

• Some scenarios benefit more than others. The percentage of frames where audio helps: Soccer 84.63%, Music 81.40%, Bike Repair 79.55%, Rock Climbing 79.41%, Basketball 72.15%, Health 67.16%, Dance 66.14%, Cooking 64.13%. Gains are strongest with multiple spatially distributed sound sources.

• Audio provides a genuine geometric signal. Negative controls show performance degrades as audio–motion correspondence is disrupted: synchronized audio achieves the highest AUC@5 (0.81), followed by temporally offset audio at 1–5 s (0.68), same-scene shuffled audio (0.66), and cross-scene audio replacement (0.50), which falls below the vision-only baseline (0.57).

• Even flat directionality leaves a benefit. Under omnidirectional sound with flat DOA spectra, binaural embeddings still yield gains over vision-only of 8%/6%/3% for AUC@5/10/20.

• Failure modes. The model struggles in near-silent or diffuse-source conditions where directional cues are weak, and when a single dominant sound starts or stops between frames (e.g., a musician beginning to play only in the second frame).

Methodology in Plain English

The system takes a pair of RGB frames plus the synchronized multi-channel audio recorded at each frame, and predicts the 6-DoF relative camera pose between them (in both directions). It has two parts.

First, a spatial audio encoder converts each audio segment into a spectrogram and derives two features. One is an analytic direction-of-arrival spectrum (a 360-entry vector describing sound intensity around the azimuth circle, computed with MUSIC++), which shows where major sound sources are relative to the camera. The other is a learned binaural embedding, produced by training a model on a "novel view acoustic synthesis" task: given the source view's binaural audio and the target view's monaural audio, predict the target's binaural audio. Training this model uses an L1 loss on the spectrogram of the left-minus-right channel difference, which avoids having to predict exact spectrogram values while still forcing the model to capture inter-channel spatial structure. The two features are concatenated into a single spatial audio embedding.

Second, an audio-visual pose predictor adapts Reloc3r, a state-of-the-art vision-only relative pose transformer. Each image goes through its own branch with shared weights; cross-attention between the branches' decoders establishes visual correspondences. The audio embedding is concatenated with the visual feature entering each branch's pose estimation head—a late-fusion design that lets informative audio augment vision while preventing uninformative audio from overriding strong visual features. The pose loss combines rotation and translation error terms summed over both prediction directions.

Training and testing use real video. Ego-Exo4D supplies 132 hours of egocentric sequences with 7-microphone Aria audio; frame pairs are sampled one second apart using official splits with no cross-split overlap. HM3D-SS is used as a second testbed to compare against SLfM, using 2 microphones and SLfM's splits. Corrupted test sets are created by applying random combinations of Gaussian blur, Gaussian noise, and color jitter (brightness, contrast, saturation, hue) at random strengths. Evaluation uses AUC@5/10/20 on Ego-Exo4D and mean absolute rotation azimuth error on HM3D-SS.

Why This Matters

Impact on research. This is the first demonstration, to the authors' knowledge, that passive in-the-wild audio can improve relative camera pose estimation—a classic spatial challenge long treated as vision-only. It opens a new modality for embodied perception and 3D scene understanding, and it shows that spatial audio can be learned and used on real, unconstrained video rather than only in simulation or 3D scans. The negative-control experiments set a methodological standard by showing the gains reflect genuine audio geometry rather than dataset priors.

Real-world applications:

• Robust camera tracking for AR/VR and robot navigation in dark, crowded, blurred, or occluded environments where vision degrades.
• Improved SLAM and 3D reconstruction from consumer footage captured on devices with multiple microphones (Aria glasses, GoPro, Insta360 Go, Ray-Ban Meta glasses).
• Video stabilization or post-processing that uses ambient sound to infer camera motion in footage where visual tracking fails.
• Assistive or embodied agents that need spatial awareness in settings where acoustics (a stage, a crowd, a workshop) tell a clearer positional story than the camera feed.

Industry relevance. Because multi-microphone capture is now common on commercial wearables and action cameras, and because the approach uses only "free," non-intrusive ambient sound, it can be integrated into existing video pipelines without emitting signals or changing capture hardware. The late-fusion design also means graceful degradation on devices with only one microphone.

Future Directions

• Handling dominant-sound transitions. Failures occur when a single loud source starts or stops between the two frames; developing audio features robust to abrupt acoustic onsets and offsets is a clear next step.
• Extending beyond relative pose. The paper notes that relative pose is a core primitive for multi-view and temporally-extended systems; applying this audio-visual signal to full trajectory estimation, SLAM, or multi-view reconstruction is a natural extension.
• Adapting to other capture devices and microphone counts. The method is demonstrated with 7 microphones (Ego-Exo4D) and 2 (HM3D-SS), and monaural embeddings degrade gracefully; testing more device configurations, calibrations, and poorer audio would clarify practical limits.
• Improving simulation-based evaluation and calibration. DOA features performed poorly on HM3D-SS, which the authors attribute to inaccuracies in computing direction-of-arrival in simulation and missing precise microphone position parameters in the SoundSpaces configuration—an open problem for benchmarking against methods like SLfM.

Target Audience

Researchers and practitioners in computer vision, robotics, AR/VR, and multimodal machine learning who work on camera pose estimation, SLAM, 3D scene understanding, or audio-visual learning. It is also relevant to engineers building spatial computing and wearable-camera products who want to understand where ambient audio can substitute for or reinforce unreliable visual tracking. Readers need a solid background in pose estimation and audio processing to follow the technical details, though the high-level argument about passive sound as a spatial signal is accessible to a broader audience.

Authors’ abstract

Understanding camera motion is a fundamental problem in embodied perception and 3D scene understanding. While visual methods have advanced rapidly, they often struggle under visually degraded conditions such as motion blur or occlusions. In this work, we show that passive scene sounds provide cues complementary to vision for relative camera pose estimation for in-the-wild videos. We introduce a simple but effective audio-visual framework that integrates direction-of-arrival (DOA) spectra and binauralized embeddings into a state-of-the-art vision-only pose estimation model. Our results on two large datasets show consistent gains over strong visual baselines, plus robustness when the visual information is corrupted. To our knowledge, this represents the first work to successfully leverage audio for relative camera pose estimation in real-world videos, and it establishes incidental, everyday audio as an unexpected but promising signal for a classic spatial challenge. Project: http://vision.cs.utexas.edu/projects/av_camera_pose.