Enhancing XR Auditory Realism via Multimodal Scene-Aware Acoustic Rendering

AI summary

Overview

Research area: Human-Computer Interaction, specifically Extended Reality (XR) spatial audio rendering, combining on-device multimodal perception (depth, camera, 6-DoF pose) with geometric acoustics and Room Impulse Response (RIR) synthesis.

Technical level: Advanced. The paper assumes familiarity with acoustic rendering concepts (RIRs, RT60, early reflections, late reverberation, image source methods), deep learning architectures (DeepLabv3+, MobileNetV2, ASPP, quantization-aware training), and XR hardware constraints.

Scope: The paper presents Samosa, an on-device system that perceives a user's physical surroundings through XR headset sensors and dynamically synthesizes scene-matched spatial audio, evaluated through acoustic metrics on two collected datasets and a human expert study (N=12).

Note on the provided content: The full paper text supplied here is truncated mid-sentence in Section 6.1.2 ("Baselines"), so the quantitative results tables, the detailed comparison against baselines, and the reported outcomes of the human expert study are not present in the source content. Where results are not reported in the available text, this summary says so explicitly.

What This Paper Is About

XR headsets render spatial audio, but existing methods rarely adapt in real time to the physical room the user is actually standing in, so what users see and what they hear can disagree. That mismatch can cause cognitive dissonance and break immersion.

The goal of this work is a system that continuously reads the room's geometry, surface materials, and overall acoustic character from headset sensors, then renders audio whose reverberation and reflections match that room—fast enough and cheap enough to run on-device on XR hardware.

Key Contributions

1. A multimodal scene-aware rendering approach for XR that pairs a lightweight geometric acoustics engine with learning-based perception of geometry, material, and acoustic properties, enabling dynamic, on-the-fly parameterization of the audio renderer to match the user's current physical environment.
2. The design and implementation of Samosa, described as the first end-to-end system realizing this hybrid approach on XR devices, integrating real-time multimodal sensing to synthesize plausible RIRs on resource-constrained XR hardware.
3. A comprehensive evaluation, comprising (a) technical analysis of Samosa's acoustic accuracy and computational performance, and (b) a human expert study (N=12) reported as demonstrating significantly enhanced perceived audio realism compared to baseline methods.
4. A design-space analysis of three scene-based audio realism strategies—room geometry approximation, material segmentation, and semantic-driven acoustic parameter estimation—used to justify the system's specific position in that trade-off space.

Main Findings

• Three design dimensions structure the problem: The authors frame scene representation along Geometry (shoebox models vs. full 3D meshes vs. planar surfaces), Material (ignoring materials vs. fine-grained frequency-dependent coefficients vs. broad classification), and Acoustic Properties (RT60/EDT, reflection gain, reverberation gain or direct-to-reverberant ratio, and reverberation brightness).
• Samosa's chosen hybrid path: planar surface detection builds a localized shoebox around the listener; broad material classification is used rather than fine-grained estimation; acoustic properties combine physics-derived parameters with high-level scene context as a prior.
• Shoebox estimation accuracy and cost: the module reports position errors below 0.1 meters and orientation errors less than 0.9 radians, consuming less than 1% of CPU resources and approximately 0.48W of power on the target XR headset.
• Material segmentation model: a multi-head DeepLabv3+ with MobileNetV2 backbone predicting 9 material classes; trained for 20 hours and 22 minutes using approximately 1.17 CPU cores (GCU), 11.5 GiB of memory, and 8 TPU accelerators ('JellyDonut' TPUs); the optimized model is 2.5MB with 33,058,383 parameters.
• Scene classification model: a MobileNetV2 classifier over five scene types (conference room, living room, bedroom, outdoor, other); trained for 4 hours and 51 minutes using about 0.59 CPU cores (GCU), 11.6 GiB of memory, and 8 JellyDonut TPU accelerators; final optimized size is 956KB with 8,975,715 parameters.
• Surface material profiles are distributions, not single labels: for each of the six shoebox surfaces, the system keeps up to a maximum of 10 of the most prominent detected materials, each tracked by an area ratio and a confidence level (the paper's example is a wall represented as 70% heavy curtain and 30% glass).
• End-to-end performance: the audio rendering pipeline built on the Resonance Audio SDK has an end-to-end latency of 58ms and uses less than 2% of the CPU; the constituent models have a combined memory footprint of approximately 3.5 MB; the full system from sensor input to audio output consumes under 3% of total on-device computational resources.
• Two deployment targets: an audio processing pipeline integrated within a customized XR headset equipped with the Snapdragon XR2+ Gen 2 chipset for AR and MR use cases, and a plugin for the Unity engine to facilitate VR development.
• Evaluation data collected: scans from 25 diverse scenes using identical custom-built XR headset prototypes; the High-Fidelity Single-Scene (HFSS) dataset covers 5 distinct rooms, and the Dynamic Multi-Scene (DMS) dataset covers 20 scenes focused on transitions such as opening doors or moving between connected spaces.
• Ground truth methodology: for HFSS, a Genelec 8341 coaxial speaker was placed one-third of the room's floor diagonal away from a corner, with two B&K 4942 reference microphones (one near the center, one near an edge or corner), an exponential sine sweep equalized from 50Hz to 9kHz, a B&K 3670 USB audio interface, Farina-method RIR generation, and RT60 measured at both microphone positions using a B&K 2250 SPL meter. Apple Vision Pro FaceTime spatial audio was characterized in the same 5 rooms using an iPhone placed 10 inches from a B&K 4227A artificial mouth speaker, with output mirrored and recorded via OBS Studio.
• Quantitative evaluation results are not reported in the provided content: the paper text supplied here ends within Section 6.1.2, so the numerical acoustic-metric comparisons, baseline results, and the detailed statistical outcomes of the N=12 expert study are outside the available material. The abstract and contribution list state that Samosa showed feasibility and efficacy, and that the expert study demonstrated significantly enhanced perceived audio realism compared to baselines, but the supporting figures are not in the content above.

Methodology in Plain English

The researchers built a pipeline that runs several perception jobs in parallel and feeds them into an audio renderer.

First, depth feeds are used to detect flat surfaces, which are simplified into an axis-aligned box ("shoebox") around the listener. Because real rooms are irregular, the system remaps the listener and sound sources into the box so that distances to the three nearest orthogonal walls match the real distances, and it rotates the listener's orientation to preserve the relative direction between listener and source.

Second, camera images pass through a segmentation model that labels each pixel with one of 9 material classes. The model reuses a pre-trained MobileNetV2 backbone and adds a new prediction head, trained with a frozen backbone. Training labels were produced automatically: the Segment Anything Model created instance masks, and a vision-language model assigned a material label to each mask by comparing image and text embeddings, starting from the COCO-Stuff dataset. A second network classifies the whole scene into one of five room types; unlike the material head, it was trained from scratch with the backbone unfrozen, using room-type labels generated by a Gemini-based large language model.

Third, the 2D material labels are attached to the 3D box faces by projecting the faces into the camera view and accumulating results over time as the user looks around, producing an area-ratio and confidence profile per surface.

Fourth, scene type is mapped through a grid-search-optimized parameter vector (chosen by minimizing mean absolute error between generated and ground-truth RT60) to acoustic controls such as reverb gain, a reverb time modulator, and reverb brightness.

Finally, RIR synthesis uses the Resonance Audio SDK. Early reflections are computed with an image source method using blended, frequency-dependent absorption coefficients weighted by the material profiles. Late reverberation starts from a baseline RT60 computed with Eyring's equation from the shoebox volume and average surface absorption, then is refined by the learned scene-level parameters and rendered with a spectral reverberation technique. Early reflections are computed per source, while a single shared late reverberation model is reused for the whole scene to keep multi-source cost manageable. The resulting RIR is convolved with the input audio in real time.

Why This Matters

Impact on research. The paper argues that prior work forces a choice between high-fidelity acoustic simulation that is too expensive for mobile hardware and learning-based models that demand impractical, device-specific paired visual-acoustic datasets. Samosa's hybrid position—cheap geometry plus broad material classes plus a learned scene prior—offers a concrete alternative data point for on-device acoustic rendering, and its design-space analysis gives a structured framework for reasoning about fidelity-versus-cost trade-offs. It also connects to Sonic Interaction Design and the idea of implicit interaction, where acoustics adapt to context without explicit user commands.

Real-world applications (drawn from the paper's own application scenarios):

• Spatially anchored audio in AR and Video-See-Through. A virtual music box on a real table, a toy plane, or a virtual cat would be rendered so its sound reflects off and is occluded by real geometry.
• Co-presence and teleportation. Remote participants' voices in video conferencing would sound as if they were physically present in the listener's own room, reflecting off its actual walls and surfaces.
• XR entertainment. Music playback through simulated high-fidelity virtual speakers matched to the room, virtual concerts that adapt to a listener's living room, and AR game effects such as explosions or dialogue that interact plausibly with the physical space.
• Immersive VR scenes. In fully virtual environments, the system operates on reconstructed 3D scenes (for example, those produced by Zip-NeRF or real-time Gaussian Splatting methods like RadSplat) rather than on sensor-based scene perception.

Industry relevance. All authors are affiliated with Google, and the system was implemented on a customized XR headset with the Snapdragon XR2+ Gen 2 chipset as well as as a Unity plugin. The sub-3% total on-device resource usage and 58ms latency are framed as making always-on deployment realistic. The paper also positions Apple Vision Pro's FaceTime spatial audio effect as a practical baseline, indicating direct relevance to shipping consumer XR audio stacks.

Future Directions

• Report and extend the quantitative acoustic evaluation. The content available here does not include the objective results of the technical evaluation, leaving open how Samosa's synthesized RIRs compare numerically against the HFSS and DMS ground truths and against the Apple Vision Pro baseline.
• Scale material and geometry fidelity. The system deliberately uses 9 broad material classes, a single shoebox, and a maximum of 10 tracked materials per surface; whether finer-grained, frequency-dependent material estimation or richer geometry can be added without breaking the on-device budget remains an open question.
• Improve the scene-prior bottleneck. Acoustic parameters come from a five-way scene classifier and a one-time offline grid search over a validation dataset, which the authors themselves note lacks the specificity of direct estimation; replacing or enriching this prior is a natural next step.
• Validate across more environments and users. The technical evaluation spans 25 collected scenes (5 in HFSS, 20 in DMS) and a 12-person expert study; broader real-world testing across diverse rooms, transitions, and general users would test the robustness of the dynamic adaptation claim.

Target Audience

XR and HCI researchers working on spatial audio, immersive systems, and perceptual realism; audio engineers and machine learning practitioners building on-device multimodal perception pipelines; and XR platform developers interested in the practical cost profile—58ms latency, approximately 3.5 MB of model memory, and under 3% of on-device compute—of always-on scene-aware acoustic rendering. Readers without background in acoustics or on-device ML will find the design-space discussion accessible, but the evaluation sections assume technical familiarity.

Authors’ abstract

In Extended Reality (XR), rendering sound that accurately simulates real-world acoustics is pivotal in creating lifelike and believable virtual experiences. However, existing XR spatial audio rendering methods often struggle with real-time adaptation to diverse physical scenes, causing a sensory mismatch between visual and auditory cues that disrupts user immersion. To address this, we introduce SAMOSA, a novel on-device system that renders spatially accurate sound by dynamically adapting to its physical environment. SAMOSA leverages a synergistic multimodal scene representation by fusing real-time estimations of room geometry, surface materials, and semantic-driven acoustic context. This rich representation then enables efficient acoustic calibration via scene priors, allowing the system to synthesize a highly realistic Room Impulse Response (RIR). We validate our system through technical evaluation using acoustic metrics for RIR synthesis across various room configurations and sound types, alongside an expert evaluation (N=12). Evaluation results demonstrate SAMOSA's feasibility and efficacy in enhancing XR auditory realism.