Hear the World in Stereo: Learning Dynamic Spatial Correspondence for Immersive Joint Video-Audio Generation

AI summary

Overview

Research area: Computer vision and multimodal generative modeling — specifically joint video-audio (VA) generation, stereo/spatial audio synthesis, and video-to-spatial-audio (V2SA) conversion.

Technical level: Advanced. The paper assumes familiarity with Diffusion Transformers, rotary position embeddings (RoPE), LoRA adapters, cross-attention, and acoustic concepts such as ITDs/ILDs and Room Impulse Responses.

Scope in one sentence: The paper introduces StereoBind, a track-conditioned framework that binds visual source motion to stereo sound generation, together with a large-scale stereo VA dataset (StereoWorld-29K) and a benchmark (StereoWorldBench) for evaluating audiovisual spatial consistency.

What This Paper Is About

Existing joint video-audio generation models handle what is heard (semantic correspondence) and when it occurs (temporal synchronization), but they do not guarantee that the perceived location of a sound follows the visible source as it moves. The authors call this property Dynamic Spatial Correspondence and target it directly: given a text description and a motion track, the system should generate video and stereo audio in which the visible sound-producing source follows the track while the left-right auditory spatial cues evolve consistently with that motion.

Key Contributions

1. StereoBind, described as the first framework for track-conditioned stereo VA generation, which models consistency between source motion and acoustic location through three components: Visual Motion Binding (VMB), a Spatial Track Encoder (STE), and Residual Track RoPE (RT-RoPE).

2. StereoWorld-29K, presented as the first large-scale stereo VA dataset with explicit spatial supervision, built through two pipelines: a Visual-Audio Spatialization (VAS) Pipeline that spatializes real-world audio from source trajectories, and a Spatial Data Synthesis (SDS) Pipeline that generates track-controlled scenes.

3. StereoWorldBench (SWBench), introduced as the first benchmark for measuring the alignment between generated stereo audio and source motion, complementing conventional audiovisual quality metrics.

4. Architectural mechanisms: VMB for source-aware audiovisual binding, STE with layer-specific Track-Adaptive modulation for absolute spatial conditioning, and RT-RoPE with a trainable Stereo LoRA for relative motion modeling.

Main Findings

• Best stereo spatial fidelity among compared models (Table 1): StereoBind reports ILD-W of 2.754, SELD-Acc of 0.757, SMR-Err of 14.86, AST-Ang of 0.465, and AST-Cal of 0.314. For comparison, MiniMax-H3 reports ILD-W 3.435, SELD-Acc 0.725, SMR-Err 24.64, AST-Ang 0.257, and AST-Cal 0.131; LTX-2.3 reports ILD-W 3.574, SELD-Acc 0.617, SMR-Err 39.34, AST-Ang 0.221, and AST-Cal 0.119.

• Best or tied-best generation quality on several axes: StereoBind achieves the best Motion Smoothness (0.996), Imaging Quality (0.697), Audio PQ (6.86), and AV Sync (0.262) in Table 1, while Subject Consistency (0.955) is comparable to other models (MiniMax-H3 reaches 0.968, Ovi 0.962).

• Note on AV Sync direction: The table marks AV Sync with a downward arrow (lower is better). StereoBind's value of 0.262 is lower than Ovi (0.278), MiniMax-H3 (0.301), LTX-2.5 (0.367), LTX-2.3 (0.374), PrismAudio (0.479), and See2Sound (0.406), and the paper describes StereoBind as achieving the best AV Sync.

• Every ablated component matters (Table 2): Removing RT-RoPE, STE, or VMB degrades spatial fidelity. The largest SMR-Err increase comes from removing STE (27.38, versus 14.86 for the full model), and removing VMB drops SELD-Acc to 0.506 from 0.757.

• VMB token count trades off capacity against cost: Performance is not strictly monotonic across metrics. N_VMB = 16 gives AST-Ang 0.322 and SMR-Err 18.65; N_VMB = 32 gives SELD-Acc 0.691; N_VMB = 128 gives the highest SELD-Acc (0.747), AST-Ang (0.471), and AST-Cal (0.321) with the lowest SMR-Err (13.47), but the paper reports that performance largely saturates at N_VMB = 64 and that 128 yields only marginal gains at higher memory overhead, so 64 is adopted.

• Qualitative spatial behavior: Using Signed Inter-channel Level Difference (SILD) and Inter-channel Differential Spectrogram (IDS) visualizations, the authors report that baseline SILD trajectories often deviate from reference spatial evolution and show inconsistent IDS structure, while StereoBind's stereo dynamics closely follow source motion.

• Human perceptual results: The paper states the method achieves the best human perceptual results as reported in Appendix E; the appendix content and any corresponding numbers are not included in the provided text.

• Training cost: 8K iterations on 8 NVIDIA H20 GPUs, approximately two days, using the original LTX-2.3 training objective.

Methodology in Plain English

StereoBind is built on the LTX-2.3 VA generation backbone, which has separate video and audio branches. A frozen Motion Track IC-LoRA is added to the video branch so the video can follow a supplied track, and trainable LoRA (rank 32, scaling factor 32) is injected into the attention and feed-forward layers of the audio branch.

Three mechanisms tie the audio to the visual motion:

• Visual Motion Binding (VMB): A set of 64 learnable tokens per block integrates information in a reference–target–audio order — first reading the motion-reference representation, then attending to the current target-video state being denoised, and finally letting the audio stream read out the fused result. A residual term captures how the target video modifies the reference-conditioned representation. This gives the audio a source-aware sense of what is moving and how.

• Spatial Track Encoder (STE): The 2D motion track (projected to a latent dimension of 256 over 121 temporal positions) is processed by a temporal Transformer, resampled to the audio token length, and turned into layer-specific shift, scale, and gate parameters through a learnable Track-Adaptive table. These modulate the audio queries and control how strongly the retrieved spatial update is injected, so each Transformer block gets absolute position information fitted to its own audio features.

• Residual Track RoPE (RT-RoPE): The pretrained temporal rotary embedding is augmented with track-dependent residual phases derived from normalized horizontal and vertical track coordinates, with frequency parameter θ_track = 2 and tanh-bounded learnable modulation strengths. Because this changes the positional geometry of attention, a lightweight LoRA adapter is added to audio self-attention while pretrained weights stay frozen. The relative phase between two audio tokens then carries signed relative motion displacement.

Data construction: The real-world branch draws from VGGSound, OpenHumanVid, FoleyBench, and LU-AVS, filtered by quality criteria and by Qwen3-Omni for motion-aware semantics. The VAS Pipeline identifies sound sources with Qwen3-Omni, localizes them with Grounded SAM, verifies candidates audiovisually, tracks the chosen source with AllTracker, recovers geometry with VGGT, and renders stereo audio via gpuRIR using a 10×10×4 m room, T60 = 0.25 s, and virtual left/right receivers. The synthetic branch (SDS) samples motion and semantic configurations, uses Qwen3.5 to write motion descriptions and VA prompts, and maintains a Scene Distribution Bank to reduce repetition; a human-centric branch uses 47 speaker profiles and 41 environments. Static-left and static-right sources each default to 0.15 of samples, leaving 0.70 for dynamic motion; source extent is point-like with probability 0.65 and area-like with 0.35.

Why This Matters

Impact on research: The paper reframes joint VA generation around where sound comes from and how that location evolves, adding a measurable target (SWBench) and a supervision source (StereoWorld-29K) where prior work focused on semantic and temporal alignment. It also shows that spatial cues can be learned without explicitly estimating ITDs or ILDs — instead, the track conditions the audio through learned modulation and modified rotary geometry.

Real-world applications:

• VR and AR headsets, where users need sounds to come from directions consistent with visible objects as they move.
• Interactive games and immersive entertainment, where perceived sound position must track on-screen action.
• Video-to-spatial-audio post-production, since the paper evaluates V2SA models (PrismAudio, See2Sound) that take a generated video and synthesize its spatial audio.
• Synthetic scene and content generation for spatial media, supported by the SDS Pipeline's track-controlled scene synthesis.

Industry relevance: The work comes from Xidian University and vivo BlueImage Lab (vivo Mobile Communication Co., Ltd.), with a project page hosted under vivocameraresearch. That placement points toward consumer mobile and camera-centric immersive content, where generating two-channel audio with meaningful left-right cues — rather than merely two channels of audio — is the practical requirement.

Future Directions

• Extending dynamic spatial correspondence to richer 3D scenes, as the authors state directly in the conclusion.
• Handling multiple simultaneous sound sources, rather than a single dominant tracked source, which is also named in the conclusion.
• Modeling more complex source–listener dynamics beyond the current setup.
• Addressing scalability and data limits: the paper notes that calibrated real-world spatial recording is costly and that existing Audio-Visual Segmentation models generalize poorly to unconstrained scenes, which motivated the VAS Pipeline. Whether the same recipe extends to more open-domain footage is an open question.
• The VMB capacity study suggests diminishing returns past 64 tokens; more parameter-efficient ways to inject spatial context could be explored.

Target Audience

Researchers and graduate students in multimodal generative modeling, spatial audio, and diffusion-based video generation; engineers building VA or video-to-spatial-audio systems for AR/VR, gaming, and immersive media; and practitioners working on audiovisual datasets and evaluation benchmarks for spatial consistency.

Authors’ abstract

Recent joint video-audio generation models have achieved strong semantic correspondence and temporal synchronization. However, applications such as AR/VR and interactive gaming further require stereo audio to provide an immersive sense, which remains largely overlooked. Effective stereo audio requires the perceived sound location to evolve consistently with the motion of its corresponding visual source. We refer to this property as Dynamic Spatial Correspondence and propose StereoBind, a framework that binds visual source motion to stereo sound generation. StereoBind uses motion tracks to coordinate visual motion and stereo audio through three complementary mechanisms. Visual Motion Binding establishes source-aware audiovisual correspondence, the Spatial Track Encoder captures absolute source positions, and Residual Track RoPE models relative motion. For supervision and evaluation, we construct StereoWorld-29K, a large-scale stereo audio-video dataset with paired motion tracks, and StereoWorldBench for measuring audiovisual spatial consistency. Experiments show that StereoBind substantially improves spatial alignment in stereo audio generation over existing models while preserving overall audiovisual quality.