SceneBind: Binding What and Where Across Vision, Audio and Language

AI summary

Overview

Research area: Multimodal representation learning — joint semantic and 3D spatial understanding across vision, audio and language (Computer Vision, arXiv:2607.15265v1 [cs.CV], 16 Jul 2026).

Technical level: Intermediate. The paper builds on familiar CLIP-style contrastive pretraining and object-slot decoding, but assumes comfort with bipartite matching, contrastive losses (InfoNCE), cross-attention decoders, and spatial audio cues.

Scope in one sentence: SceneBind introduces a scene representation that pairs a global semantic embedding with object-centric "semantic-spatial slots" so that a single model can capture both what is in a scene and where it is, across image, binaural audio and text.

What This Paper Is About

Existing omni-modal encoders such as AudioCLIP and ImageBind are good at matching semantics across image, audio and text, but they largely ignore explicit spatial structure — they compress a whole scene into one embedding, treat rotations or flips as semantic equivalents, and often downmix audio to mono. SceneBind addresses this gap by representing each scene as a global embedding plus a set of object-centric slots that carry a semantic embedding, spatial attributes (camera-coordinate azimuth, elevation, distance) and a confidence score, then matching scenes with a scheme that combines global similarity with object alignment.

Key Contributions

1. SceneBind representation: A unified omni-modal scene representation combining a global semantic embedding with object-centric semantic-spatial slots (s_k, r_k, c_k), where each slot holds a semantic embedding, camera-coordinate azimuth/elevation/distance attributes, and a confidence score.
2. A curated real-world binaural audio-visual dataset and benchmark: Built from 21,927 in-the-wild videos with binaural audio, producing a training split of 38,430 clips with 207,836 objects and a human-verified Binaural benchmark of 1,066 clips, plus a Sphere360 zero-shot benchmark built from 360° ambisonic videos.
3. SceneBind Matching: An inference-time scheme that fuses a global scene similarity score with an object-centric slot-alignment score (weighted by slot confidence) for cross-modal scene retrieval and object grounding.
4. A two-stage training protocol: Global semantic alignment, object grounding and object-centric contrastive objectives trained first on mixed data (AudioCaps, MS-COCO, Binaural) and then fine-tuned on Binaural only, while integrating with pretrained frozen encoders (SigLIP2, M2D-CLAP) at low overhead.

Main Findings

• Cross-modal scene retrieval: SceneBind achieves the best overall results across A↔V, A↔T and V↔T retrieval (test sample pool size 1046), reaching 65.3 on V↔T, reported as +28% over pretrained encoders and +48% over fine-tuned baselines, and 17.0 on A↔T (+52%). Its average scene retrieval score is 34.7.
• Spatial retrieval: SceneBind reaches 28.9 R@1 and 48.0 mAP on object-level spatial retrieval, described as +153% and +62% over the best baseline. Fine-tuning existing encoders yields minimal gains here, which the authors attribute to their reliance on global semantics.
• Zero-shot Sphere360 hard-pool retrieval: SceneBind scores 29.3 average on the Sphere360 benchmark of spatially augmented views with identical semantics, reported as +56% over the best fine-tuned baseline.
• Object grounding: SceneBind reaches 20.1% overall grounding accuracy using audio and 19.1% using visual input. Spatial attribute accuracies are above chance, including 83.2% for audio elevation, and 67.1% (visual left/right) and 66.1% (visual distance). Audio gives stronger elevation recovery; vision performs better on left/right azimuth and distance.
• Zero-shot egocentric audio-visual localization: Without task-specific training, SceneBind reaches cIoU 52.65 at threshold 0.2, 33.73 at 0.3 and 19.47 at 0.4, with AUC 24.39, versus 38.71 / 19.42 / 10.51 and AUC 18.38 for the prior AVLoc method. A binaural-only variant (SceneBind (AV)) reaches 43.94 at 0.2 and AUC 20.80.
• Ablation — object semantic loss: Removing object-level semantic supervision drops grounding from 19.6 to 8.8 and spatial retrieval from 38.4 to 35.2, while scene retrieval rises slightly from 34.7 to 36.0 due to global bias.
• Ablation — contrastive losses: Removing the intra-scene contrastive loss degrades the average from 30.9 to 27.2, with spatial falling from 38.4 to 31.1 and grounding from 19.6 to 16.1. Removing the batch-level loss mainly hurts spatial retrieval (38.4 to 36.5).
• Ablation — data and schedule: Training on Binaural only gives scene 33.8, spatial 35.2, grounding 19.5; training all data in a single stage gives 35.2 / 36.5 / 19.9; the two-stage schedule (all data, then Binaural) gives the best balance at 34.7 / 38.4 / 19.6.
• Matching policy: Global-only matching scores 25.4 on spatial retrieval, while global plus slot matching raises this to 38.4. Slot-only matching keeps spatial performance at 38.4 but collapses scene retrieval (AV 21.3 to 11.5; A/V→T 34.2 to 17.3).
• Slot scaling: Performance saturates beyond about 10 slots, with roughly 10 sufficient to capture scene-level structure. Visual tasks benefit from more slots (peaking around 25) given higher object density, while audio peaks with fewer (around 5).
• Qualitative behavior: In A→V retrieval, SceneBind ranks the ground-truth park scene first and identifies cues such as "bird sounds" and "a child crying on the right"; in one scooter example the correct image is selected by a higher slot score despite a lower global score.

Methodology in Plain English

The researchers treat a scene as a 3D entity that can be observed through an image, binaural audio, or text, and they want one shared representation for all three. For vision, they reuse the frozen visual tower of SigLIP2 to get patch tokens that preserve image layout. For audio, they reuse the frozen audio tower of M2D-CLAP for semantics and train a small alignment module to map those tokens into SigLIP2's embedding space; separately, they convert the binaural waveform into a four-channel time-frequency representation (left/right magnitude and phase) and feed it into a convolutional spatial encoder trained from scratch, adding the result to the semantic audio tokens.

Each modality then goes through a cross-attention decoder. A learned global query produces the scene-level embedding; a set of learned object queries produce object slots and confidence scores. A spatial decoder then uses each object slot as a query to predict azimuth, elevation and distance via shared classifiers. Text is handled by a frozen text encoder, with spatial labels from object clauses used directly as targets.

Training works in two stages. Supervision combines a symmetric InfoNCE global alignment loss over audio-visual, audio-text and visual-text pairs with object-level losses. Because the number of annotated objects varies per scene, the model always predicts a fixed number of slots and uses bipartite matching — scoring slot-to-object pairs by semantic cosine similarity, spatial match probability and confidence — to decide which slot answers to which annotation. Matched slots are then grounded to their text clause and supervised for azimuth, elevation and distance with Gaussian-smoothed cross-entropy over discretized bins, so nearby spatial bins get smaller penalties. Intra-scene and cross-scene contrastive losses push slots for the same object together across modalities and keep different objects apart. Stage one mixes AudioCaps, MS-COCO and the new Binaural dataset; stage two fine-tunes on Binaural only to sharpen spatial grounding.

At inference, SceneBind Matching keeps slots above a confidence threshold, lets each query slot pick its best candidate slot by a confidence-weighted semantic-spatial score, and combines the resulting object score with the global scene score.

Why This Matters

The paper targets a concrete gap in multimodal AI: current encoders answer "what is this scene" but not "where is it." Making location an explicit part of the representation opens the door to systems that reason about spatial layout rather than just content, and the results show this does not come at the cost of semantic performance — SceneBind improves scene retrieval while adding spatial discrimination.

Real-world applications:

• Embodied AI and robotics: Agents that need to link a spoken or textual object reference to a direction and distance in a real room, as the paper notes for embodied AI and robotics settings.
• Spatially grounded multimodal world models: The paper explicitly frames semantic-spatial perception as enabling spatially grounded multimodal world models.
• Audio-visual localization: The zero-shot egocentric localization results point to uses in hearing aids, AR overlays, or video editing where a system must find which region of an image produced a sound.
• Spatial media search and retrieval: Retrieving scenes from large libraries by text, image or binaural audio where the query specifies a location ("the child crying on the right"), which the paper demonstrates in its qualitative retrieval examples.

Industry relevance: The method attaches to frozen pretrained encoders (SigLIP2, M2D-CLAP) with only a few additional tokens, which keeps adoption cost low for teams already using CLIP-style pipelines. The dataset curation pipeline and open benchmark also give the field shared evaluation ground for spatial multimodal tasks.

Future Directions

• Scaling real-world spatially aligned data: The authors state SceneBind may benefit from scaling spatially aligned multimodal data with richer in-the-wild annotations.
• Longer temporal windows: Extending beyond the two-second audio-image clips to model object motion, long-range scene dynamics, and temporally consistent semantic-spatial reasoning.
• Balancing spatial versus semantic supervision: The ablations show non-spatial data dilutes spatial performance while improving semantics; better recipes for combining data sources remain open.
• Improving slot-to-text matching: Hungarian one-to-one assignment helped T→A/V but hurt A/V→T because predicted slots are noisy relative to ground-truth clauses, suggesting further work on handling noisy predicted slots.

Target Audience

Researchers and practitioners in multimodal representation learning, spatial audio, 3D scene understanding, cross-modal retrieval, and embodied AI. It is most useful to readers who already know CLIP-style contrastive pretraining and want to understand how to add explicit spatial structure to shared embedding spaces. Engineers building retrieval, grounding, or audio-visual localization systems on top of pretrained encoders will also find the training protocol and matching scheme directly applicable.

Authors’ abstract

We present SceneBind, an omni-modal representation of realistic scenes with joint semantic and 3D spatial understanding across vision, audio and language. Existing omni-modal encoders excel at instance-level semantics (i.e., what is present), but often lack explicit spatial structure (i.e., where it is). SceneBind addresses this gap by representing each scene as a semantic-spatial entity, combining a global semantic embedding with object-centric semantic-spatial slots. This representation explicitly captures object-level semantics, spatial attributes, and uncertainty. We further propose SceneBind Matching, a semantic-spatial matching scheme that integrates global scene similarity with object alignment, supporting cross-modal scene retrieval and object grounding. To train and evaluate SceneBind, we curate a novel real-world binaural audio-visual dataset with structured semantic and spatial annotations, and propose a training protocol for aligning semantic and spatial signals across modalities. SceneBind is compatible with large-scale pretrained semantic encoders, adds lightweight spatial modeling with only a few additional tokens. It achieves state-of-the-art scene and spatial retrieval while enabling strong zero-shot transfer to downstream tasks such as audio-visual localization.