Objects as Audio-Visual Modal Sound Fields

AI summary

Overview

Research area: Computer vision and audio-visual multisensory object modeling — specifically, reconstructing physically grounded impact sounds for 3D objects from images plus a few recordings.

Technical level: Advanced. The paper assumes familiarity with linear modal analysis, 3D Gaussian Splatting, differentiable rendering, DINOv2 features, and score distillation sampling.

Scope: The paper introduces AV-MSF, an object-level representation that fuses multi-view images with a small number of impact sound recordings to render impact sounds at novel contact locations, and demonstrates contact localization and text-driven sound editing as downstream applications.

What This Paper Is About

Modern 3D reconstruction recovers an object's shape and appearance from images, but says nothing about how the object sounds when struck — even though impact sounds reveal material, stiffness, and thickness that vision alone cannot determine. Existing solutions either run expensive physics simulations, or train purely data-driven generative models that require large datasets and produce physically ungrounded audio. AV-MSF's goal is to reconstruct a compact, physically interpretable acoustic model of a single object from multi-view RGB images and only a few real impact recordings, so that impact sound can be rendered at any new contact position on the object's surface.

Key Contributions

1. The AV-MSF representation itself: a physics-based object-level modal sound field reconstructed from multi-view RGB observations and a few impact sound recordings, in which global modal frequencies and dampings, a residual noise component, and a spatially varying neural gain field are jointly learned.

2. A geometry-aware visual prior for acoustics: a 3D Gaussian Splatting representation enriched with lifted and symmetry-aligned DINOv2 features, which conditions the prediction of location-dependent modal gains. This visual prior is what makes few-shot reconstruction work.

3. State-of-the-art impact sound rendering: the paper reports that AV-MSF substantially improves novel-contact-position impact sound synthesis over both physics-based (DiffSound) and data-driven (SonicGauss) baselines on two real-world datasets.

4. Two downstream applications: contact localization (inferring the 3D impact location from an unseen recording) and object sound editing (text-driven modification of acoustic parameters via Audio Score Distillation, with a hierarchical frequency parameterization and a physical rescaling rule for the gain field).

Main Findings

• Outperforms both baseline families on novel-position rendering. Averaged results on ObjectFolder Real and RealImpact (lower is better): AV-MSF reaches L1 0.013 / L1 Log 0.951 / ENV 0.014 / CDPAM 1.35e-4 on ObjectFolder Real and L1 0.021 / L1 Log 0.996 / ENV 0.017 / CDPAM 2.16e-4 on RealImpact, versus DiffSound (0.031 / 1.298 / 0.030 / 2.53e-4 and 0.029 / 1.533 / 0.024 / 2.39e-4) and SonicGauss (0.033 / 1.281 / 0.018 / 2.01e-4 and 0.039 / 1.580 / 0.025 / 2.43e-4).

• A 2× improvement claim in the few-shot regime. In the 20% training-data setting the authors report a 2× improvement over prior physics-based methods and over data-driven baselines pretrained on large-scale simulated data.

• KNN is a surprisingly strong baseline. With K=3, nearest-neighbor averaging over training impacts scores L1 0.014 / L1 Log 0.930 / ENV 0.014 / CDPAM 1.53e-4 on ObjectFolder Real — close to or better than some learned baselines, because the evaluation datasets are dominated by symmetric objects where nearby surface points share similar spectra.

• The advantage is clearer on non-symmetric objects. On a less symmetric subset of ObjectFolder Real, AV-MSF improves over KNN across all metrics: L1 0.0110 vs 0.0135, L1 Log 0.9259 vs 0.9724, ENV 0.0131 vs 0.0138, CDPAM 1.53e-4 vs 2.15e-4.

• Why the baselines fail. DiffSound's inverse-rendering parameter estimates are error-prone and place frequency bands or dampings incorrectly; SonicGauss is data-hungry — even after pretraining on ObjectFolder 2.0, few-shot fine-tuning remains ineffective, and its generative nature introduces hallucinated or distorted spectral patterns.

• Every component of the method matters (ablation). Removing the visual prior gives L1 0.028 / L1 Log 1.148 / CDPAM 4.20e-4; removing modal-parameter initialization gives the worst L1 (0.045) and is described as prone to training collapse; removing feature alignment gives L1 0.019 / L1 Log 1.002; removing the residual component causes a dramatic L1 Log degradation to 5.764. The full model scores 0.019 / 0.927 / 0.015 / 2.00e-4 on the ablation subset.

• The residual component protects the gain field. Removing it worsens contact localization error from 38.4% to 43.8% on the ablation subset, suggesting the gain field otherwise absorbs non-modal effects.

• Contact localization beats DiffSound. Reported RMED (lower is better): AV-MSF 34.61% versus DiffSound 41.78%.

• Sound editing beats Audio-SDS on the reported metric. UMAP score (lower is better): Generation 3.077, Audio-SDS 4.234, AV-MSF 2.753.

Methodology in Plain English

The method takes two inputs per object: calibrated multi-view photos, and a handful of impact recordings, each paired with the 3D surface point where the strike occurred.

Build a 3D visual representation. The researchers reconstruct the object with 3D Gaussian Splatting to get a dense cloud of Gaussian centers. They extract 2D features with the pretrained DINOv2 encoder and lift those features onto the 3D Gaussians, producing a geometry-aware feature field. To keep features consistent across symmetric or repeated regions (such as the two wings of a swan sculpture), they automatically detect object symmetries by measuring geometric alignment error of candidate transformations, then average features across symmetric counterparts.

Initialize the acoustics from physics. Because directly fitting a sound field from a few recordings is non-convex and prone to bad local minima, they first extract modal parameters from the recordings themselves. Each recording is decomposed via Short-Time Fourier Transform into narrow-band time-domain signals; peaks that appear too late or are insufficiently damped are discarded. Damping and gain for each candidate mode come from a log-linear regression on the log magnitude of the Hilbert transform. Frequencies that appear consistently across recordings become the object's global modes, and the global damping for each mode is the average across recordings. A separate residual component — learnable per-frequency-band magnitudes, initialized from the lowest-energy segments of the recordings and applied to filtered white noise — absorbs background noise, contact-force variation, and microphone-distance effects.

Predict location-dependent gains with attention. Gaussian centers are clustered into K spatial groups, and features within each cluster are averaged into a region descriptor. For a query impact point, the model finds the nearest Gaussian center, takes its local feature, computes relative offsets to all cluster centers, and positionally encodes them. An attention module over the region descriptors produces a context vector, which is concatenated with the local feature and passed through an MLP to output one gain per mode.

Train in two stages. A warm-up stage minimizes MSE between the predicted gains and the gains extracted from the recordings, anchoring the gain field in a well-conditioned regime. The full training stage optimizes everything end-to-end with a multi-scale STFT reconstruction loss applied both to the complete waveform and to an early-time segment where impact energy is concentrated.

Applications. For contact localization, the object-intrinsic frequencies and dampings are used to extract mode gains from a new recording, and the surface position minimizing cosine distance to the predicted gain field is returned. For sound editing, Audio-SDS distills text guidance from a frozen text-to-audio diffusion model into the modal parameters, using a global pitch-scaling factor plus per-mode residual refinements to bridge large spectral gaps between materials such as wood and metal; a physical rescaling rule keeps the spatial gain field consistent with linear modal analysis after material edits.

Experimental setup. The evaluation uses ObjectFolder Real (100 objects across seven material types, 30–50 impacts each; 20% of recordings selected by farthest-point sampling for training) and RealImpact (150,000 recordings from 50 objects with 5 impact locations each struck 600 times; only the closest microphone is used, giving 5 recordings per object under leave-one-out cross-validation). All recordings are force-normalized, peak-aligned, and truncated to 3 seconds. Metrics are L1 distance, L1-log distance, envelope distance, CDPAM, and RMED for localization.

Why This Matters

Impact sound is the sensory channel that reveals what an object is made of when appearance is ambiguous — a glass and a plastic goblet look alike but sound completely different. Making that channel reconstructable from ordinary images plus a few taps turns passive 3D scans into physically interactive assets. The paper's particular contribution is showing that a compact, interpretable modal parameterization plus a visual prior can beat both expensive physics inverse-rendering and large-scale data-driven generators in the few-shot regime.

Real-world applications:

• VR/AR and gaming: spatially consistent impact sounds at any contact point on a scanned object, for immersive and physically plausible contact feedback.
• Robotics and embodied agents: training multimodal agents with rendered impact audio, and inferring contact location from sound alone (the 34.61% RMED localization result).
• Content creation for film and games: generating Foley-style impact audio for 3D assets without recording sessions or hand-tuned physics parameters.
• Multisensory asset digitization: scanning a physical object into a combined visual and acoustic asset that can be re-materialed or re-tuned through text prompts.

Industry relevance: the method targets a practical cost problem — physics-based modal synthesis is expensive and data-driven generation needs large corpora, while AV-MSF needs multi-view photos and a few recordings per object. That profile fits asset pipelines in games, AR/VR, and simulation, where per-object data collection time is the bottleneck.

Future Directions

The paper does not state explicit future work; the following are open questions it raises.

• Breaking the position-invariance assumption. The formulation assumes modal frequencies and dampings are object-intrinsic and only gains vary with contact location, but the authors observe that damping does vary with location for some real objects, and they make it only optionally learnable. How much accuracy is left on the table, and can damping be modeled as spatially varying as a default?

• Beyond the Rayleigh damping approximation. Linear modal analysis is described as a numerical approximation that does not hold for all real-world objects. Extending the representation to nonlinear or non-Rayleigh regimes is unresolved.

• Scaling to scenes and non-symmetric objects. The evaluation sets are dominated by symmetric objects where nearest-neighbor averaging is already competitive, and RealImpact provides only 5 distinct impact locations per object. Testing on larger, less symmetric, and multi-object collections is an obvious next step.

• Improving localization accuracy. Contact localization at 34.61% RMED improves on the baseline but is far from precise; the paper's own figure includes a failure case for contact localization, and the ablation shows the residual component materially affects this error.

• Generalizing the sound editing pipeline. The editing results are reported on a single UMAP metric, and edits are constrained to remain consistent with linear modal analysis; whether editing generalizes across object categories and prompts is not established.

Target Audience

Researchers and practitioners in computer vision, computer graphics, and audio-visual machine learning who work on 3D reconstruction, multisensory object modeling, neural audio synthesis, or physically based sound rendering. It is also relevant to engineers building VR/AR, gaming, or robotics pipelines who need efficient per-object acoustic assets, though those readers will need background in modal analysis and differentiable rendering to follow the formulation.

Authors’ abstract

While modern 3D reconstruction excels at modeling object geometry and appearance, it largely ignores the rich acoustic cues revealed through physical interaction. Object impact sounds convey material, stiffness, and structural properties that complement vision, yet existing impact sound modeling approaches either rely on expensive physics-based simulation or require large datasets to generalize in a purely data-driven manner. We introduce Audio-Visual Modal Sound Field (AV-MSF), a novel object-level acoustic representation reconstructed from multi-view images and only a few impact sound recordings. AV-MSF builds on 3D Gaussian Splatting integrated with dense 3D visual feature to provide a strong geometry-aware prior, and represents the impact sound field using compact, physically meaningful modal parameters, enabling robust few-shot reconstruction. Experiments on two real-world datasets show that AV-MSF achieves state-of-the-art impact sound rendering, outperforming both physics-based and data-driven baselines. Furthermore, we demonstrate downstream applications enabled by our representation, including contact localization and object sound editing.