How Much Physics Can Vision Learn from Audio? Probing and Distilling Damping Structure

AI summary

Overview

Research area: Computer vision and multimodal representation learning, specifically physical-property understanding and cross-modal (audio-to-vision) knowledge distillation.

Technical level: Intermediate. The paper assumes familiarity with frozen foundation-model encoders, linear/ridge probing, rank correlation, and contrastive distillation, but explains its central physical quantity (the quality factor Q) from first principles.

Scope: The paper measures how much information about an object's damping — summarized by the dimensionless quality factor Q — is linearly decodable from frozen audio versus frozen visual embeddings, and whether that structure can be transferred from a frozen audio teacher into a frozen CLIP encoder via a small trained projection head.

What This Paper Is About

Vision models are good at recognizing what objects look like but poor at inferring how they physically behave — a ceramic cup and a plastic cup look similar yet shatter or bounce very differently. Audio carries this information directly, because the way an object rings after being struck reveals its damping.

The paper asks a narrow, measurable question: if frozen audio embeddings can rank impacts by their quality factor Q far better than a single video frame can, how much of that gap can be closed by distilling the audio teacher into a frozen visual encoder — and how much of any gain is just material-category recognition in disguise?

Key Contributions

1. A continuous physical probe instead of categorical labels. The authors propose probing frozen encoders for the quality factor Q = πf₀/β, a dimensionless measure of how many oscillation cycles an object completes before its ring dies away, rather than for material categories as prior evaluations did. Audio embeddings reach ρ = 0.73 versus 0.28 for a single CLIP frame.

2. Evidence that the visual shortfall is partly temporal. Widening the visual input from one frame to four frames spanning the impact raises vision-only LOMO correlation from 0.234 to 0.285 (+0.051) with no acoustic supervision, indicating some of what a single frame misses is visible dynamics rather than hidden material structure.

3. Acoustic-to-visual distillation through a frozen backbone. A frozen PE-AV audio teacher is distilled into a frozen CLIP ViT-L/14 student through a trainable three-layer MLP head using a symmetric InfoNCE objective; the distilled four-frame model reaches 0.338 on materials withheld from both the head and the probe, versus 0.234 for a single-frame baseline (+0.104, 15 of 17 materials, Wilcoxon p = 0.002).

4. Controls that bound the claim. A capacity ablation (three-layer MLP on log(Q+1) with no teacher: 0.275; linear probe: 0.300), a shuffled-teacher control (0.359 at 10.6% material agreement), and a material-label reference (0.430 pooled) attribute most of the improvement to acoustic correspondence rather than extra parameters or category recognition.

Main Findings

• Audio carries damping; single-frame vision largely does not. On the held-out video split, PE-AV audio reaches ρ = 0.727 [0.707, 0.746] and CAV-MAE audio 0.701, while CLIP video reaches 0.281 [0.247, 0.313], DINOv2 0.271, and CAV-MAE video 0.151. PE-AV joint embeddings reach 0.696 and CAV-MAE joint 0.608.

• The audio advantage survives unseen materials. Under leave-one-material-out (train on 16 materials, evaluate on the 17th), PE-AV audio retains ρ = 0.600 [0.528, 0.658] and CAV-MAE audio 0.595, while CLIP video drops to 0.227 and DINOv2 to 0.239.

• Co-training on audio does not put physics into the visual pathway for free. PE-AV's video branch reaches only 0.362 pooled and 0.197 under LOMO; CAV-MAE's video branch reaches 0.151 pooled and 0.206 under LOMO — far below their own audio branches.

• Within a single material, vision is near chance. PE-AV audio reaches 0.67 on metal and 0.57 on wood, whereas every visual encoder is at or below 0.36 on metal and near zero (|ρ| ≤ 0.19) on wood, ceramic, and plastic. CLIP scores 0.06, 0.03, −0.14, and −0.19 on metal, wood, ceramic, and plastic respectively.

• Audio captures the damping term β better than pitch f₀. Across 17,958 impacts, Q is rank-correlated with f₀ at ρ = 0.87 and anti-correlated with β at ρ = −0.54 (f₀ and β are near-independent at ρ = −0.11). PE-AV audio reaches ρ = 0.729 on β versus 0.700 on f₀, and under LOMO β holds at 0.686 while f₀ falls to 0.485. Vision shows the opposite ordering when pooled (CLIP: 0.347 on f₀ versus 0.221 on β), consistent with pitch being partly inferable from visible size.

• Temporal input helps, but modestly. Pooled correlation rises from 0.281 (single frame) to 0.298 (four frames plus deltas), with overlapping intervals; the effect is clearer under true LOMO, where the same change is worth +0.051.

• Distillation moves a useful fraction of the signal. Pooled ρ rises to 0.440 (three-seed range) from 0.298 without a teacher; under true LOMO the distilled four-frame model reaches 0.338 versus 0.285 for the matched four-frame baseline (+0.053, 14 of 17 materials, p = 0.001).

• Gains concentrate on granular and natural surfaces. The five largest single-frame-to-distilled improvements are gravel 0.364 → 0.708, grass 0.377 → 0.610, dirt 0.285 → 0.605, leaf 0.290 → 0.541, and rock 0.247 → 0.469 — together about three-quarters of the total gain.

• Pooled correlation is a weak measure of physical understanding. Mapping the true material label to its per-category median Q already reaches ρ = 0.430, and a CLIP material classifier at 45.6% accuracy mapped the same way reaches 0.279. A pooled number near 0.28 is therefore indistinguishable from category recognition, which is why the argument rests on true LOMO.

• Roughly a third of the raw distillation gain survives destroying the correspondence. Against 0.298 with no distillation, a fully shuffled teacher (10.6% material agreement) reaches 0.359 and positional indexing (13.3% agreement) reaches 0.360, while the correctly matched teacher reaches 0.466 (first seed; three-seed mean 0.440). The larger part of the gain, +0.107, requires the teacher to correspond to the paired frame.

• What transfers is preferentially the damping term. Under true LOMO, the distilled single-frame student improves on both targets but most on β (0.257 → 0.375), versus f₀ (0.137 → 0.177).

• The extraction has a selection effect. The R² ≥ 0.8 fit filter retains 63% of annotated impacts (17,958 of 28,444), at strongly material-dependent rates: water retains 24%, gravel 33%, dirt 36%, cloth 83%, glass 91%.

Methodology in Plain English

Getting a physics label from sound. The authors start from the Greatest Hits dataset — 977 videos of objects struck with a drumstick, carrying annotations for 975 videos, 28,444 individual impacts across 17 material categories. For each impact they take the audio's amplitude envelope and fit an exponential decay with an additive noise constant, bounding the damping rate β to [0.1, 1000] s⁻¹, and read the resonant frequency f₀ from the dominant spectral peak above 50 Hz. Combining the two gives Q = πf₀/β. They keep only fits with R² ≥ 0.8. As a sanity check, median Q rises monotonically across materials: cloth 15, carpet 23, wood 55, plastic 89, metal 134, ceramic 327, glass 412.

Probing what encoders already know. Four frozen encoders are tested: PE-AV (audio-visual, trained on 100M pairs, with separate audio, video, and joint embeddings), CAV-MAE (a masked autoencoder trained on AudioSet), CLIP ViT-L/14 (400M image–text pairs), and DINOv2 ViT-L/14 (self-supervised, no language). Video inputs are single frames at the annotated moment of impact; audio inputs are 2-second clips centered on the impact. A ridge regression (λ = 1.0) maps frozen embeddings to log(Q+1), and performance is reported as Spearman rank correlation ρ, so the question is whether a model orders impacts by how much they ring, not whether it gets absolute values right. Intervals come from 1,000 bootstrap resamples over samples, or 10,000 resamples over materials for leave-one-material-out figures.

Distillation. The teacher is the frozen PE-AV audio encoder; the student is frozen CLIP ViT-L/14 with a trainable three-layer MLP head (GELU, LayerNorm, 1024-dimensional output) as the only learned component. Input is four frames at −0.1, 0, +0.1, and +0.2 seconds around the impact, each encoded independently and concatenated with their successive differences. Training minimizes a symmetric InfoNCE loss (τ = 0.07) between projected visual and teacher audio embeddings, with batch size 256 and AdamW at learning rate 10⁻⁴. The feature variant and epoch count were fixed on a 20% validation split of training videos (selecting four frames plus deltas, two epochs); the test split was used for no selection, and three seeds are reported. No Q label ever enters training. Crucially, the head is trained only on training-split impacts (22,793 of 28,444, from 780 of 975 videos), because the teacher's embeddings already predict Q so well that any impact seen during distillation would effectively leak its target.

The stricter evaluation. The authors define "true LOMO": the held-out material is excluded from the projection head as well as from the probe. This matters because Spearman ρ is undefined for constant predictions — a model that outputs a category's median Q scores zero within a held-out material — so the protocol removes any credit for category recognition.

Controls. Three controls bound the result: a material-classifier-to-median-Q reference (0.279), a capacity ablation comparing a linear probe (0.300) and a teacher-free three-layer MLP (0.275) against the distilled head, and a shuffled-teacher control (0.359) alongside positional indexing (0.360).

Why This Matters

Impact on research. The paper makes a methodological point as much as an empirical one: pooled correlation on a physical-property task is dominated by material-category identity, and a label-to-median baseline reaches ρ = 0.430 — close to the distilled encoder's 0.440. Any claim of physical understanding measured this way is therefore weak unless category information is neutralized. The true-LOMO protocol and the shuffled-teacher control are transferable to other cross-modal distillation work. The paper also reframes part of the appearance–reality gap as a single-frame limitation rather than a fundamental visual one, since four frames supply about half the improvement at no acoustic cost.

Real-world applications:

• Robotic manipulation: distinguishing a shatterable ceramic cup from a bounceable plastic one before grasping or dropping it, where material labels alone do not predict behavior.
• Scene understanding from video alone: estimating how much an object will ring, absorb, or recoil when deployed where microphones are unavailable or unreliable.
• Material and product inspection: flagging objects whose damping deviates from expectation, such as a cracked vs. intact component.
• Simulation and graphics: assigning physically plausible damping parameters to objects reconstructed from visual capture.

Industry relevance. The setup is deliberately cheap to deploy: both encoders stay frozen and only a small projection head trains, so the approach fits organizations that already run CLIP-family vision models and want to add physical grounding without retraining a backbone or instrumenting every scene with microphones. The reported gains are modest in absolute terms, and the authors are explicit that most of the teacher's signal does not cross — this is a characterization of a limit, not a shipped capability.

Future Directions

• Target β directly. Damping is the component audio predicts best (0.729 pooled, 0.686 under LOMO for PE-AV audio, nearly independent of f₀ at ρ = −0.11) and the one the distilled student improves on most (0.257 → 0.375). The authors name β as the natural next target for transfer.

• Test beyond one dataset and one visual domain. Evaluation is confined to Greatest Hits (977 videos, 17 materials) and controlled drumstick footage; whether the head learns a general appearance-to-damping mapping or dataset-specific associations is unresolved.

• Address the R² ≥ 0.8 selection effect. The filter keeps 63% of impacts at strongly material-dependent rates, and the five largest per-material gains fall on materials retaining a third to just over a half of their impacts. The discarded impacts cannot be assessed with Q at all, since the target is undefined there.

• Connect decodability to decisions. The paper measures transferred structure by probing accuracy alone; whether better Q decodability yields better physical decisions remains to be shown on a task whose outcome is not a function of Q. The authors also note that properties impact sound encodes weakly or not at all — friction, for example — may behave differently, and the four-frame model is evaluated on Q alone (the f₀/β probe uses a single-frame student).

Target Audience

Researchers and engineers working on multimodal representation learning, physical reasoning benchmarks, and robot perception will get the most from this paper, particularly those interested in what frozen encoders do and do not expose and in how to evaluate cross-modal distillation honestly. It is also useful for practitioners who need to audit claims about "physical understanding" in vision models: the label-to-median baseline and the shuffled-teacher control are directly reusable diagnostics. Readers should be comfortable with embedding probing, rank correlation, and contrastive objectives; the physics itself is explained clearly enough for newcomers, and the paper's careful framing of its own modest effect sizes makes it a good model of restrained empirical reporting.

Authors’ abstract

Physical understanding is crucial for decision making, yet vision models struggle to infer properties such as damping that govern how materials behave. Audio encodes such properties directly, because an object's acoustic response to impact reveals its damping through measurable decay. We probe frozen encoders for the quality factor $Q$, a dimensionless measure of how many oscillations an object completes before its ring dies away, and report how well each embedding orders impacts by $Q$ using the Spearman rank correlation $ρ$, for which $ρ=1$ is a perfect ordering and $ρ=0$ an unrelated one. Audio embeddings reach $ρ=0.73$ and retain most of that advantage on materials withheld from training, while a single video frame taken at the moment of impact, encoded by CLIP, supports only $ρ=0.28$. Having established where the signal lives, we ask how much of that gap vision can recover. We distill a frozen audio teacher into a frozen CLIP backbone through a trained projection head and find that this recovers a useful fraction of the audio advantage. Because pooled correlations on this task are dominated in practice by category identity, we also evaluate under a leave-one-material-out protocol in which the held-out material is withheld from the projection head as well as the probe. Distillation with a four-frame input improves the correlation there from 0.234 to 0.338 (+0.053 over the matched four-frame baseline, 14 of 17 materials), a gain that no category-level strategy can produce. Widening the visual input to four frames spanning the impact supplies half of that improvement at no acoustic cost, which indicates that part of what a single frame misses is visible dynamics.