Physics-Aware Deepfake Detection via Distance-Speech Consistency

AI summary

Overview

Research area: Audio-visual deepfake detection, computer vision, and applied acoustics.

Technical level: Intermediate — the paper combines acoustic propagation theory (inverse-square law, SNR, C50) with standard vision components (face detection, monocular depth), so familiarity with audio-visual learning helps, but the core idea is accessible.

Scope: The paper proposes and evaluates a deepfake detector that checks whether speech energy varies with speaker-to-camera distance as real-world acoustic physics predicts, rather than checking lip–speech synchronization.

What This Paper Is About

Most audio-visual deepfake detectors work by checking whether lip movements match the speech, which requires a clearly visible mouth and works best on static, frontal talking-head videos. In real in-the-wild footage — handheld recordings, vlogs, on-the-street interviews — the speaker moves and the face is often occluded, blurry, or turned away, which breaks those detectors. This paper instead asks whether the physical relationship between how far a speaker is from the camera and how loud/clear their speech is has been preserved, since deepfake generation pipelines often break that coupling.

Key Contributions

1. A distance–speech consistency measure. The authors introduce a physical cue based on the fact that in real recordings, speech energy attenuates with speaker-to-camera distance (the inverse-square law), expressed through two relative acoustic measures: Signal-to-Noise Ratio (SNR) and Clarity at 50 ms (C50).

2. A detector built on that measure. They extract speaker distance from video and distance-dependent acoustic measures from speech, aggregate them by distance-wise binning, and feed five statistics into a lightweight MLP to produce a fake/real score.

3. Evaluation on dynamic in-the-wild videos. They curate DF-Dynamic (40 test videos with noticeable speaker motion) and report that the physics cue outperforms two lip-sync-based detectors on it.

4. Evidence of complementarity. A simple score-averaging ensemble with lip-sync-based detectors improves performance consistently across datasets. The authors state this is the first work to use audio-visual physical constraints as a primary cue for deepfake video detection.

Main Findings

• Large gain on the dynamic subset (DF-Dynamic, 40 curated test videos): the proposed method reaches 0.7449 ROC-AUC, compared to 0.5096 for SpeechForensics and 0.5517 for AuViRe.

• Strong in-domain result on DF24 (mixed static and dynamic): the proposed method reaches 0.7329 ROC-AUC, versus 0.6957 for SpeechForensics and 0.6890 for AuViRe.

• Comparable, not superior, cross-dataset result on RW (static-heavy): the proposed method reaches 0.6510 ROC-AUC, versus 0.7819 for SpeechForensics and 0.6430 for AuViRe.

• Ensembles help. Ours & SpeechForensics reaches 0.7786 on DF24 and 0.7971 on RW; Ours & AuViRe reaches 0.7797 on DF24 and 0.6634 on RW.

• C50 statistics matter most. Removing the C50-based statistics (Var_C50, H_C50) causes the largest degradation: 0.6160 on DF24 (−0.1169) and 0.6090 on RW (−0.0420).

• SNR statistics contribute less on DF24. Removing the SNR-based statistics (Var_SNR, H_SNR) gives 0.7187 on DF24 (−0.0262) and 0.6528 on RW (+0.0018).

• Both variance and entropy carry signal. Removing variance-related features gives 0.6646 on DF24 (−0.0683) and 0.6246 on RW (−0.0264); removing entropy-based features gives 0.7097 on DF24 (−0.0232) and 0.6021 on RW (−0.0489).

• The variance ratio adds information. Removing the variance ratio r_var gives 0.7136 on DF24 (−0.0193) and 0.6495 on RW (−0.0015).

• Sound separation is essential. Using raw audio instead of SAM-Audio drops ROC-AUC to 0.5945 on DF24 and 0.5671 on RW, compared to 0.7329 and 0.6510.

• Qualitative trend confirmed. In authentic videos, SNR and C50 increase as the speaker moves closer to the camera; in deepfake videos this distance–acoustic trend is inconsistent.

Methodology in Plain English

The method rests on a simple physical fact: as a sound source moves farther from a microphone, its intensity drops roughly with the square of the distance. The authors do not try to measure absolute loudness, because real recordings have automatic gain control and device normalization. Instead they use two ratio-based acoustic measures that are robust to those effects:

• SNR, which compares the received speech energy to environmental noise energy.
• C50, which compares the energy arriving in the first 50 ms (direct sound and early reflections) to the energy arriving later (late reverberation).

The pipeline works as follows. Speech is separated from background noise using SAM-Audio with the text prompt "speech." SNR and C50 are then computed over 300 ms temporal windows, a length chosen to cover typical syllable durations while capturing reverberation.

On the video side, RetinaFace detects facial landmarks to localize the speaker, and MiDaS produces a dense depth map. Averaging the depth values inside the facial region gives an estimated speaker distance, and the inverse of that value is used as the relative distance proxy.

The distance estimates are then used to group the acoustic measures into bins. Within each bin, the authors compute the variance and entropy of SNR and C50, and take the expectation over bins, yielding five features: Var_SNR, Var_C50, H_SNR, H_C50, and the variance ratio r_var. The intuition is that deepfakes produce more dispersed acoustic measures within a distance bin (higher variance) and reduced acoustic variability (lower entropy), while real recordings follow the expected attenuation trend. A multi-layer perceptron with a single hidden layer maps this five-dimensional vector to a scalar score, where values closer to 1 indicate fake and values closer to 0 indicate real.

Training is done on the DF24 training set — fine-tuning the baseline detectors and training the proposed detector from scratch — with evaluation on the DF24 test split (in-domain) and the RW test set (cross-dataset). DF24 contains 300 training and 180 test videos; RW contains 330 test videos; DF-Dynamic contains 40 curated test videos. Both datasets are collected from media platforms and span varying camera motion, environmental noise, and editing artifacts. Samples with missing modalities or invalid links were excluded for reproducibility.

Why This Matters

Impact on research. The paper opens a detection direction that does not depend on the mouth region at all, which matters because lip-sync-based detectors are reported to reach ROC-AUC above 0.95 on relatively static benchmarks but degrade when faces are occluded or moving. It also shows the physics cue is complementary: score-averaging ensembles improve results on both DF24 and RW, suggesting future detectors can combine appearance cues with physical ones.

Real-world applications:

• Social media and platform moderation, where user-uploaded videos are frequently handheld, moving, and uncurated rather than studio-quality.
• Journalism and newsroom verification, where a reporter may need to screen footage of an on-the-street interview or a vlog where the face is not always visible.
• Video conferencing and remote identity verification, where callers move relative to the camera and lighting and framing vary.
• Legal and forensic evidence review, where the provenance of submitted video or audio recordings is contested.

Industry relevance. Deepfake detection is relevant to content platforms, media forensics vendors, and security providers. Because the method is lightweight at the decision stage — a five-dimensional feature vector fed to a single-hidden-layer MLP — and because it complements rather than replaces existing detectors, it is plausible as an additional signal layered onto existing moderation pipelines rather than a standalone replacement.

Future Directions

• Addressing close-microphone and noise instability. The authors note that microphones placed close to the speaker weaken the distance–speech relationship, and that rapidly changing background noise destabilizes the physics-aware acoustic measurements.
• Reducing dependence on depth and face estimation. Errors in depth and face estimation can degrade final performance, so more robust distance estimation is an open problem.
• Extending beyond human speech. The framework is described as speech-agnostic and may extend to general object-level scenarios that follow similar acoustic propagation principles.
• Broadening benchmark coverage. Evaluation here centers on DF24, RW, and the 40-video DF-Dynamic subset; larger and more diverse dynamic-video benchmarks would test how broadly the physical cue holds.

Target Audience

Researchers and practitioners in audio-visual deepfake detection and media forensics; computer vision engineers working on multimodal video understanding; and acoustics or audio processing researchers interested in applying propagation physics to media authenticity. It is also useful for industry teams building moderation or verification pipelines who want a detection signal that does not depend on visible lip regions.

Authors’ abstract

Recent progress in generative models has made talking-head deepfakes highly realistic and easy to create, jeopardizing public trust. However, most audio-visual deepfake detectors rely on lip-speech synchronization and are designed for static, frontal videos, limiting their reliability in dynamic, in-the-wild recordings where the speaker moves and lip cues are degraded or missing. As an alternative, we propose a physics-aware detector for dynamic speaking videos that leverages an acoustic constraint: in real recordings, speech energy varies predictably with speaker-to-camera distance, whereas deepfake generation can break this coupling. Our method combines distance estimates from video with distance-dependent acoustic measures from speech to detect deepfakes. Experiments on in-the-wild benchmarks show that the proposed physical cue is effective in dynamic scenarios, and that it complements lip-sync-based detectors, with a simple ensemble achieving consistent gains across datasets. Codes are available at https://byulharang.github.io/PADD/.