Investigating self-supervised representations for audio-visual deepfake detection

AI summary

Overview

• Research area: Audio-visual deepfake detection, specifically the evaluation of self-supervised (SSL) feature representations as frozen backbones for detecting manipulated video.
• Technical level: Intermediate. The paper assumes familiarity with self-supervised learning, linear probing, AUC evaluation, and deepfake detection datasets, but its arguments are empirical rather than mathematically dense.
• Scope: A systematic, multi-faceted study of twelve pretrained self-supervised encoders (audio-only, visual-only, and audio-visual) evaluated on four deepfake datasets for detection, anomaly detection, interpretability, and cross-feature complementarity.

Note: the paper content provided is truncated — it ends mid-sentence in Appendix D (the description of SpeechForensics). Appendix tables referenced in the text (Tab. 5, Tab. 6, Fig. 7) are not included in the available content.

What This Paper Is About

Deepfake detectors increasingly rely on large pretrained self-supervised models, but it is unclear which representations actually capture forensic cues versus which exploit dataset shortcuts. Existing work typically tests one backbone in isolation, either as a frozen feature extractor or buried inside a complex architecture, so results across studies are hard to compare. This paper evaluates a broad set of audio, visual, and audio-visual SSL features under a deliberately minimal classifier (linear probing) to determine how useful, how interpretable, and how complementary these representations are for audio-visual deepfake detection.

Key Contributions

1. A systematic multi-modal, multi-domain benchmark. The authors evaluate twelve self-supervised representations spanning audio-only (Wav2Vec XLS-R 2B, Auto-AVSR (ASR), AV-HuBERT (A), BRAVEn (A)), visual-only (CLIP ViT-L/14, FSFM, Video-MAE-large, Auto-AVSR (VSR), AV-HuBERT (V), BRAVEn (V)), and audio-visual (Auto-AVSR, AV-HuBERT) features across four datasets: FakeAVCeleb, AV-Deepfake1M, AVLips, and DeepfakeEval 2024.

2. Adaptation of linear probing to the video domain with inherent explainability. A per-frame linear classifier is combined with log-sum-exp pooling, which approximates a max operation over frames, so a video is flagged fake if any single region is fake. Because the classifier is linear, the same weights yield frame-level temporal scores and patch-level spatial maps (via feature averaging or Grad-CAM).

3. A multi-faceted evaluation protocol that goes beyond supervised testing. Two anomaly detection proxy tasks trained only on real data — next-token prediction and audio-video synchronization — plus temporal localization against AV-Deepfake1M annotations, spatial alignment against human click annotations in ExDDV, and correlation/fusion analyses.

4. Evidence that standard supervised evaluation is vulnerable to dataset shortcuts. Randomly initialized AV-HuBERT models score far above chance on classification but fall to chance level on synchronization and to moderate values on next-token prediction, exposing spurious cues such as leading silence.

Main Findings

• Many features encode deepfake information in-domain. Linear probes reach strong AUC on the scientific datasets. On AV-Deepfake1M, Wav2Vec2 scores 100, AV-HuBERT (A) 100, AV-HuBERT (V) 93.7, and AV-HuBERT (audio-visual) 99.9; on FakeAVCeleb, AV-HuBERT (A), AV-HuBERT (V), and BRAVEn (V) all score 100. The paper notes deepfake cues can be picked from different angles: audio (Wav2Vec2, AV-HuBERT (A)), motion (Video-MAE), and static visual content (CLIP).
• Audio-informed features generalize best. On FakeAVCeleb and AV-Deepfake1M, audio representations lead. On AVLips and DeepfakeEval 2024, which lack or have incomplete audio manipulations, audio-informed visual models win: AV-HuBERT (V) reaches 98.3 / 90.5 (AVLips / DFE-2024) and BRAVEn (V) reaches 98.9 / 96.7. BRAVEn (V) has the best out-of-domain mean at 84.6%, followed by AV-HuBERT (V) at 80.0%.
• Simple linear probes reach state-of-the-art. Training on AV-Deepfake1M with 23k samples, BRAVEn (V) averages 91.1 (All4) and 90.5 (Last3), versus SpeechForensics (84.1 / 89.4), AuViRe (81.1 / 74.7), AVAD (71.5 / 77.7), AVFF (74.8 / 67.1), and RealForensics (75.2 / 80.1). The authors attribute SpeechForensics' competitive performance to its reliance on similar AV-HuBERT features and an anomaly-detection formulation.
• Randomly initialized models are not random. A randomly initialized AV-HuBERT (A) scores 99.8 on FakeAVCeleb in-domain and 97.8 out-of-domain, and AV-HuBERT (V) random scores 83.8 / 52.5. This indicates architectures implicitly encode discriminative but spurious information, which the paper links to cues such as leading silence.
• Anomaly detection is more robust but feature-selective. On random features, synchronization performance collapses to chance (50.0) and next-token prediction drops to 64.4 on AV-Deepfake1M. AV-HuBERT (A) features are essential for next-token prediction (90.6 AUC on AV-Deepfake1M), while AV-HuBERT (V) features are crucial for synchronization. The best anomaly setting, synchronization on AV-HuBERT (A) + AV-HuBERT (V), reaches 87.3 on AV-Deepfake1M and 96.3 on FakeAVCeleb.
• Temporal explanations capture manipulations, but audio models also watch silence. Localization performance is close to classification performance for most features, with large drops only for random models and FSFM. Qualitative examples show audio models attending to leading silence while still focusing on manipulated regions; Wav2Vec2 appears to select transition boundaries; AV-HuBERT (V) produces the cleanest predictions and CLIP the noisiest.
• Spatial alignment to humans is plausible but less precise than a learned predictor. A CLIP-based model trained on ExDDV reaches 71.3% AUC on its test set. Explanation MAE improves with model confidence and beats frame center (0.117 MAE) and random locations (0.270 MAE), but falls short of a predictive click model (0.055 MAE) and face center (0.058 MAE). The paper notes the predictive click model barely outperforms face center, suggesting human annotations may mainly indicate that artifacts occur somewhere on the face.
• Visual representations are more complementary than audio ones. Cross-model Pearson correlations are generally weak to moderate, with the strongest correlations within modalities. AV-HuBERT (V) correlates more with audio models than with other video models because it focuses on lip movements and was trained jointly with audio. Video-MAE gains the most from fusion, benefiting more from CLIP than from the more complementary AV-HuBERT (V).
• Realistic in-the-wild data remains hard. On DeepfakeEval 2024, the best out-of-domain result is 76.0% AUC (BRAVEn (V)) and the best in-domain result is 75.5% (BRAVEn (V)), which the authors interpret as intrinsic dataset difficulty rather than features latching onto superficial patterns.

Methodology in Plain English

The researchers take pretrained self-supervised models and freeze them, adding only a single linear layer on top. Each video frame (or chunk) is converted into an embedding; the linear layer scores each embedding; then a log-sum-exp pooling function aggregates frame scores into a single video score, approximating a "max" so that one suspicious moment can flag the whole video. Once trained, the same linear weights can be read back out as per-frame scores for temporal explanations, and propagated to patch level (or supplemented with Grad-CAM) for spatial heatmaps.

To check whether models are leaning on shortcuts, the authors add two tasks trained only on real videos. In next-token prediction, a small decoder-only Transformer (4 layers, 4 heads, feature dimension 512, feedforward dimension 1024) predicts the next frame's representation from prior frames using mean squared error, and the maximum frame-level error becomes the fakeness score. In audio-video synchronization, a four-layer MLP scores whether an audio frame matches its corresponding video frame rather than neighboring frames within a temporal neighborhood of 30 frames; misalignment indicates manipulation. Training details include 100 epochs with early stopping (patience 10) and Adam at a learning rate of 10⁻³ for linear probing, and 50k real AV-Deepfake1M videos (45k train, 5k validation) for the anomaly tasks.

Evaluation uses AUC (average precision is reported in Appendices E.1, Tab. 5 and Tab. 6, which are not included in the provided content). For temporal localization, per-frame scores over AV-Deepfake1M test videos containing at least one fake segment are treated as predictions against annotated fake segments, with AUC averaged across fake videos. For spatial explanations, Grad-CAM maps from a CLIP model are compared to human click annotations in ExDDV using mean absolute error. Finally, Pearson correlations between model predictions and downstream fusion performance quantify complementarity.

Why This Matters

Impact on research. The paper challenges the standard practice of reporting supervised AUC as evidence that a model has learned forensic cues. By showing that randomly initialized AV-HuBERT reaches 99.8 AUC on FakeAVCeleb in-domain and 97.8 out-of-domain, it demonstrates that benchmark scores can be driven by dataset shortcuts rather than manipulation detection. It also argues that frozen self-supervised features matter more than classifier capacity: a Transformer backend gave similar results to a linear probe.

Real-world applications:

• Content moderation platforms screening user-uploaded video for synthetic speech or face manipulation.
• Newsroom and fact-checking workflows that need explainable evidence — frame-level or region-level — for why a clip was flagged.
• Platform trust-and-safety pipelines that require robustness to distribution shift, since in-the-wild data (DeepfakeEval 2024) is the hardest setting tested.
• Fraud and identity-verification systems handling video-based KYC or remote interviews, where lip-sync manipulation is the primary threat.

Industry relevance. The finding that simple linear probes on off-the-shelf features beat more complex detectors is directly useful for teams who need deployable, inexpensive detectors rather than bespoke architectures. The released project page (https://bit-ml.github.io/ssr-dfd) and the paper's explicit description of checkpoints support reproducibility.

Future Directions

• Closing the in-the-wild gap. Neither the tested representations nor the compared state-of-the-art approaches generalize well to DeepfakeEval 2024; the authors call for more specialized solutions that account for missing modalities and shifts in the video domain.
• Stronger anomaly detection through better feature selection. Next-token prediction requires AV-HuBERT (A) and synchronization requires AV-HuBERT (V); identifying or designing features that jointly satisfy both remains open.
• Better use of human annotations. Since the predictive click model barely beats face center prediction, it is unclear how much localization signal ExDDV annotations actually carry, which limits how tightly spatial explanations can be validated.
• Principled feature fusion. The paper finds that complementarity alone does not predict fusion gains — Video-MAE benefits more from CLIP, with which it is more aligned, than from the more complementary AV-HuBERT (V) — leaving the design of fusion strategies an open question.

Target Audience

Researchers and engineers working on media forensics, deepfake detection, and multimodal representation learning, particularly those choosing backbone features for a detector or benchmarking against FakeAVCeleb, AV-Deepfake1M, AVLips, and DeepfakeEval 2024. It is also relevant to evaluation-methodology researchers interested in spurious correlations and shortcut learning, and to practitioners in trust and safety who need to interpret model decisions rather than only report aggregate accuracy.

Authors’ abstract

Self-supervised representations excel at many vision and speech tasks, but their potential for audio-visual deepfake detection remains underexplored. Unlike prior work that uses these features in isolation or buried within complex architectures, we systematically evaluate them across modalities (audio, video, multimodal) and domains (lip movements, generic visual content). We assess three key dimensions: detection effectiveness, interpretability of encoded information, and cross-modal complementarity. We find that most self-supervised features capture deepfake-relevant information, and that this information is complementary. Moreover, models primarily attend to semantically meaningful regions rather than spurious artifacts (such as the leading silence). Among the investigated features, audio-informed representations generalize best and achieve state-of-the-art results. However, generalization to realistic in-the-wild data remains challenging. Our analysis indicates this gap stems from intrinsic dataset difficulty rather than from features latching onto superficial patterns. Project webpage: https://bit-ml.github.io/ssr-dfd.