SARA: Stress Test Reasoning in Audio Deepfake Detection

AI summary

Overview

• Research area: Explainable audio deepfake detection (ADD), Audio Language Models (ALMs), chain-of-thought reasoning, and adversarial robustness of reasoning traces.
• Technical level: Advanced.
• Scope: A diagnostic framework named SARA (Shift Analysis of Reasoning in Audio) that stress-tests five open-source ALMs across acoustic perception, reasoning-verdict coherence, and reasoning-verdict dissonance under both acoustic and linguistic adversarial attacks.

What This Paper Is About

Binary deepfake detectors output only a "real" or "fake" label, which is insufficient in high-stakes settings such as forensic audio analysis and legal proceedings where auditable, step-by-step explanations are required. Reasoning-capable Audio Language Models promise "glass-box" transparency by generating intermediate explanations, but the paper asks whether that reasoning actually supports the verdict, and how it behaves when the audio is deliberately perturbed. SARA is introduced to measure whether ALM reasoning remains grounded, coherent, and diagnostically useful under adversarial strain.

Key Contributions

1. A three-tier diagnostic framework (SARA) that evaluates ALM reasoning along three dimensions tied to three research questions: RQ1 acoustic perception (does the model accurately perceive the raw audio signal?), RQ2 reasoning-verdict coherence (do intermediate reasoning steps entail the final predicted label?), and RQ3 reasoning-verdict dissonance (does the reasoning trace show internal conflict even when the classification head is deceived?).
2. A modality-specific characterization of adversarial degradation, showing that acoustic and linguistic perturbations produce fundamentally different statistical shifts: acoustic attacks cause an average coherence degradation of 14.20% (p < 0.0001) and measurable dissonance increases (up to a +19.4% shift in Phi-4-multimodal), while linguistic attacks achieve a higher aggregate Attack Success Rate (61.73% vs. 47.38% for acoustic attacks) with only a 3.91% average coherence drop.
3. An adversarial detection method that uses reasoning traces alone, training a binary classifier on reasoning coherence and perception metrics to reach up to a 77.59% F1 score in detecting perturbed audio without accessing the raw acoustic signal.
4. A reasoning-data synthesis protocol (iterative supervised fine-tuning and generation, motivated by DeepSeek-R1-style distillation and Audio-Reasoner-style audio-conditioned QA) used to adapt the binary ASVspoof 2019 dataset for reasoning tasks, analyzed for sufficiency and generalizability.

Main Findings

• Reasoning trades accuracy for explanation in most models: Chain-of-thought (RSN) mode degrades overall accuracy by an aggregate of 3.15% (p = 0.069) for Granite, Phi-4, and Gemma. The loss stems mainly from bona fide speech detection, with Real F1 declines of −17.57% for Gemma and −21.0% for Granite, indicating a bias toward false positives.
• One clear outlier: Audio-Flamingo-3 improves substantially with explicit reasoning, rising from 92.11% to 97.00% accuracy (Real F1 72.35% to 87.30%). Qwen2-Audio remains stable across modes (∼98.0% accuracy). Traditional baselines set a high bar, with AASIST-2 at 99.58% accuracy and CLAD at 98.78%.
• Acoustic grounding is weak across the board: The best model, Qwen2-Audio, achieves a mean perception accuracy of only 49.40%. Averaged across five models, sensitivity is lowest for Speed (mean 40.54%) and strongest for Liveliness (mean 54.24%).
• Perception correlates with reasoning performance in conflicting directions: Liveliness perception is marginally positively correlated with RSN-mode accuracy (r = 0.827, p = 0.084), while Disfluency perception shows a statistically significant negative correlation (r = −0.945, p = 0.015).
• Explicit reasoning barely changes perception: A paired t-test across all perception metrics shows a mean absolute decrease of 1.67% for RSN models, which is not statistically significant (t = 1.9104, p = 0.07).
• Acoustic attacks break coherence and expose internal conflict: Under acoustic perturbation, reasoning-verdict coherence falls by an average of 14.20% (p < 0.0001). Dissonance rises by 5.3 points for Audio-Flamingo-3, 10.6 points for Granite-3.3-8b, and 19.4 points for Phi-4-multimodal, suggesting the reasoning layer retains a record of acoustic anomalies even when the classification head is bypassed. RSN mode shows consistently higher ASR than NON mode (paired t-test, p = 0.0003).
• Linguistic attacks deceive quietly: Under linguistic perturbation, explicit reasoning marginally reduces aggregate ASR from 71.75% to 61.73% (p = 0.27), with negligible coherence change (ΔΦCoh = −3.91, p = 0.33) and dissonance change (ΔΨDiss = −4.03, p = 0.143). Linguistic attacks do not trigger the same "reasoning friction" as acoustic attacks.
• Attack detection from reasoning traces works: LightGBM achieves 77.59% aggregate F1 with 76.72% precision and 78.48% recall. Linguistic attacks are detected more accurately than acoustic ones (87.10% vs. 76.97% F1). Logistic regression over 87,560 samples (35 ALM-attack pairs) shows significant negative coefficients for prosody coherence (β = −0.266, p < 0.001), disfluency (β = −0.145, p < 0.001), and liveliness (β = −0.135, p < 0.001); when Prosody reasoning erodes from coherent to incoherent, the probability of an adversarial attack increases by 6.07%.
• Findings generalize across synthetic data sources: On a transfer dataset synthesized with Audio-Flamingo-3, the divergence between attack modalities persists, with Qwen2-Audio dropping to 95.9% and Granite to 98.9%. However, the adversarial detector shows sensitivity to the synthetic generator, yielding a low zero-shot F1 of 53.65%.

Methodology in Plain English

The researchers take a binary deepfake dataset (ASVspoof 2019) and turn it into a reasoning task by synthesizing chain-of-thought explanations, following a human reasoning taxonomy with six aspects: Prosody, Disfluency, Speed, Speaking Style, Liveliness, and Quality. That taxonomy came from analyzing 24,240 open-text responses from 1,212 participants, reconciled by three raters to a consensus of κ = 0.82.

They then prompt five open-source ALMs — Qwen2-Audio-7B-Instruct, Phi-4-multimodal-instruct, gemma-3n-E4B-IT, granite-speech-3.3-8B, and Audio-Flamingo-3-HF — in two modes: a non-reasoning mode that emits direct verdicts and a reasoning mode that emits explanations before verdicts. Traditional detectors (AASIST-2, RawNet-2, CLAD) provide comparison baselines.

Three measurements are computed. First, acoustic perception: assemble an expert-annotated benchmark from MMSU and MMAU-Pro, ask the model questions about specific acoustic attributes, and score whether answers match ground truth. Second, coherence: use a verification function to check whether each reasoning aspect logically entails the final verdict, regardless of whether that verdict is correct. Third, dissonance: restrict attention to cases where the final verdict is wrong, and measure how often the reasoning still flags anomalies that contradict that wrong label.

To stress-test these, the authors attack audio two ways. Linguistic attacks use the TAPAS framework with TextFooler word substitutions converted back to speech via Kokoro TTS, across four sub-attacks combining gender (male, female) and accents (American, British). Acoustic attacks use three CLAD protocols: Background Noise, Time & Pitch, and Shape & Space. Comparisons between original and perturbed conditions are expressed as shifts (ΔΦ for coherence, ΔΨ for dissonance) alongside Original Accuracy and Attack Success Rate. Finally, they train classifiers on the reasoning metrics to test whether perturbation can be detected from the text of the reasoning trace alone.

Why This Matters

This work reframes audio deepfake detection from a silent binary judgment into an auditable reasoning process, and shows that reasoning traces carry a diagnostic signal even after the final verdict has been compromised. It also documents a real trade-off: adding chain-of-thought can hurt detection accuracy on bona fide speech, and it can increase susceptibility to acoustic manipulation. The paper situates itself against regulatory expectations such as the amended US Federal Rule of Evidence 702 and the EU AI Act's transparency and explainability mandates, which require automated outputs to be auditable.

Real-world applications:

• Forensic audio analysis and legal proceedings, where analysts need step-by-step justifications and a fail-safe signal when a detector is being evaded.
• Content moderation and media integrity pipelines, using reasoning-trace statistics as a lightweight detection signal that does not require re-processing the raw acoustic waveform.
• Red-teaming and evaluation of audio language models, providing attack protocols and audit metrics to benchmark robustness before deployment.
• Emotion recognition and related speech tasks, since the authors note that dimension-specific reasoning vulnerabilities may transfer to adjacent speech tasks.

Industry relevance: the involvement of Cisco Research and the focus on mid-sized, open-source ALMs (~7B–8B parameters) makes the framework directly applicable to deployable systems. The finding that linguistic attacks are harder for standard classifiers but leave a distinct signature in reasoning traces is a practical insight for teams building defense-in-depth around ALM-based detectors.

Future Directions

• Language coverage: Experiments are exclusively on English-language datasets; generalization to non-English syntactic and morphological structures remains an open question.
• Model scale and scope: The study covers mid-sized ALMs (approximately 7B–8B parameters) and a small subset of traditional detectors; large omni-models such as Qwen3-Omni-30B-A3B-Instruct are not examined, nor is an exhaustive list of legacy classifiers.
• Human-annotated reasoning: The framework relies on AI-synthesized chain-of-thought traces, which bounds generalizability. The authors call for validating reasoning-based defenses with larger, human-curated audio reasoning datasets, while noting that their hybrid human-labeled seeds did not outperform pure synthetic generation.
• Detector transferability: The adversarial detector's low zero-shot F1 of 53.65% on the AF3-synthesized transfer set suggests that aligning training data with the target model's output distribution is needed for robust deployment.

Target Audience

Researchers and practitioners working on audio deepfake detection, explainable AI, adversarial robustness, and audio-language model evaluation. It is also relevant to forensic analysts, trust-and-safety teams, and policy-facing engineers who need to assess whether a model's stated reasoning can be trusted, and to anyone designing red-teaming benchmarks for multimodal reasoning systems.

Authors’ abstract

Audio Language Models (ALMs) offer a promising shift towards explainable audio deepfake detections (ADD), moving beyond \textit{black-box} classifiers by providing transparency to their predictions via reasoning traces. However, such reasoning may not support the model predictions, reflecting poor coherence, or, worse, may rationalize incorrect predictions with plausible but misleading explanation. Moreover, the behavior of ALM reasoning under adversarial attacks remains under-explored, raising questions about the practical reliability of such explanation capabilities. To address this gap, this study introduces \textbf{SARA} (\textbf{S}hift \textbf{A}nalysis of \textbf{R}easoning in \textbf{A}udio), a diagnostic framework that evaluates ALM reasoning across three dimensions: acoustic perception, reasoning-verdict coherence and dissonance. We test five open-source ALMs against both acoustic and linguistic adversarial attacks. We show that acoustic attacks significantly degrade reasoning-verdict coherence (average decrease of 14.20\%), frequently inducing internal logical conflicts. Conversely, linguistic attacks achieve higher attack success rates while maintaining reasoning coherence. We further demonstrate that the textual coherence of generated reasoning traces also serves as a latent indicator of adversarial inputs, enabling effective detection of perturbed audio (0.78 in F1) \textit{without accessing the raw acoustic signal}. These findings suggest that reasoning traces provide diagnostic utility that persists even when final classification outputs are compromised.