VeriSpoof: Beyond Real or Fake with One Audio LLM

AI summary

Overview

Research area: Speech forensics and audio language models (audio deepfake detection, source attribution, and explainable AI), submitted to arXiv under eess.AS.

Technical level: Advanced. The paper assumes familiarity with self-supervised speech encoders, LoRA/DoRA adapters, equal error rate (EER) evaluation, and open-set verification protocols, though the core ideas are explained in accessible terms.

Scope (one sentence): The paper presents VeriSpoof, a single audio language model built on a Qwen3-ASR encoder and a MiniCPM5-2B LLM that detects spoofed speech, attributes its generator, describes recording conditions, localises synthesized spans, and verifies whether two recordings share a source — while attaching rationales whose statements can each be checked against the audio or labels.

What This Paper Is About

Existing speech deepfake detectors return a single score saying how likely a recording is synthetic, but a forensic analyst also needs to know which system produced it, whether two recordings came from the same source, and on what evidence the conclusion rests. Current open detectors answer one question per model, and the explanations offered by existing audio language models are unchecked free text that may be fluent but false or irrelevant to the decision. VeriSpoof's goal is to unify all of these forensic questions in one set of weights and make every explanation independently verifiable and causally tied to the model's verdict.

Key Contributions

1. One model, five forensic tasks, no task-specific heads. Detection, generator attribution (generator, acoustic model, vocoder), recording-condition analysis, synthesized-span localisation and source verification are all posed as questions with candidate answer sentences, and the likelihood of each sentence is read out as a score via a single log-likelihood-difference formula.

2. Grounded, checkable rationales. Each rationale is a fixed-field sequence (observations, channel_status, localization, source_profile, then a decision written last), where every statement has a checker outside the model: waveform measurements for observations, augmentation logs for channel status, labelled spans for localisation, and generator metadata for the source profile.

3. Counterfactual training for faithfulness. In 20% of rationale examples the stated localisation is reversed and only the decision is trained, so the verdict follows the stated evidence rather than being written beside an independently reached conclusion.

4. A released evaluation suite and models. VeriSpoof-Bench contains 54 test subsets with 177,852 items covering detection, attribution, source verification, recording conditions, grounded reasoning and faithfulness, released alongside VeriSpoof-Base, VeriSpoof-Plus, inference code with a vLLM backend, and a demo.

Main Findings

• Best open-weight detector on Speech-DF-Arena. VeriSpoof-Plus reaches 2.14% pooled EER and 1.79% mean EER across the 14 sets (2.34 M trials), ranking third of 28 systems on the public snapshot of 31 Oct 2026 and first among open-weight systems; VeriSpoof-Base, trained only on the official training splits of eight corpora (about 0.38 M utterances), reaches 4.43% pooled, against 9.52% for the next-best open-weight system (DF-Arena-1B).

• The same weights verify sources. On pairs from generators unseen in training, VeriSpoof-Plus decides whether two recordings share a source at 6.17% EER for new models of known architectures and 15.00% for new architectures, versus 26.5% and 24.8% for a logistic-regression pair probe on the frozen DF-Arena-1B anti-spoofing encoder.

• Representation decides detection; the LLM decides comparison. With the encoder fixed, changing how its features are used lowers ASVspoof 5 EER from 9.6% to 3.1% while source verification stays at 16–19%; swapping the LLM moves detection by only 0.5 points but source verification by 5.5 points. Linear probes recover the acoustic model from encoder states at 0.84–0.90 accuracy, yet answers range from chance (Qwen2.5-1.5B, 0.52) to 0.86 (MiniCPM5-2B), so the information reaches the LLM in every run but is only sometimes composed into an answer.

• Source supervision is necessary for neural-codec fakes. At a matched number of authenticity labels, source supervision helps on 8 of 16 sets and hurts on one; with five times more authenticity labels instead, CodecFake+ EER stays at 12.6% against 2.2%.

• Rationales are checkable and useful. SNR and level statements are correct for 88–100% of recordings; the decision reached after writing a rationale catches partial fakes at 99.6% on PartialEdit versus 78.4% for the one-step answer, and catches unfamiliar fakes the direct answer misses.

• Faithfulness needs training for it. With counterfactual training the model follows a reworded, inverted localisation in 99.5% of cases across held-out edit forms, against 73.8% without it.

• General audio LLMs cannot answer these questions zero-shot. Across six general audio LLMs prompted with closed options, same-source accuracy averages 48.4–53.3% and recording-condition accuracy 16.4–22.6% (near chance); authenticity averages 52.1–64.7%. On the same items Base and Plus reach 95.9/96.5% on authenticity, 77.9/82.8% on same source and 76.9/83.0% on recording condition.

• More data sharpens the fingerprint but costs calibration. From Base to Plus, 9 of 14 common Speech-DF-Arena sets improve (Codecfake 4.97→0.12, Fake-or-Real 5.39→0.09) while three get worse (ASVspoof 5 2.59→3.40, In-the-Wild 1.20→1.90). On SpoofRadar, Plus has the better pooled EER (4.04% vs Base's 4.23%) but a worse mean (1.65% vs 3.76%), because at the pooled threshold 92.9% of XMAD-Bench's genuine Russian audiobook recordings are judged spoofed, even though within each language the classes separate at ≤1.95% EER.

• Cheap to train and run. Only 29.7 M of the 2.85 B parameters are trained; every training stage fits a single 24 GB GPU with gradient checkpointing, and the authors estimate training fits a 12 GB GPU and inference an 8 GB GPU. Base trains for 8k/20k/10k steps at batch 32 on one 48 GB GPU in about 14 GPU-hours.

Methodology in Plain English

The system hears an 8-second window of audio through a frozen Qwen3-ASR encoder, lightly adapted with DoRA adapters only in the first training stage. A learned weighted sum over the encoder's 25 layers feeds an audio tokenizer that produces two views of the recording: 16 "utterance tokens" that capture what is constant across the clip (useful for attribution and verification) and 52 frame tokens covering 0.16 s each, with text time markers every 0.8 s so the model can state spans in seconds. For a pair of recordings, the tokens of both are concatenated in the prompt.

A MiniCPM5-2B language model with LoRA adapters reads these tokens plus a question and produces an answer. Detection and source verification are not separate classifiers: the model is simply asked a yes–no question with two full-sentence answers, and the difference in log-likelihood between the positive and negative sentences becomes the detection score, which can be thresholded. Two small auxiliary losses (a one-class authenticity loss on pooled audio tokens and a per-token genuine/synthesized head) support perception and give a free synthetic-speech profile over time.

Training runs in three stages: perception alignment (representation learning with the LLM frozen), forensic question tuning (multi-task instruction tuning over all perception questions), and grounded rationale tuning (rationales plus counterfactual examples at a lower learning rate, with the earlier question-answering tasks replayed). Rationale targets are computed by the data loader from the exact augmented window the model receives, so every statement is true of its input by construction, and the decision is always written last and must follow the stated localisation. Faithfulness is then tested by editing statements the model never saw edited in training — paraphrased localisations, inserted spans, and contradicted channel or observation statements — and checking whether the decision moves appropriately (for evidential statements) or stays put (for non-evidential ones).

The authors also run controlled studies that change one factor at a time on identical test trials: swapping the LLM, changing how encoder features are used, and comparing source supervision against equal numbers of authenticity labels.

Why This Matters

Forensic conclusions that cannot be explained cannot be contested, and a single uncalibrated score is weak evidence in a real investigation. This work shows that one open model can answer a whole chain of forensic questions and that its explanations can be audited statement by statement, which changes what "explainable deepfake detection" can claim. It also isolates where performance actually comes from — the audio representation for detection, the language model for comparing recordings — which redirects effort in a field that often assumes the LLM is doing the heavy lifting.

Real-world applications:

• Journalism and fact-checking: triaging a leaked voice message, identifying the likely generator, and linking it to other recordings already in a case.
• Platform trust and safety: grouping detected synthetic uploads by tool family to trace a campaign rather than flagging clips one by one.
• Legal and insurance forensics: producing a decision whose supporting observations (SNR, clipping, bandwidth, synthesized spans) can be independently re-measured and challenged.
• Audio editing and archiving: catching partially edited speech — the case where a one-step verdict is most likely to fail — via synthesized-span localisation.

Industry relevance: the reported training footprint (under 30 M trained parameters of 2.85 B, an estimated 12 GB training and 8 GB inference budget) makes an open forensic model deployable on commodity hardware, and the released benchmark gives vendors and auditors a shared protocol rather than each publishing on private splits.

Future Directions

• Cross-domain threshold calibration. The SpoofRadar results show that added data shift score distributions on unseen genuine speech (92.9% of Russian audiobook recordings judged spoofed at the pooled threshold) even when within-language separability is preserved; per-domain threshold setting is identified as necessary but not solved.

• Degraded and re-recorded audio. Source verification on noisy or codec-processed copies remains hard at 23–30% EER, so robustness of the "source fingerprint" under channel effects is an open problem.

• Attribute composition under unseen pairings. Naming an acoustic model seen in training but paired with an unseen vocoder drops from 84.7% to 69.8% when moving to the larger Plus model, and the composition study shows the answer depends heavily on which LLM is used — how to make composition reliable across backbones is unresolved.

• Unseen architectures and broader generator coverage. Verification at 15.00% EER on wholly new architectures versus 6.17% on new models of known architectures suggests cues are tied to architecture families; extending the balanced-source training recipe (211 spoof sources in Plus) is the natural next test.

Target Audience

Researchers and engineers working on speech anti-spoofing, audio forensics and audio language models; benchmark maintainers and evaluation scientists interested in faithfulness testing for audio explanations; and practitioners in trust-and-safety, media verification or legal audio analysis who need auditable rather than single-score detection. Readers seeking a gentle introduction will find the architecture and task framing readable, but the results sections assume comfort with EER, open-set protocols and parameter-efficient fine-tuning.

Authors’ abstract

A speech deepfake detector estimates whether a recording is synthetic, but a forensic analyst also asks which system produced it, whether recordings share a source, and what evidence supports each conclusion. We present VeriSpoof, an audio language model that unifies five tasks with shared weights: deepfake detection, generator attribution (generator, acoustic model and vocoder), recording-condition analysis, synthesized-span localisation, and source verification. Each decision is supported by grounded rationales whose statements can be independently verified. Answers are scored by sentence likelihood, allowing detection and source verification to be thresholded without separate heads. Rationale statements can be checked against waveform measurements, span labels, or generator metadata. Counterfactual examples train decisions to follow the stated evidence. Across the 14 Speech-DF-Arena test sets, VeriSpoof-Plus achieves 2.14% pooled EER, ranking third of 28 systems and first among open-weight systems. VeriSpoof-Base, trained only on official training splits of eight corpora, achieves 4.43% EER, less than half that of the next open-weight detector. With the same weights, VeriSpoof-Plus verifies whether recordings from generators unseen in training share a source at 6.2% EER. Controlled studies show that the audio representation governs detection, the LLM governs recording comparison, and source supervision is necessary for neural-codec fakes. Statement-checkable rationales catch partial fakes missed by one-step answers (99.6% vs. 78.4% on PartialEdit) and determine decisions under edits unseen in training (99.5%). Training updates only 29.7M of 2.85B parameters, fitting a single 24 GB GPU with gradient checkpointing. We release both models, inference code, a demo, and VeriSpoof-Bench, the evaluation suite covering these questions.