Replay, Synthetic Speech, and Anti-Spoofing

Anti-spoofing asks a narrow question: is this audio sample bona fide, under a defined capture and attack model? The word defined is carrying the sentence. Somebody had to name in advance what an attack could be — replay, speech synthesis, voice conversion, codec manipulation, an injected signal. The detector only ever learns to answer inside that list. It never answers alone, either. Its score goes to a speaker verification system that acts on it. So the two have to be evaluated as one thing.

No detector, then, can certify every future recording as genuine. Attackers adapt. Generative models change. Replay paths vary. And an ordinary channel shift can resemble a spoof artifact when nobody is attacking at all. A stronger detector matters less than the work done before it: whether the attack surface was drawn widely enough to cover what the system will meet in service. Without that, the error and cost figures describe the threat model someone chose. Not the decision they are being used to support.

The rest of this lesson is mostly one measurement repeated in different places. An unchanged detector is scored first against the attacks it was trained on, then against attacks it was not. The gap between those two numbers is the whole subject.

That bet gets lost like this. One detector was scored twice and never changed in between. B01 was the CQCC-GMM baseline of the ASVspoof 2019 logical-access database. On the development set — the attacks it had trained on — it scored a pooled minimum t-DCF of 0.0123 and 0.43% EER. On the evaluation set, whose generators were mostly ones it had never heard, the same system scored 0.2366 min t-DCF and 9.57% EER. Nothing was altered but the question. The error rose roughly twentyfold. The database paper gives the cause in one line: “This was due to the presence of new attacks that were not included during CM training.”

A pooled headline hides even that. Per attack, B01's evaluation EER ran from 0.00% on A07 and on the known reference attack A16 up to 26.15% on the unseen A13. By the primary metric the costliest attack was a third one, A17, at 0.9820 min t-DCF with 19.62% EER. So the single figure 9.57% contains one generator the baseline caught perfectly and another it caught barely at all. Reporting only the average is what lets a system look uniformly competent when its competence is entirely conditional on lineage. The gap is workable rather than fatal. The top logical-access entry, team T05, reached 0.0069 min t-DCF and 0.22% EER — best of the 48 teams that entered that scenario, out of 63 across both scenarios in ASVspoof 2019. But nobody would know which situation they were in from a development-set score alone.

Doing that on purpose is a sequence of three stages, and each one lives on what the stage before it decided. Defining the attack surface is where the assumptions are made: which generators, which replay paths, which codecs count as attacks worth catching. Separating the attack assets is where those assumptions get spent. The generator families, replay devices and post-processing chains pushed to the evaluation side of the line are exactly the ones named at the first stage. That is the whole instruction in the title. Hold out the generator, not just the speaker. Six known systems on one side; eleven unknown ones plus two labelled reference attacks on the other. That is the separation carried out at the scale of a public database. ASVspoof 5's disjoint A01–A08, A09–A16 and A17–A32 is the same separation with 32 algorithms and eleven codec conditions instead.

Operating defensively is the third stage. It is where the assumptions meet an attacker who has read them. Anything the first stage left off the list arrives here with no check waiting for it. The cost of that omission is not an error rate. It is whatever the system was authorising.

Unseen-attack testing and layered controls both have to be written down somewhere, and the document they get written into is a protocol. Two of the four terms below are not this lesson's coinage but an international standard's. ISO/IEC 30107-3:2023 fixes bona fide versus attack presentation, and the paired error rates APCER and BPCER. APCER counts the attacks that got through — the “proportion of attack presentations using the same PAI species incorrectly classified as bona fide presentations in a specific scenario”, in the definition given by Busch and Thieme, the standard's editors. Note what the second edition then tells testers not to do. Do not collapse the pair into a single figure via EER or HTER. Report BPCER at a fixed APCER instead: BPCER20 at APCER 5%, BPCER100 at APCER 1%. Similar-sounding words imply different evidence, different units, different decision rights. And a protocol is public by design. Which makes it precisely what an attacker studies before deciding where to step outside it.

There is one artefact that works against all four, and ASVspoof shows what it looks like at full size. An unseen-attack evaluation is what lets another team check whether training and testing happened on the same spoof generator family. It is the only thing that lets you prove they did not.

Build it in the order of the three stages. Write down first what your definition of the attack surface assumes: which generators, which replay devices, which post-processing chains you have decided an attack can come from. Then find one counterexample to that list — a cloning service, a codec, a room you never named — and note that nothing in the current evaluation would catch it. ASVspoof 5's eleven codec conditions C01 to C11 exist because that counterexample is usually a channel rather than a voice. Then write down what operating defensively does on the day it arrives: which threshold moves, who is told, what gets stepped up, what gets rolled back.

Containment is the stage that gets skipped. There is a dated case where it was the only stage left, because no detector stood anywhere in the path. On 21 January 2024 a robocall campaign went to New Hampshire phones carrying a generative-AI deepfake of President Biden's voice, with spoofed caller ID. 9,581 calls were initiated; the Enforcement Bureau verified a sample of 3,000. On 26 September 2024 the Federal Communications Commission adopted a forfeiture order, released four days later. Its first paragraph reads: “We impose a penalty of $6,000,000 against Steve Kramer (Kramer) for effectuating an illegal robocall campaign that targeted potential New Hampshire voters two days before the state’s 2024 Democratic Presidential Primary Election”. The violation cited was of the Truth in Caller ID Act. A related federal suit, League of Women Voters of New Hampshire v. Kramer, was filed in the District of New Hampshire on 14 March 2024 and terminated on 21 November 2025. No equal error rate appears anywhere in that sequence. The consequence arrived as a regulator's dollar figure and a federal docket, on a timescale of months and years. That is the timescale a containment plan is actually written against.

That leaves the question of what to report. Equal error and cost metrics under stated attack priors describe the detector on its own, and the object is smaller than it looks. The priors are the attack surface again, with numbers attached to it.

Tandem cost is not a figure of speech either. It is the tandem detection cost function, t-DCF, introduced by Kinnunen and Lee in 2018 and adopted as the primary ASVspoof metric from 2019. It carries six parameters: false-alarm and miss costs for the countermeasure, the same two for the speaker verifier, and the prior probabilities of target and spoof trials. The stated attack priors are literally two of the six, written into the metric rather than left in a footnote. It exists because of a limitation its authors put first in their own abstract: “First, the CM EER is not necessarily a reliable predictor of performance when ASV and CMs are combined.” A17 extracted 0.9820 min t-DCF from B01, against its 19.62% EER. That is the same sentence in numeric form. The two disagree about which attack was worst.

Three hard slices finish the portfolio. Attacks from generator families the detector never saw in training are the eleven-unknown condition applied to your own system. False rejection of genuine users is the legitimate redesign the banknote detector turns away, and ISO/IEC 30107-3:2023 tells you how to report it: BPCER at a fixed APCER, not folded into one number. And how long an incident takes to detect matters, because the fourth failure mode takes for granted that some attacks land. The forfeiture order arrived months after the calls did.