Speaker Embeddings, Verification, and Identification

Whatever the claim, what the system holds is a vector. Speaker recognition maps utterances to representations intended to preserve speaker-related cues. Those representations get put to one of three jobs. Verification compares a claimed identity against enrollment. Closed-set identification searches a population. Open-set identification searches a population and must also reject people who are not in it. Which of the three you are doing has to be settled before the architecture is.

Voice is variable, replayable, and synthesizeable. Health, emotion, channel, language and environment all move it. The same speaker scores differently across a head cold, a raised voice, a new handset, a switch of language, a noisy room. An embedding records what a microphone caught under those conditions. It is not something carried in the body.

What comes back at the end is a similarity score under a threshold somebody chose. Reading it as the probability that this is the enrolled person requires calibration. Calibration that has been checked, not assumed.

The scoring pipeline is not hypothetical either. The common speaker verification system in the ASVspoof 5 challenge was an ECAPA-TDNN encoder with cosine similarity scoring. An embedding, a cosine, a threshold. That is the architecture under discussion, and the number it emits is a cosine.

An embedding records what a microphone caught. A synthesised voice is built precisely to be what the microphone would have caught.

In March 2023 a data journalist opened his own Centrelink self-service account with AI-generated audio of his own voice and his customer reference number. He is Nick Evershed, at Guardian Australia. The voiceprint authentication he defeated was the one then offered by Services Australia and the Australian Taxation Office. The AI Incident Database catalogues the case as Incident 523, “Australian Journalist Able to Access Centrelink Account Using AI Audio of Own Voice”, incident date 15 March 2023, with both agencies named as deployers. The Guardian's Full Story covered it as “How an AI voice clone fooled Centrelink”.

The similarity score cleared the threshold. Nothing independent checked for liveness between the synthetic audio and the account. The score was the last gate.

A review that reads only false-match and false-non-match rates at the operating threshold would not have caught this. The standard governing those two rates says so itself. It is ISO/IEC 19795-1, the document NIST SP 800-63B-4 points to when it requires that “Biometric performance SHALL be tested in accordance with [ISO/IEC19795-1]”. Its scope clause draws the line: “Not within the scope of this document is the measurement of error and throughput rates for people deliberately trying to subvert the intended operation of the biometric system (e.g. by presentation attacks).” So the evidence everyone thinks to ask for is measured under a standard that has excluded the attacker by definition. That is the difficulty the rest of the lesson circles.

Every one of those questions assumes you already know which job the number describes. The three jobs cannot be cleared on one body of evidence.

Verification tests whether audio matches a claimed enrolled identity. Its errors are false matches and false non-matches against a single claim. That is the pair ISO/IEC 19795-1 defines and then declines to measure under deliberate subversion. Defining the identity task is the work it drives, and the trap is treating similarity as a posterior probability.

Closed-set identification selects one identity, assuming the speaker is enrolled. It is under no obligation to conclude that the speaker is outside the population. Enrollment design is the work here. Detection error tradeoff and calibration are the useful evidence. The failure to look for is speaker overlap across train, calibration and test.

Open-set identification searches that same population and also has to reject strangers. That is a separate failure, carrying rates of its own: open-set miss and false identification. Calibrating the decision is the work. The failure that catches teams out is enrollment captured on one device and deployed on another.

Each is best used when its assumptions are documented.

A single similarity threshold conceals four separate decisions, and each carries its own evidence. Define the identity task: choose verification, closed-set identification, open-set search, or clustering. Design enrollment: specify utterance count, language, device, recency, consent, and update policy. Calibrate the decision: set thresholds for population, attack model, cost, and quality conditions. Add layered controls: combine anti-spoofing, possession, confirmation, rate limits, and recourse.

Enrollment is the second of them. It is easy to miss, because it looks like a setup step rather than a choice. A reference recording is captured on one device, in one language, at one age, on one day. It goes on being the reference after all four have changed. The third condition in the opening list — enrolled on one device, deployed on another — is exactly that gap widening with time. Enrollment is a policy, and it has a date on it. Nick Evershed's Centrelink voiceprint was enrolled before a cloned copy of his voice was something a journalist could assemble. Nothing in the enrollment record knew that the world around it had moved.

Calibration is the third decision. The ASVspoof 5 organisers named it as newly exposed when their baselines came back with EERs no lower than 29%. Every later control is built on top of these earlier choices. None of them is in a position to question them.

Four words carry that chain, and in conversation they collapse into one idea about matching voices. Each implies different evidence, a different unit, and a different set of decision rights. Anyone documenting a voice matching system has to keep them apart. The fourth does the quietest damage. It names an operating point that is almost never the point you ship. And the same three letters, EER, carried the ASVspoof 5 baselines' collapse to no lower than 29%, on 138,688 bona fide and 542,086 spoofed utterances.

Which is why the pair worth reporting is false match and false non-match at the thresholds actually in operation, not at the equal error point. Even that pair says nothing about whether similarity is being read as a posterior probability. The cloned audio cleared the operating threshold. In these two numbers it would appear as a success. ISO/IEC 19795-1 is explicit that it is not measuring that case: “Not within the scope of this document is the measurement of error and throughput rates for people deliberately trying to subvert the intended operation of the biometric system (e.g. by presentation attacks).”

The attack lives in a different standard, with different metrics. ISO/IEC 30107-3 covers presentation attack detection testing and reporting. It defines APCER, BPCER and the impostor attack presentation accept rate, IAPAR. It also warns against reading those numbers across sources: “Error rates for PAD mechanisms are not necessarily comparable across similar tests, and error rates for PAD mechanisms are not necessarily reproducible by different test laboratories.” An APCER from one lab is not a rival to an APCER from another.

So say what unit each figure is in, and whose voices were enrolled. Say how uncertain it is. Say on what devices and in what languages it was measured, with detection error tradeoff and calibration beside it.

A release gate for voice verification is written to be attacked. By now the attack it has to survive is a familiar one: a similarity score read as if it were the probability that the speaker is who they claim to be. The four steps are these. Define the threat model, including replay, synthesis, voice conversion, coercion and account takeover. Build independent partitions, separating speakers, sessions, channels and attack generation assets. Choose the threshold by cost, reporting error rates and confidence intervals at the intended population. Plan recovery, supporting alternative authentication, enrollment revocation and dispute review.

The first step no longer has to argue that synthesis is a real category. In February 2024 the US Federal Communications Commission ruled on it: “In this Declaratory Ruling, we confirm that the TCPA’s restrictions on the use of “artificial or prerecorded voice” encompass current AI technologies that generate human voices.” A cloned voice is an artificial voice, because a person is not speaking it. Such calls are unlawful without the called party's prior express consent, effective on release. The synthetic caller in your threat model has a docket number.

Three things go in writing, and they are the three the next team will need. Write down what your definition of the identity task assumes. Write down one case that breaks it — Incident 523 is available free of charge. Write down which added control takes over when it does.

That third line is where the fourth condition from the opening list is finally paid off. NIST SP 800-63B-4 has already written the shape of the answer. A biometric only as a second factor, alongside a physical authenticator. Always a non-biometric alternative on offer. For voice, its answer is shorter still: “Biometric comparison based on voice SHALL NOT be used.”