Audio Dataset Design, Rights, Consent, and Provenance

A speech corpus is usually judged the way any sample is judged: how many speakers, how many rooms, how many devices, how many languages, how many hours. But collection source, speaker relationship, license, consent, withdrawal, channel, room, language, device, time and downstream use all sit in the same list. Only some of them are statistics. The rest decide whether the team was allowed to hold the recording at all. No amount of coverage settles them.

The two cannot be separated in audio, because the recording is the person. Public access does not imply unrestricted training rights. It does not imply biometric consent, or permission to generate a person's voice. De-identification, the usual way out, barely works either. Strip the name from the metadata and the voice itself is still there. So is the conversation happening behind it, where the speaker was, how they sounded when unwell, and who they were talking to. Identity, background conversation, location, health cues, social context — all of it is still in the waveform. A bigger model does not help here.

So the question that decides a corpus is not how broad it is. It is whether the purpose and the prohibited uses declared at collection survive into how the dataset is later maintained, corrected and retired. Without that link, a collection can be broad and balanced and still be data the team was never permitted to use. The rest of this lesson shows what that costs, through a federal judge, a federal regulator, a vendor's own apology and a controlled experiment. The gap gets priced.

A third team built a corpus the way neither of those was built, and a federal judge has now described how. Two voice actors sold recordings on Fiverr, for $1,200 and $400. The stated purposes were "academic research purposes only" and "test scripts for radio ads". Lovo employees bought them and used them to train Genny, the company's commercial text-to-speech model. On 10 July 2025, in Lehrman v. Lovo, Judge J. Paul Oetken ruled that Paul Lehrman and Linnea Sage could proceed on New York Civil Rights Law §§ 50–51, consumer-protection and breach-of-contract claims. Most of the federal copyright and Lanham Act claims were dismissed. What survived is the part about what the speakers were told.

What they were told is on the record. A Fiverr message reached Linnea Sage from the user "tomlsg", alleged to be Lovo co-founder Tom Lee. It reads: “These are test scripts for radio ads. They will not be disclosed externally, and will only be consumed internally, so will not require rights of any sort.” The audio was obtained lawfully. It was paid for. It was delivered with a purpose attached. The purpose is exactly what the training use walked past.

Report only how broadly the recordings cover speakers, devices, rooms, and times, and none of that appears in the numbers at all.

Put the corpora side by side and the questions separate cleanly. Legal availability asks whether the recording can be accessed or licensed under stated terms. CC BY 4.0 answers it for LibriSpeech. A paid Fiverr order answered it for the recordings Lovo bought. Informed consent asks something narrower: whether the person recorded agreed to this particular use. Common Voice puts that question at the front of its collection. Linnea Sage was answered on it in advance, wrongly: “These are test scripts for radio ads. They will not be disclosed externally, and will only be consumed internally, so will not require rights of any sort.” Technical provenance asks what neither of the first two touches. Can you still show, later, where a given recording came from and what has been done to it since? LibriSpeech answers that with diarization checks and per-speaker minute caps. A purchased clip folded into a training set answers it with nothing.

What answers one leaves the other open. A permissive license does not produce consent. Payment does not produce consent either: the §§ 50–51, consumer-protection and breach-of-contract claims went forward against a company that had paid. Consent given once does not produce a record of what happened afterwards. Legal availability means the recording can be accessed or licensed under stated terms. It means nothing beyond that.

Provenance is the question a process is supposed to answer, and it sits at a specific place in the sequence. A dataset process opens by defining the purpose and the prohibited uses. It closes by governing the lifecycle — corrections, removals, retirement. Between the two sits the step that maps rights and participants: who is on these recordings, on what terms, having agreed to what.

That is the step that gets dropped. It gets dropped because it is the only one that requires finding something out rather than writing something down. Drop it, and whatever the purpose statement assumed about the people on the tape arrives at lifecycle governance unchecked. Lovo was not missing a document. The purpose was stated in writing, on Fiverr, to the person being recorded. What was missing was the step where someone checks the training use against that statement before the model ships. So the check happened in a courtroom instead, on 10 July 2025.

Documentation going stale is the fourth of those failures, so the test of a dossier is not whether it exists. It is whether anyone else can use it. It is only a dossier when another team can pick it up and check, without you, whether a voice in the test set was already in the training set. That is the first failure, caught by a stranger. A release note written in your own words will never do that.

You do not have to invent the format. Datasheets for Datasets, published in 2018 by Timnit Gebru and six colleagues, asks for one. Communications of the ACM ran it in 2021. The analogy is an electronic component's datasheet, and the proposal is one line of the abstract: “By analogy, we propose that every dataset be accompanied with a datasheet that documents its motivation, composition, collection process, recommended uses, and so on.” The published sections cover motivation, composition, collection process, recommended uses, distribution and maintenance. Collection process and maintenance are the two an audio corpus cannot afford to leave blank.

Three things go on the page beyond that template. Write down what the declared purpose and prohibited uses assume about who is on the recordings and what they agreed to. Add one counterexample: a use that fits the stated purpose and is still wrong, the way training Genny fitted a paid Fiverr order for radio-ad test scripts. Then say what the lifecycle rules require when that counterexample turns up.

A counterexample is cheap to write down and expensive to honour. The evaluation is where that shows. On 31 May 2023 the FTC and DOJ sued Amazon over children's Alexa voice recordings. The complaint alleged that Amazon retained them indefinitely, and that transcripts were not deleted from all its databases even when parents asked. Amazon agreed to a $25 million civil penalty, to delete inactive children's Alexa accounts, and to a prohibition on using deletion-requested voice data "for the creation or improvement of any data product". Samuel Levine, Director of the FTC's Bureau of Consumer Protection, put the principle in one line: “COPPA does not allow companies to keep children’s data forever for any reason, and certainly not to train their algorithms.” Amazon's own statement reads "While we disagree with the FTC's claims and deny violating the law, this settlement puts the matter behind us." That is what deletion verification costs when someone else measures it. The request was made. The deletion was not shown. $25 million and a ban on using the data followed.

The other conflict is arithmetic. Coverage will sometimes fight withdrawal latency, deletion verification and access incidents, for the obvious reason: honouring a withdrawal removes a speaker, and removing a speaker costs coverage. That is where the evaluation changes the decision instead of decorating it. Which is why the second set of numbers belongs beside the first, not in a separate document nobody reads.

And the abstain threshold now has something to be set against. A 2026 study measured what speaker leakage is worth. On the DAIC-WOZ depression corpus, researchers held training-set size constant and varied only speaker overlap. The split runs over 189 subjects and 6,545 speech segments: a control group of 151 subjects with 5,117 segments, a target group of 38 subjects with 1,428 segments. Their introduction reports the result: “Across architectures, performance improves dramatically under speaker-overlapped evaluation but drops sharply under strict speaker independence (e.g., 97.65% to 58.74% accuracy for a fine-tuned Wav2Vec model).” Same corpus, same amount of training data. 97.65% against 58.74% is the whole of the gap between a model that recognises depression and one that recognises voices. Agree the threshold before the measurement arrives, not after.