SPARC: Spoken Profile Authorship Recognition Corpus

AI summary

Overview

Research area: Speech and language privacy — specifically, whether transcribed speech (text alone, with acoustics removed) leaks speaker identity and demographic attributes.

Technical level: Intermediate. The paper uses standard speaker-verification metrics (EER, AUC, TPR at low false-positive rates), but the setup and conclusions are explained in accessible terms.

Scope (one sentence): The paper introduces SPARC, a benchmark of podcast transcripts from over 12,500 speakers, and uses it to test whether embedding-based models and LLMs can link transcripts to the same speaker or to shared sex, age, and accent labels.

What This Paper Is About

Speech anonymization has concentrated on the acoustic signal, on the assumption that stripping pitch and timbre from audio is enough to protect a speaker. But a transcript still preserves word choice, syntax, discourse markers, and conversational habits, and those traces may be enough to link an anonymous transcript back to a known person or to infer sensitive traits about them. The paper asks how much speaker information survives in text alone, and whether that leakage can be reduced by releasing less text.

Key Contributions

1. SPARC, a large-scale transcript-privacy benchmark. Built from two podcast corpora, it contains over 12,500 speakers and 1.5M quality-filtered utterances, with speaker-level sex, age, and accent labels that are either from source metadata or model-inferred. It provides standardized verification trials across probe-length and enrollment-size budgets: 124,356 task-specific trials for NaturalVoices and 65,174 for SPoRC.

2. A systematic evaluation of eight adversaries. Five representation-based models (SentenceBERT, PCM, CISR, LUAR, and a TF-IDF baseline) and three LLMs (GPT-5-mini, Llama-3.3-70B-Instruct, Qwen3.5-9B) are compared on identical speaker- and attribute-verification trials, with episode- and metadata-based topic proxies to control for shared subject matter.

3. Evidence that LLMs retain speaker-verification signal under topic controls. LLM adversaries stay well below chance even in the harder within-episode condition, whereas the embedding-based baselines collapse toward chance there.

4. A low-false-positive analysis showing average-case metrics misrank models. Two LLMs with nearly identical EER and AUC differ by more than 2x in the fraction of positive trials accepted at a 1% false-positive rate.

Main Findings

• Speaker identity is verifiable from transcripts across episodes. In the cross-episode (different-topic) condition, GPT-5-mini and Llama-3.3-70B reach 0.285 and 0.281 EER on NaturalVoices and 0.253 and 0.262 on SPoRC, well below SentenceBERT at 0.351 and 0.350. AUC follows the same ordering, 0.77–0.80 for the LLMs versus 0.70 for SentenceBERT.

• Embedding models collapse when topic is controlled, LLMs do not. Within an episode, SentenceBERT degrades from 0.350 to 0.460 EER, LUAR from 0.378 to 0.452, PCM from 0.386 to 0.462, and CISR to 0.492 (effectively chance). Llama-3.3-70B retains 0.322 EER and GPT-5-mini 0.345, both at 0.721 AUC, against combined-setting values of 0.284 and 0.277.

• TF-IDF is the strongest non-LLM model. Its EER rises only mildly within an episode (0.362 to 0.409), and on SPoRC it reaches 0.318 EER and 0.740 AUC, suggesting surface word-choice statistics carry speaker-linked signal. It still trails both strong LLMs in every condition.

• Sex-label verification splits by model family. SentenceBERT sits near chance across conditions (0.468–0.508 EER), while GPT-5-mini and Llama-3.3-70B reach 0.377 and 0.420 EER cross-episode and 0.437 and 0.422 within-episode.

• Age-group agreement is not reliably verifiable. Performance stays near chance everywhere; the best EERs are 0.447 cross-episode and 0.440 within-episode, both from PCM. Llama-3.3-70B even falls below chance in the combined condition (0.546 EER, 0.433 AUC), pointing to systematic bias rather than sensitivity.

• Accent shows the clearest two-tier gap. Embedding models sit at chance cross-episode (0.494 EER for SentenceBERT; 0.494–0.534 AUC across embedding baselines), while GPT-5-mini reaches 0.268 EER (0.771 AUC) and Llama-3.3-70B 0.320; within-episode they reach 0.350 and 0.332. The task is a coarse North American versus Other binary, so it does not establish fine-grained dialect verification.

• Probe length is the binding constraint. Cutting the probe from 200 to 50 words raises GPT-5-mini's EER by 0.065 (0.285 to 0.350), versus 0.047 for Llama-3.3-70B and 0.038 for SentenceBERT. Enrollment size matters less: cutting from L=20 to L=5 costs GPT-5-mini 0.041, SentenceBERT 0.024, and Llama-3.3-70B 0.044.

• Reducing both compounds only mildly. The joint minimal setting (50-word probes, L=5) raises EER by 0.076 for GPT-5-mini, 0.064 for Llama-3.3-70B, and 0.046 for SentenceBERT. Even so, the LLMs retain 0.35–0.36 cross-episode EER, still far below the 0.5 chance rate.

• Low-FPR results overturn the average-case ranking. At 1% FPR, GPT-5-mini accepts 0.277 of positive trials versus 0.133 for Llama-3.3-70B and 0.143 for SentenceBERT; at 0.1% FPR the gap is 0.121 versus 0.011, with non-overlapping bootstrap confidence intervals for GPT-5-mini ([0.265, 0.293] at 1%). SentenceBERT, despite a much worse EER than Llama-3.3-70B, matches it at 1% FPR.

Methodology in Plain English

The authors started from two public English podcast datasets. NaturalVoices supplied 2,344 speakers across 3,500 episodes and 779,494 utterances, with age and sex labels from its metadata. SPoRC contributed 10,165 speakers, 15,555 episodes, and 795,591 utterances; because SPoRC lacks utterance-level speaker metadata, only its single-speaker episode subset was used, restricted to the middle 50% of episodes by transcript word count.

Both corpora went through one cleaning pipeline. Utterances needed a signal-to-noise ratio of at least 15 dB (estimated with WADA-SNR for NaturalVoices and Brouhaha for SPoRC). Segments with background music were removed using an Audio Spectrogram Transformer classifier, keeping only utterances whose summed probability over 97 AudioSet music categories was at most 0.1. Finally, two ASR systems, Whisper-large-v3 and nvidia/canary-qwen-2.5b, had to agree: a segment was kept only if the word error rate in each direction was at most 0.1. Canary's transcripts became the final text.

Age and sex for SPoRC were inferred with Vox-Profile, and accent for both corpora with Voxlect over 16 English dialect classes, using a Whisper-large-v3 model for accent and a WavLM-large model for age and sex. Utterance-level predictions were averaged per speaker. Accent was binarized into North American versus Other because North American accounts for roughly 77–78% of speakers in both datasets and no other dialect exceeds 6%. Three independent annotators checked 50 recordings per attribute, finding model–majority agreement of 98% for sex, 88% for accent, and 64% for age.

Evaluation imitates speaker verification. An adversary sees an enrollment set of utterances from a known speaker plus a probe whose identity or attribute membership is unknown, and must decide whether the two come from the same speaker or share an attribute label. Trials vary by condition: same speaker in the same podcast (easy positive), same speaker across podcasts (hard positive), different speakers across podcasts (easy negative), and different speakers in the same podcast (hard negative). Probes accumulate whole utterances until within plus or minus 5 words of the target length. For SPoRC, negatives were matched on title/description similarity and podcast category, and speaker negatives also on sex, age group, and accent. The adversarial models are scored by cosine similarity (for the five representation-based methods) or by a 0–100 likelihood score from a single zero-shot prompt (for the LLMs). Results are reported as EER, AUC, and true positive rate at 1% and 0.1% false-positive rates.

Why This Matters

Impact on research. Privacy evaluations for speech have treated anonymization as an audio problem. This paper shows that removing the acoustic signal does not remove the speaker: the strongest LLM adversaries reach 0.25–0.29 EER on cross-episode trials across two independent corpora. It also shows that the metrics the field typically reports can misstate risk. Two models with near-identical EER and AUC (0.285 vs. 0.281; 0.790 vs. 0.789) differ by more than 2x in positive trials accepted at a 1% false-positive rate. Benchmarking transcript privacy with average-case numbers alone can therefore hide the operating points where re-identification actually matters.

Real-world applications.

• Auditing transcript release policies for recorded meetings, podcasts, call centers, and voice-assistant logs, where publishing text is often assumed to be safer than publishing audio.
• Informing GDPR-style risk assessments, given that transcripts are typically treated as less sensitive than voice recordings.
• Evaluating text-anonymization and authorship-obfuscation defenses against modern LLM adversaries rather than against older embedding models.
• Guiding redaction budgets: the ablation results give a concrete sense of how much protection a 50-word versus 200-word release actually provides.

Industry relevance. Any organization that stores, shares, or sells transcripts produced by ASR, including cloud meeting platforms, podcast hosts, and transcription services, faces the linkage risk this benchmark measures. The finding that shrinking the released text hurts the strongest adversaries most is directly actionable for product design, and the finding that it does not eliminate leakage sets a realistic ceiling on what redaction alone can promise.

Future Directions

• Build a more balanced corpus for dialect evaluation. Roughly 77–78% of speakers are labeled North American and no other dialect exceeds 6%, and negative pairs make up only 19% of combined NaturalVoices accent trials. Individual-dialect verification remains untested.

• Attribute decisions to specific linguistic cues. The paper does not separate lexical, syntactic, discourse, stylistic, or ASR-derived cues from residual context, which is needed to explain why LLMs extract a sex and accent signal but not an age signal.

• Test calibrated or likelihood-based scoring. The LLMs emit coarsely quantized 0–100 scores that cluster at round values, limiting ROC resolution at low FPR; one response per model–trial pair was collected, so generation variability is unquantified.

• Evaluate defenses and broader data conditions. The authors position SPARC as a tool for developing transcript-level privacy defenses, and note that the corpus contains only English, filtered, single-speaker podcast speech, leaving noisy, overlapping, multilingual, and code-switched settings open.

Target Audience

Privacy and security researchers working on speech and text; NLP researchers studying authorship attribution and author profiling; practitioners who design or audit transcription and voice-assistant data pipelines; and policy-facing readers interested in how much identifying information survives when audio is replaced by text. Readers should be comfortable with verification-style metrics such as equal error rate and false-positive rates to get the most from the results tables.

Authors’ abstract

Speech anonymization has focused on the acoustic signal, leaving open how much speaker information the transcribed words themselves retain. In this work, we study pairwise speaker and demographic-attribute verification from transcribed speech. We introduce SPARC (Spoken Profile Authorship Recognition Corpus), a benchmark built from two podcast corpora comprising over 12,500 speakers, 1.5M quality-filtered utterances, and speaker-level inferred or source-provided labels for sex, age, and accent, and evaluate five attribution models and three LLMs on identity- and attribute-verification trials with episode- and metadata-based topic proxies. LLM adversaries distinguish same-speaker pairs from text alone. Attribute-label agreement is harder to verify, but LLMs perform sex- and accent-label verification above chance. Finally, releasing only short probe texts (50 words) degrades the strongest adversaries the most, offering a partial reduction in observed linkage.