ZeSTA: Zero-Shot TTS Augmentation with Domain-Conditioned Training for Data-Efficient Personalized Speech Synthesis

AI summary

Overview

• Research area: Speech synthesis (cs.SD), specifically low-resource personalized text-to-speech (TTS), zero-shot TTS (ZS-TTS) data augmentation, and speaker adaptation.
• Technical level: Intermediate. The method itself is simple, but the paper assumes familiarity with TTS building blocks such as text encoders, acoustic generation modules, speaker embeddings, and metrics like SECS, CER, and WER.
• Scope: The paper proposes ZeSTA, a domain-conditioned fine-tuning framework with real-data oversampling that uses zero-shot TTS output as synthetic training data for personalized speech synthesis when only a very small amount of real target-speaker audio exists, without changing the base TTS architecture.

What This Paper Is About

Personalizing a TTS model to a new speaker normally requires a decent amount of that speaker's recordings, which is often unavailable. The authors show that when a zero-shot TTS system is used to synthesize extra training speech for such low-resource speakers, naively mixing that synthetic audio with the few real recordings makes the output more intelligible but noticeably less similar to the target speaker's voice. Their goal is to keep the intelligibility benefit of synthetic data while preventing the loss of speaker identity.

Key Contributions

1. A domain-conditioned training (DC) scheme: each training sample is tagged as real or synthetic via a lightweight domain embedding, so the model optimizes p(y | x, d) and inference is performed with d = real. This requires no change to the base TTS architecture.
2. Real-data oversampling (OS): the scarce real target-speaker utterances are repeated during fine-tuning (a factor of three in the reported setup) to further stabilize adaptation and improve speaker similarity.
3. A systematic empirical study: experiments across two datasets (LibriTTS and the in-house YoBind corpus) and two open-source ZS-TTS source models (Fish-Speech and CosyVoice 2), with objective metrics (SECS, CER, WER) and subjective tests (MOS naturalness and an ABX speaker-similarity preference test with 18 listeners).
4. Ablations on design choices: effect of domain embedding size (16, 64, 256), the impact of additional scaled synthetic data, and a comparison of speaker-matched versus speaker-mismatched synthetic augmentation.

Main Findings

• Naive synthetic mixing trades speaker similarity for intelligibility: With Real 10% + Synth 90% and no DC/OS, SECS drops from 0.818 to 0.765 on LibriTTS with Fish-Speech and from 0.813 to 0.764 on YoBind, while CER/WER improve substantially (e.g., LibriTTS CER 5.932 to 4.738, WER 12.520 to 10.348).
• Domain conditioning restores speaker similarity: Adding DC alone raises SECS to 0.807 on LibriTTS and 0.792 on YoBind with Fish-Speech, close to the real-only Real 10% values of 0.818 and 0.813, at the cost of a slight CER/WER increase (e.g., LibriTTS CER 4.962, WER 10.879).
• Oversampling adds further gains when combined with DC: DC + OS reaches SECS 0.815 on LibriTTS and 0.799 on YoBind with Fish-Speech, and 0.815 / 0.804 with CosyVoice 2, while partially recovering intelligibility. OS alone produces only limited and unstable improvements.
• Consistent across generators: The same pattern appears for both Fish-Speech and CosyVoice 2, indicating the approach is not tied to a specific ZS-TTS source.
• Synthetic data scaling helps intelligibility: Adding extra synthetic speech from 800 randomly sampled VCTK transcripts (filtered to keep only outputs with Whisper medium WER below 5%) slightly lowers SECS (e.g., 0.812 on LibriTTS with Fish-Speech) but improves CER and WER substantially (4.123 / 9.467).
• Subjective tests favor the method: MOS naturalness for DC + OS (3.92 on LibriTTS, 3.58 on YoBind with Fish-Speech) is comparable to Real 100% (3.67 / 3.50) and naive mixing (3.86 / 3.50). ABX preference over the baseline was 70.8% (LibriTTS) and 66.7% (YoBind) for Fish-Speech, and 61.8% / 60.2% for CosyVoice 2, all statistically significant under a two-sided binomial test (p < 0.05).
• Moderate domain embedding size is best: On LibriTTS with Fish-Speech under DC without OS, size 64 gave SECS 0.807, CER 4.962, WER 10.879; size 16 gave comparable SECS (0.807) but worse intelligibility (CER 5.118, WER 11.229), and size 256 gave slightly lower SECS (0.800) with no ASR-metric gain. The default speaker embedding hidden size was reduced from 256 to 64.
• Speaker-consistent synthetic data matters: Under identical DC-without-OS conditions on LibriTTS with Fish-Speech, speaker-matched augmentation reached SECS 0.807, CER 4.962, WER 10.879, whereas speaker-mismatched (same-gender, different speaker) augmentation fell to SECS 0.792 with CER 5.817 and WER 12.354, barely better than the Real 10% baseline of 0.818 / 5.932 / 12.520.
• Latent visualization: A t-SNE plot shows real and speaker-matched synthetic representations partially overlap rather than separate cleanly, whereas speaker-mismatched synthetic data show more pronounced separation, consistent with a larger domain gap.

Methodology in Plain English

The authors start from a low-resource scenario: for each target speaker they keep only 10% of the training utterances as real recordings (Real 10%) and synthesize the remaining 90% (Synth 90%) using an off-the-shelf zero-shot TTS model, prompted with the longest available real utterance for that speaker. Two source models with different architectures are used: Fish-Speech and CosyVoice 2.

The target model is VITS, pretrained on VCTK with the standard multi-speaker split, then fine-tuned per speaker. The key idea is that the text encoder already produces speaker-agnostic linguistic features, so synthetic speech still teaches the model how to pronounce text correctly. To keep the synthetic audio from pulling the voice away from the target speaker, a small domain embedding (size 64) tells the acoustic generation module whether the current sample is real or synthetic. At inference, this label is simply set to real. On top of that, the few real utterances are repeated three times during fine-tuning so the model sees them more often than the much larger synthetic pool.

Evaluation uses SECS (speaker embedding cosine similarity, computed with ECAPA-TDNN embeddings trained on VoxCeleb and VoxCeleb2), and CER/WER measured with Whisper medium, plus 100 randomly selected LibriSpeech test-clean texts for ASR-based metrics. To model a real-world augmentation pipeline, they also generate extra speech from 800 sampled VCTK transcripts and filter out hypotheses where Whisper medium reports WER of 5% or higher. Objective results are averaged over three runs with different random seeds per speaker. Subjective testing used 18 listeners for MOS naturalness and an ABX preference comparison against the naive-mixing baseline.

Why This Matters

Impact on research. The paper isolates a concrete failure mode of synthetic data augmentation for speaker adaptation and offers a minimal architectural fix. Because DC adds only a small embedding and OS is a sampling change, the recipe is directly portable to other multi-speaker TTS systems that already condition on speaker information. It also documents an under-explored trade-off: synthetic speech can improve intelligibility metrics while harming speaker identity, which matters when interpreting WER numbers reported for synthetic-data-trained models. The speaker-matched versus speaker-mismatched ablation adds evidence that speaker consistency, not just low acoustic variability, drives the benefit.

Real-world applications.

• Custom voice creation for users who can only record a short script, enabling a personalized voice assistant or reading app.
• Voice restoration and accessibility tools, where a small archive of a person's speech is all that is available.
• Localization and content production, where a narrator's voice must be extended to new scripts or languages with minimal recording budget.
• On-device or lightweight TTS deployment, since the base VITS architecture is unchanged and remains efficient.

Industry relevance. The paper comes from Maum AI and Humelo, and the in-house YoBind dataset was recorded for voice assistant TTS applications. The results target the practical economics of voice cloning: getting acceptable speaker similarity and intelligibility from roughly a tenth of a speaker's data, using openly available zero-shot models as data generators rather than training a large generative model per customer.

Future Directions

• Extending ZeSTA to other TTS architectures and designing architecture-specific conditioning strategies, which the authors explicitly name as future work.
• Investigating what makes domain-conditioned behavior work: the t-SNE analysis shows partial overlap rather than strict separation, so the mechanism behind the similarity recovery is not fully explained.
• Determining the optimal domain embedding capacity in a more principled way, since performance was non-monotonic across sizes 16, 64, and 256 and only LibriTTS with Fish-Speech was tested.
• Understanding the speaker-matched versus speaker-mismatched gap better, including whether the requirement for speaker-consistent synthetic data can be relaxed when reference audio is extremely short.

Target Audience

Researchers and engineers working on personalized or low-resource TTS, speech data augmentation, and speaker adaptation will get the most from this paper. It is also useful for practitioners building custom-voice products who need a practical recipe for combining synthetic and real recordings, and for readers interested in how synthetic data shifts the intelligibility versus speaker-similarity trade-off in speech models.

Authors’ abstract

We investigate the use of zero-shot text-to-speech (ZS-TTS) as a data augmentation source for low-resource personalized speech synthesis. While synthetic augmentation can provide linguistically rich and phonetically diverse speech, naively mixing large amounts of synthetic speech with limited real recordings often leads to speaker similarity degradation during fine-tuning. To address this issue, we propose ZeSTA, a simple domain-conditioned training framework that distinguishes real and synthetic speech via a lightweight domain embedding, combined with real-data oversampling to stabilize adaptation under extremely limited target data, without modifying the base architecture. Experiments on LibriTTS and an in-house dataset with two ZS-TTS sources demonstrate that our approach improves speaker similarity over naive synthetic augmentation while preserving intelligibility and perceptual quality.