DEFINE: Exemplar-Guided Accent Control for Zero-Shot TTS

AI summary

Overview

Research area: Zero-shot text-to-speech (TTS) and accent conversion, in the audio and speech processing category (cs.SD).

Technical level: Intermediate. The paper assumes familiarity with flow-matching TTS, conditioning mechanisms, parameter-efficient adaptation (LoRA), and standard speech evaluation metrics (speaker similarity, WER, UTMOS). The core idea is explained clearly, but the training objective and injection equations require some background.

Scope: The paper introduces DEFINE (Disentangled Exemplar Framework for Identity and Novel-accent Expression), an end-to-end system built on F5-TTS that lets a user pick the speaker voice from one audio clip and the accent from a separate, short accent clip, with a single inference-time weight controlling accent strength.

What This Paper Is About

Zero-shot TTS can clone an unseen speaker from a few seconds of reference audio, but the reference recording carries both the speaker's identity and their accent, so the two cannot be requested separately. DEFINE addresses this by conditioning speaker identity on the conventional reference prompt and target accent on a separate audio exemplar, so a model can, for example, generate a reference speaker's voice with a different accent than the one in the prompt. The goal is to make accent a controllable, independently specified factor inside a single TTS model, at inference time and without accent labels.

Key Contributions

1. DEFINE, a zero-shot TTS framework that specifies speaker identity and target accent from separate audio examples, requiring no accent label at inference time and no post-synthesis waveform conversion step.
2. Prototype anchoring, in which learned per-accent prototypes (a table of C = 23 rows) provide direct supervision for an exemplar encoder, mapping short accent clips to a stable conditioning space despite the weak accent supervision that the flow-matching objective alone provides.
3. Continuous inference-time accent control through a single guidance weight w, which varies accent strength after training; w = 0 removes the accent-conditioning contribution and increasing w progressively strengthens the target accent.
4. Empirical evidence that the approach generalizes beyond its training accent set, including on accents seen in no training data, while preserving speaker identity better than a two-model TTS-to-voice-conversion cascade.

Main Findings

• Accent control on seen accents: DEFINE raises accent-probe accuracy (ACC) from 0.065 with guidance disabled to 0.196 at guidance weight w = 15, on 1,134 seen-accent items. This matches the F5-TTS to Seed-VC cascade (0.188) while achieving 0.024 higher speaker similarity (0.624 vs. 0.600) and comparable predicted quality (UTMOS 3.93 vs. 3.92).
• Out-of-domain accents: On 252 out-of-domain items, DEFINE (0.087) is statistically indistinguishable from the cascade (0.083). It improves over the same model with the accent disabled by +0.040 ACC (95% CI [+0.004, +0.079], paired over items) and over stock F5-TTS by +0.064 ([+0.032, +0.099]).
• Held-out accents are the weak point: On 378 held-out-accent items the cascade remains stronger (0.175 vs. 0.127 for DEFINE).
• A smooth control trade-off: Speaker similarity falls monotonically as w grows (from 0.694 at w = 0 to 0.587 at w = 23 on seen accents), so a single post-training scalar selects the desired operating point. Exemplar-based ACC peaks at w = 19 for both seen (0.214) and out-of-domain accents (0.111).
• Prototype anchoring is essential: Without it, no guidance weight improves on switching the accent off (w* = 0, ACC 0.079); an encoder trained from the flow-matching loss alone does not steer the decoder. Removing the contrastive prototype term costs a further 0.053 ACC on the held-out set.
• The encoder is the bottleneck: Prototype lookup, which receives the accent label and reads the learned table, reaches 0.354 ACC on seen accents at w = 15 versus 0.196 from exemplars. Nearest-prototype accuracy falls from 0.873 on training speakers to 0.540 on unseen speakers (chance 1/23 = 0.043), pointing to speaker overfitting in the encoder rather than conditioning capacity or guidance.
• Conditioning design matters: A cross-attention conditioning variant pushed to w = 23 reaches 0.175 ACC but collapses to UTMOS 1.52 and SPK 0.22, meaning unintelligible audio that still fires the probe; its usable weight in the ablation table is 0.
• Pooling more exemplars barely helps accent, slightly helps identity: Mean-pooling three exemplars instead of one changes ACC by +0.000 (95% CI ± 0.058, seen) and +0.003 (CI [-0.026, +0.032], held-out), and adds 0.019 SPK.
• Human listeners prefer DEFINE: In a listening test on seen accents with 11 listeners, 10 trials per part, and 110 votes per part against a chance rate of 0.333, DEFINE took 77 of 110 votes (0.70, 95% CI [0.61, 0.78]), the cascade 23 (0.21), and F5-TTS 10 (0.09). DEFINE was the most-picked system for 9 of 11 listeners (binomial test against chance, p = 0.0014).
• The perceived accent comes from the guidance term: In the second listening part comparing w = 0, w = 11, and w = 23 from the same model, the votes were 11, 52, and 47. Guidance off falls far below chance (p < 10^-7), while the two active settings are not separable (p = 0.69).
• Efficiency: Each solver step uses one fused classifier-free call and one accent-conditioned call, giving three sequence evaluations against two for F5-TTS, whereas the cascade adds a full conversion pass over the waveform.

Methodology in Plain English

The team starts from a pretrained F5-TTS backbone, a non-autoregressive TTS model trained with conditional flow matching on mel spectrograms, and keeps it frozen. Instead of retraining it, they add rank-16 LoRA adapters to the attention query and value projections, feed-forward layers, and adaLN modulation, which gives 6.3M trainable parameters out of 343M.

The accent comes from short audio clips, encoded by a frozen XLS-R model read at layer 15, pooled into a single vector, and passed through a small trained network to produce a 128-dimensional accent embedding. During training, a learned table holds one prototype per training accent (C = 23 rows). The accent encoder is aligned to these prototypes with a cosine term plus a contrastive term (temperature τ = 0.1), and the decoder is conditioned on a Bernoulli mixture (π = 0.5) of the encoder output and the prototype, with a stop-gradient that keeps the flow-matching loss from updating the encoder. At inference the prototype table is discarded, so unseen accents can be represented directly from their exemplars.

The accent vector is injected by shifting the timestep and text embeddings, with RMS scaling to match the size of each embedding; without that scaling the added term is roughly 100 times smaller than the timestep embedding and is ignored during training. Because the timestep embedding controls adaLN, the accent signal reaches every transformer block. During training the accent vector is set to zero with probability 0.15 to train the accent-free inference branch. Two auxiliary losses shape the embedding space: a supervised contrastive loss pulls clips with the same accent label together, and a speaker classifier with a gradient reversal layer discourages speaker information from leaking into the accent embedding.

At inference, accent strength is set by a single weight using a modified guidance formula that separates the accent-free prediction, the unconditional prediction (with classifier-free guidance scale s = 2), and the accent-conditioned prediction.

Training data combines the English subset of Common Voice, which supplies naturally accented speech from many speakers per accent, and speaker prompts from an in-house set of studio-quality recordings synthesized with a commercial TTS system. Because Common Voice couples speaker and accent, Seed-VC is used to transfer Common Voice utterances into the timbre of studio speakers with different accents, producing 94.7k deliberately mismatched prompt-target pairs alongside 137k real recordings.

Why This Matters

Impact on research. The paper demonstrates that speaker identity and accent can be treated as independent factors inside a single zero-shot TTS model, driven only by audio exemplars and without accent labels at inference. The ablation results are also a useful negative finding: prototype anchoring does the heavy lifting, and an encoder trained by the flow-matching objective alone fails entirely, with the remaining bottleneck identified as speaker overfitting in the encoder rather than conditioning capacity or guidance strength.

Real-world applications:

• Media dubbing and localization, where a voice actor's identity is preserved while the accent is adjusted for a target market.
• Personalized voice assistants or accessibility tools that keep a user's familiar voice while adapting regional pronunciation.
• Voice cloning for audiobooks, games, or virtual characters, where a single voice asset can be reused across multiple accent settings.
• Speech data augmentation and accent-robustness testing for downstream ASR or speech-understanding systems.

Industry relevance. The method needs no retraining to change accent strength, requires no accent labels at deployment, and avoids a separate waveform-conversion stage. The authors report that the single-pass model uses three sequence evaluations against two for F5-TTS, while the cascade adds a full conversion pass, which is a practical argument for production systems where latency and pipeline complexity matter.

Future Directions

• Improving the exemplar encoder, which the paper identifies as the bottleneck: prototype lookup reaches 0.354 ACC on seen accents versus 0.196 from exemplars, and nearest-prototype accuracy drops from 0.873 on training speakers to 0.540 on unseen ones, so reducing speaker overfitting in the accent representation is the most direct next step.
• Closing the held-out accent gap, where the two-model cascade still outperforms DEFINE (0.175 vs. 0.127).
• Better conditioning mechanisms at high guidance, since cross-attention reached 0.175 ACC at w = 23 but collapsed to UTMOS 1.52 and SPK 0.22, indicating that the current injection scheme, not the decoder, limits how far accent strength can be pushed.
• Extending beyond the current accent taxonomy, which combines 10 trained accents of the evaluation taxonomy with 13 further accent classes that are never evaluated; whether the prototype-anchored representation scales to wider accent coverage remains open.

Target Audience

Researchers and engineers working on zero-shot TTS, voice cloning, accent conversion, and controllable speech synthesis, particularly those interested in disentangled conditioning and parameter-efficient adaptation of frozen backbones. It is also relevant to practitioners building dubbing, localization, or personalized voice products who need post-training control over accent strength without accent labels at inference. Readers without background in flow matching or LoRA will find the ablation and listening-test results accessible, but the method section requires some speech-modeling familiarity.

Authors’ abstract

Zero-shot text-to-speech (TTS) can reproduce an unseen speaker from a short reference recording, but typically entangles speaker identity and accent within the same reference. We introduce DEFINE, an end-to-end framework that decouples these factors by conditioning speaker identity and target accent on separate audio exemplars. A single inference-time guidance weight continuously controls accent strength without retraining. Built on F5-TTS with parameter-efficient LoRA adaptation, DEFINE maps short accent exemplars into a conditioning space using an exemplar encoder supervised through learned accent prototypes, requiring neither accent labels at inference time nor post-synthesis waveform conversion. On seen accents, increasing accent guidance improves accent-probe accuracy from 6.5% to 19.6%. More importantly, a single DEFINE model generalizes accent control beyond its training accent set: on seen and out-of-domain accents, though not on held-out accents, it matches the accent transfer performance of a two-model TTS-voice-conversion cascade while achieving higher speaker similarity and comparable predicted speech quality. These results demonstrate that speaker identity and accent can be independently controlled from audio exemplars within a single zero-shot TTS model, including for accents unseen during training.