Accent Analogy Guidance: More Speaker Similarity at Equal Accent in Cross-Lingual Voice Cloning

AI summary

Overview

Research area: Cross-lingual zero-shot text-to-speech (TTS), specifically inference-time accent control for voice cloning and automatic dubbing (cs.SD).

Technical level: Advanced. The paper assumes familiarity with classifier-free guidance (CFG), diffusion/flow-matching samplers, and embeddings/logits/velocities, though the central idea — subtracting a direction estimated from the model's own two outputs — is conceptually simple.

Scope: The paper introduces Accent Analogy Guidance (AAG), a training-free sampler term that removes a cloned reference's source-language accent while preserving speaker identity, and evaluates it against the identity–accent trade-off curve that CFG reweighting traces, across five TTS models and multiple test sets.

What This Paper Is About

When a zero-shot TTS model clones a speaker from a short reference recording and is asked to speak a different language, it copies not only the voice but also the reference's pronunciation, so the dub sounds foreign-accented. Existing inference-time fixes rescale the same reference evidence, which moves accent and speaker identity together along a single trade-off curve: to sound more native, you must give up speaker similarity. This paper proposes a way to move off that curve — getting more native accent at the same speaker similarity — without any retraining.

Key Contributions

1. Accent Analogy Guidance (AAG): A training-free sampler term that estimates an "accent direction" by contrasting the model's own predictions for a synthetic proxy voice rendered once in the source language and once in the target language, so the speaker cancels in the difference and only the accent remains.

2. An evaluation protocol based on the identity–accent curve: A method is scored by ΔSIM — its speaker similarity above the model's own reweighting curve at equal accent — with 95% confidence intervals from a speaker bootstrap and paired per-line resampling, rather than as a single operating point.

3. A broad comparison against seven reweighting/additive-direction variants: Four reweightings of the same CFG residuals (a weight switch across steps, reference-free early steps, a per-token floor on expected text evidence, and a Fisher-metric projection of the reference residual) plus three added directions (guidance toward a target-language draft, language-tag contrast, and accent-label instruction contrast).

4. A predictive premise-and-reach diagnostic: A per-model premise test plus the reach of that model's own reweighting curve indicates in advance whether and roughly how much AAG can gain — including correctly anticipating the one model where it gains nothing (X-Voice).

Main Findings

• Reweighting stays on one curve; AAG leaves it. By the blind judge, none of the seven inference-time variants lies 0.05 or more above the model's reweighting hull on the 49-line dubbing pilot or the Zeroth-Korean and THCHS-30 sets. Five of 17 variant–set pairs are significantly above it, all by under 0.05 and at low accent.

• OmniVoice gains on three test sets. ΔSIM is +0.11 to +0.27 at accents of 3.51 to 4.28 on a 1–5 scale at speaker similarity 0.29, where reweighting keeps 0.02. On held-out Korean lines, accent rises from 3.51 (tuned) to 4.28 — within 0.05 of a reference-free voice at 4.33 — while SIM stays at 0.294 where the hull keeps only 0.02.

• AAG beats the best variant on every set. By +0.09 to +0.28 with 95% CIs above 0.

• Identity survives. In closed-set identification among all 41 speakers (chance 2.4%), AAG keeps most of the tuned setting's identifiability (Zeroth 95 vs. 97%, THCHS 99 vs. 90%, held-out 69 vs. 81%), whereas reweighting pushed to the same accent drops to 81%, 71%, and 4% on held-out.

• Cost is small. WER stays within one point of the tuned setting (3.8 vs. 4.1, 5.1 vs. 4.1, 8.0 vs. 9.2%) and UTMOS within 0.20. Each proxy adds two network evaluations per step (3.3 vs. 2.0 s per OmniVoice line on an RTX 3090). No model is trained.

• β is a single knob across the whole trade-off. On Zeroth, AAG leaves the hull from β = 5, and at β = 8 it is as native as the no-reference voice (4.51 vs. 4.45) at SIM 0.27 instead of 0.03.

• It transfers without retraining. On MaskGCT's alternative setting (1.5, 5, 2), AAG is more native than every reweighting setting (3.10 vs. at most 3.04) while keeping SIM 0.48 instead of 0.29, +0.20 [+0.03, +0.22] above the hull. On F5-TTS, where reweighting barely moves accent (at most 1.76), AAG is more native than every reweighting setting, by +0.36 [+0.14, +0.57] for (2, 5, 2) and +0.42 [+0.18, +0.68] for (3, 3, 5). On CosyVoice 2 the primary setting (1, 1, 1.67) is above the hull by +0.07 [−0.03, +0.08] and more native than any setting on its curve, including its own cross-lingual mode (+0.37 [+0.11, +0.64]).

• X-Voice is the predicted exception. Where the premise fails (with the voice fixed, its accent barely depends on the reference's language), AAG gains nothing: −0.05 [−0.20, +0.09] on the pilot with one seed.

• Premise alone doesn't predict the size of the gain, but premise plus reach does. Premise alone yields a Spearman correlation of +0.07 with the eight outcomes; adding the reach of the model's reweighting curve accounts for them.

• Controls tie the gain to the accent direction. Wrong-language proxies (Spanish for Korean lines) remove the benefit on the pilot (accent 3.49 vs. 4.33; ΔSIM +0.03, n.s.). The part of AAG's direction parallel to ℓ_c − ℓ_t keeps at most 37% of the gain, so it is not a per-token cut of a_s. Proxy identity cancels on average: leakage is +0.02 to +0.06 with two proxies (+0.17 with one).

• Two independent measures agree with the LLM judge. Whisper large-v3's language-ID logit follows the judge across the 45 pilot settings (Spearman 0.97) and also puts AAG above the hull on the pilot (+0.20 [+0.08, +0.33]), Zeroth and held-out — though not on THCHS, Spanish targets, or wrong-language proxies. The judge itself tracks human experts on 154 sentences (Spearman 0.74 [0.65, 0.81]) with repeat-rating agreement κ 0.92.

• A twelve-listener panel confirms both halves. Twelve raters (eleven Korean-L1, one Spanish-L1; six near-native or fluent in English) judged AAG more native than the tuned setting in 85% of decided answers [64, 99] and closer to the reference speaker than equal-accent reweighting in 85% [69, 97]. The trade-off is audible: against the tuned setting, the tuned voice was judged closer in 77% [53, 94]. On naturalness they preferred AAG to equal-accent reweighting 73% [52, 92], the opposite of UTMOS's ranking.

• Limitations. An LLM judge is the primary measure

Authors’ abstract

In cross-lingual zero-shot text-to-speech, the accent of the reference leaks into the target speech. We propose accent analogy guidance (AAG), a training-free sampler term that subtracts an accent direction estimated from the model's own predictions for one synthetic voice rendered in both languages, so the voice cancels and only the accent remains. By a blind LLM accent judge on real dubbing data, reweighting classifier-free guidance between reference and text, and its variants, stay near one identity-accent trade-off curve; we score a method by its speaker similarity above that curve at equal accent ($Δ$SIM). Across four open TTS models AAG lies above the curve: on OmniVoice $Δ$SIM is +0.11 to +0.27 on three test sets (accent 3.51 to 4.28 on a 1-5 scale at speaker similarity 0.29, where reweighting keeps 0.02); MaskGCT and CosyVoice 2 also lie above their curves, and on F5-TTS it is more native than any reweighting setting. An LLM-free language-ID measure and a twelve-listener panel agree. A premise test and the reach of a model's own curve indicate in advance whether and roughly how much AAG can gain, predicting the one model where it gains nothing (X-Voice).