Building and Evaluating Fixed-Voice Thai TTS from Synthetic Speech

AI summary

Overview

• Research area: Thai text-to-speech (TTS), knowledge distillation from a large zero-shot voice-cloning model into a compact fixed-voice model, and TTS evaluation methodology.
• Technical level: Advanced. The paper assumes familiarity with TTS architectures (VITS, StyleTTS2, Kokoro), phoneme frontends, grapheme-to-phoneme conversion, CTC-based ASR, and rejection sampling.
• Scope: A single technical report describing how a 15-second voice reference is turned into a quality-controlled synthetic Thai corpus and distilled into an 82M-parameter on-device Thai–English TTS model, with a complementary evaluation framework and teacher-support analysis.

What This Paper Is About

Deploying Thai TTS in low-resource settings currently forces a choice between a large voice-cloning model that needs reference audio and GPU inference, or a compact fixed-voice system that needs a licensed speaker-specific corpus. The authors study a third route: using a large zero-shot voice-cloning teacher (OmniVoice) as a programmable data source that converts a short reference (e.g., 15 seconds) into a compact fixed-voice student (an 82M-parameter Kokoro/StyleTTS2 model) trained entirely on synthetic speech. The paper asks which pipeline components—text preparation, synthetic generation, quality filtering, rejection sampling, and frontend choices—actually affect the student, how to measure their effects, and where the teacher's limitations remain.

Key Contributions

1. An end-to-end recipe for converting a short voice reference into a quality-controlled Thai synthetic corpus and a compact fixed-voice student.
2. An evaluation framework that separates sentence-level CER, targeted pronunciation correctness (Challenge-Set Keyword Accuracy), pause placement (pause precision, PPER, intra-word pause rate), speaker similarity, and speaking rate.
3. Controlled evidence for the effects of pause filtering, quality filtering, rejection sampling, pretrained initialization, and frontend policy—including keyword-accuracy gains from frontend changes without retraining the acoustic model.
4. A teacher-support analysis through oracle best-of-K sampling, plus an Isan dialect adaptation study showing that a 15-second reference can transfer voice identity and dialect forms to the fixed-voice student.

Main Findings

• Final model: Wayu-Paxa-TTS-Edge is an 82M-parameter fixed-voice Thai–English TTS system supporting on-device inference. It reaches 68.2% Challenge-Set Keyword Accuracy (85.5% of Gemini 3.1) and 91.4% pause precision, which exceeds its OmniVoice teacher's 89.9% and reaches 94.8% of Gemini 3.1. It reports 3.7% CER on Thai and 1.1% CER on English.

• Lowest pause-placement error: Among the three compared systems (the student, the OmniVoice teacher, and Gemini 3.1 Flash TTS), the student has the lowest PPER (6.7%) and the lowest intra-word pause rate (1.4%).

• CER alone is insufficient: Gemini 3.1 Flash TTS beats OmniVoice on both mean CER (3.3% vs. 4.6%) and Challenge-Set Keyword Accuracy (79.8% vs. 72.8%). On 210 long sentences, Gemini 3.1 Flash TTS also achieves higher pause precision (96.4% vs. 89.9%), lower PPER (13.8% vs. 17.6%), and a lower intra-word pause rate (1.9% vs. 5.2%)—even while producing more than twice as many pauses per clip (4.44 vs. 1.79). Because PPER records whether a clip contains at least one misplaced pause, pausing less often creates fewer opportunities to incur such an error, yet OmniVoice still has the higher PPER.

• Filtering rejects 23.0% of teacher candidates: Pause placement (11.3%) and pronunciation (10.4%) failures dominate; speaking rate (3.0%) and duration (0.1%) checks remove few candidates.

• Pause filtering improves more than phrasing: Going from the unfiltered monolingual baseline (17.72 hours) to pause filtering (17.63 hours) raises Keyword Accuracy from 67.5% to 69.6%, reduces mean CER from 4.0% to 3.5%, improves pause precision from 81.3% to 88.4%, and cuts speaking-rate deviation from 12.4% to 8.2%.

• Bilingual frontend at fixed corpus size: Introducing it (still 17.63 hours) leaves Keyword Accuracy near the pause-filtered configuration (68.7% vs. 69.6%), raises mean CER from 3.5% to 4.1%, and leaves pause behavior essentially unchanged (88.4% to 88.2% precision; 14.3% to 14.8% PPER; intra-word flat at 5.7%).

• Scale-up trades coverage for quality: Expanding to 40.58 hours with content filtering slightly improves mean CER and reduces PPER and intra-word errors, but Keyword Accuracy falls from 68.7% to 67.2%, pause precision from 88.2% to 85.4%, and speaking-rate deviation rises from 8.0% to 12.2%.

• Rejection sampling reverses the trade-off: Resampling rejected candidates (41.81 hours) improves Keyword Accuracy by 2.0 points, reduces mean CER by 0.5 points relative to scale-up without resampling, improves every pause measure, and reduces speaking-rate deviation while leaving speaker similarity unchanged.

• Pretrained initialization matters most for code-switching: Pretrained initialization improves every reported metric over from-scratch training (CER 3.4% vs. 6.6%; Keyword Accuracy 69.2% vs. 51.7%; pause precision 92.8% vs. 82.0%; PPER 6.2% vs. 12.9%; intra-word 1.4% vs. 3.8%). The largest keyword gain is code-switching (+43.4 points, 22.8% to 65.5%), while the name gain is substantially smaller (+6.1 points, 47.1% to 53.2%).

• Frontend changes help without retraining: On fixed epoch-8 weights of the Thai-only model, LLM-based verbalization (DeepSeek-V4-Flash) changes 12.3% of inputs and raises Keyword Accuracy from 68.9% to 69.2%; expanded TLTK phoneme handling changes 7.3% and raises it from 69.2% to 70.0%—a total of 68.9% to 70.0% with no acoustic-model retraining.

• Final comparison: Against the Thai-only model, adding English drops English CER from 4.4% to 1.1% while speaker similarity and intra-word pause rate remain unchanged (0.882 and 1.4%), with modest reductions in Keyword Accuracy (69.2% to 68.2%) and pause precision (92.8% to 91.4%). Speaker similarity is 0.882 for the bilingual student and 0.899 for the OmniVoice teacher, 0.816 for Gemini 3.1 Flash TTS.

• Teacher headroom: Oracle best-of-K sampling on the Challenge Set raises exact accuracy from 72.8% at K=1 to 81.4% (K=6), 84.8% (K=22), 86.7% (K=54), 87.3% (K=86), and 87.9% (K=118)—15.1 percentage points above the single-sample baseline.

• Coverage limits recovery: OmniVoice reports approximately 10.5k hours of Thai speech, 98.2% from GigaSpeech 2. Across 1,060 Thai-only keywords, frequency correlates with sampling difficulty (ρ = −0.437). At K=118, coverage is 66% for unseen keywords but 98% for keywords observed 100–10k times; unresolved items are 21% for unseen keywords.

• Isan adaptation: Fine-tuning the Thai-only model for four Isan-only epochs on roughly 1.5 hours of synthetic Isan speech from a 15-second reference improves Isan CER over the teacher (5.5% vs. 6.6%) and nearly retains voice identity (0.842 vs. 0.854), but degrades Central-Thai CER (3.4% to 5.1%) and Central-Thai Keyword Accuracy (69.2% to 65.8%).

Methodology in Plain English

The pipeline has three stages. First, Thai text is sourced from WangchanThaiInstruct and an LLM-based keyword-synthesis pipeline targeting difficult expressions, then split into sentence-level chunks. Digits and embedded English spans are rewritten into pronunciation-oriented Thai or "Tinglish" because preliminary experiments found the teacher sometimes spoke digits in Chinese and rendered English with the wrong accent.

Second, the teacher creates speech. OmniVoice's Voice Design mode generates seed references from 12 speaker specifications, and OmniVoice cloning mode renders corpus text with those frozen references. Each candidate then passes through a quality filter. Content correctness is checked by transcribing with the CTC-based Thai ASR model airesearch/wav2vec2-large-xlsr-53-th and comparing "hard tokens" (out-of-vocabulary items or TLTK dictionary headwords rare in the Thai National Corpus) in phoneme space, including lexical tone; a CTC verifier is used because Whisper-style autoregressive ASR can recover the intended word from context and hide pronunciation errors. Rejected candidates are also checked for pause placement, speaking rate outside roughly ±10% of the speaker-specific range, and abnormally compressed hard-token durations. Pause-placement failures are removed; other failures are re-rendered up to four times, retaining the best take if none passes.

Third, the accepted pairs train the student. The 82M-parameter Kokoro/StyleTTS2 backbone is adapted with a script-routed Thai–English phoneme frontend using TLTK for Thai grapheme-to-phoneme conversion (Misaki handles Latin spans in the bilingual variant). Four of five Thai lexical tones reuse existing contour tokens; only the Thai low tone needed a new vocabulary entry and a learned embedding. Models train for eight epochs with AdamW, a learning rate of 1×10⁻⁴ for the main model and 1×10⁻⁵ for PL-BERT, and an effective batch size of eight. Most modules are initialized from the released Kokoro checkpoint, with style encoder and predictor encoder from StyleTTS2-LibriTTS.

Evaluation uses five axes. CER is computed with Typhoon Whisper Large V3 on a 500-utterance Thai-only set disjoint from the Challenge Set, with per-utterance CER capped at 100% to limit ASR hallucination effects. Challenge-Set Keyword Accuracy uses 1,531 sentences (Thai–English code-switching 391, names 310, rare words 410, informal spelling 210, long sentences 210), each containing one target expression checked by exact substring match against accepted forms. Pause placement is scored on the 210 long sentences, using an allowed-position mask that is the union of author-written spaces/punctuation and contextual boundaries from a text-only gemini-3.1-pro-preview mask. Speaker similarity uses ECAPA embeddings from speechbrain/spkrec-ecapa-voxceleb against a target centroid built from up to 40 teacher-generated training utterances per speaker, and speaking rate is measured in tokens per voiced second relative to the teacher.

Why This Matters

The paper provides a replicable engineering recipe for building a deployable, on-device TTS voice for a low-resource language without a licensed speaker corpus, and it argues that CER alone hides the failure modes that matter most for Thai—mispronounced names, code-switched terms, and pauses inside words. It also frames synthetic data quality as a coverage problem: strict filtering removes defects but also removes hard training texts unless rejected items are resampled.

Real-world applications:

• Interactive voice response (IVR) and call-center systems needing a single organization-specific Thai voice under latency and cost constraints.
• Personal brand or creator voice products where a short reference is available but GPU inference at scale is not affordable.
• Accessibility tools such as screen readers and on-device assistive speech that must run without reference audio.
• Dialect and local-language adaptation, demonstrated here by a 15-second Isan reference producing an Isan-accented fixed voice.

Industry relevance: the release of an open-weight 82M-parameter model and an evaluation framework lowers the barrier to building Thai TTS products, while the teacher-support analysis tells practitioners where data coverage, not model size, is the binding constraint. The ethics statement notes that synthetic voice construction can also enable unauthorized voice imitation, so references used for training should have clear provenance and consent or licensing.

Future Directions

• Improving sampling and selection to capture more of the teacher's recoverable distribution, since best-of-K shows 15.1 percentage points of headroom above the single-sample baseline.
• Expanding data coverage for expressions underrepresented in the teacher's training corpus, where 21% of items remained unresolved at K=118 for unseen keywords.
• Building a comprehensive Isan evaluation set to assess dialect naturalness and fidelity, which CER alone does not establish.
• Combining neural and rule-based approaches for Thai tokenization and phoneme conversion, since current metrics rely on heuristics.

Target Audience

Speech and TTS engineers building low-resource or on-device systems, researchers working on knowledge distillation from generative speech models, Thai and Southeast Asian language technology practitioners, and evaluation-focused researchers interested in metrics that go beyond CER. Readers looking for perceptual quality scores should note that MOS is not reported in this paper.

Authors’ abstract

In low-resource settings, deploying TTS typically requires choosing between a large voice-cloning model with costly inference or a compact fixed-voice system that requires a speaker-specific corpus. We study a third route: using a large voice-cloning model as a programmable data source to turn a short voice reference (e.g., 15 seconds) into a compact fixed-voice student trained entirely on synthetic speech. This setting makes pipeline design consequential: teacher errors become training targets, while filtering failed generations can reduce coverage of difficult texts. Thai further introduces challenges from ambiguous word boundaries, lexical tone, names and loanwords, numeric verbalization, and Thai-English code-switching. We study how text preparation, synthetic generation, quality filtering, rejection sampling, and frontend choices affect the resulting student, and where teacher limitations remain. We evaluate CER, Challenge-Set Keyword Accuracy, Prosody Pause Accuracy, speaker similarity, and speaking rate. The resulting 82M-parameter model, Wayu-Paxa-TTS-Edge, enables on-device Thai TTS without reference audio. It achieves 68.2% Challenge-Set Keyword Accuracy (85.5% of Gemini 3.1) and 91.4% pause precision, outperforming its OmniVoice teacher (89.9%) and reaching 94.8% of Gemini 3.1. It also achieves the lowest pause-placement error and intra-word pause rates among the three systems, and 3.7% and 1.1% CER on Thai and English, respectively. We open-source the model and evaluation framework for Thai TTS development.