SEAM: Spectral Embedding Alignment and Merging for Low-Resource TTS

AI summary

Overview

Research area: Machine learning for speech synthesis — specifically, cross-lingual and low-resource adaptation of conditional text-to-speech (TTS) models via vocabulary expansion.

Technical level: Intermediate. The paper assumes familiarity with token embeddings, BPE tokenization, discrete Fourier transforms, and ridge regression, but each component is explained from first principles with a full worked algorithm.

Scope: The paper introduces and evaluates SEAM, a zero-audio method for initializing embeddings of new tokens in a pretrained TTS model by spectrally merging the embeddings of known subtokens and aligning them with the pretrained conditioning manifold via closed-form ridge regression, tested across three languages (Russian, Kazakh, French) and two TTS backbones (Chatterbox, VibeVoice).

What This Paper Is About

When a text-to-speech model needs to speak a new language, it must be given embeddings for phonetic or graphemic units it has never seen. Current practice initializes these new tokens randomly (or as the average of their known subtokens) and then fine-tunes with many hours of paired audio, which is often impossible for low-resource languages. SEAM instead constructs these embeddings analytically, without any target-language audio, by treating a token's sequence of subtoken embeddings as a signal, smoothing it with a spectral low-pass filter, and projecting the result into the geometry of the pretrained embedding space.

Key Contributions

1. A spectrally grounded merging operator that preserves the temporal phonetic order of subtokens and suppresses boundary artifacts between them, unlike permutation-invariant mean pooling.
2. A closed-form self-calibration protocol that fits the projection using only existing vocabulary tokens (decomposed with the original BPE tokenizer), requiring no gradient-based optimization and no target-language audio.
3. Demonstrated data-efficiency gains over random and mean initialization, with SEAM approaching or matching baseline performance while using up to four times less training data, described as a 75% reduction in sample complexity.
4. An order-sensitivity analysis (Proposition 1 and an ablation) establishing that the gains come from order-preserving spectral merging rather than from ridge alignment alone.

Main Findings

• Sample efficiency at 25 hours: SEAM improves over mean initialization by 2.22 CER points on Russian, 10.63 points on Kazakh, and 0.87 points on French. Against random initialization, the corresponding improvements are 9.16, 27.34, and 3.77 points.
• Four-times data reduction: SEAM's results at 25h approach or match baselines trained on 100h, corresponding to a 4× reduction in required data for some languages.
• Convergence at 100 hours: At the 100h budget all methods converge and differences fall within ASR evaluation variance, indicating the advantage is sample efficiency rather than asymptotic performance. On Russian at 100h, Random is actually lowest at 7.62 CER, ahead of SEAM's 7.68 and Mean's 9.64; on French at 100h, Mean (9.06) edges out SEAM (9.26).
• Human preference despite less data: In a blind side-by-side study with 5 native speakers per language, 25 utterances each, SEAM at 25h was preferred over mean initialization at 100h for 40.5% of Russian pairs, 48% of French pairs, and 56.7% of Kazakh pairs, with ties at 25.7%, 32%, and 30% respectively.
• Graceful scaling with vocabulary size: Across |V_new| of 200, 400, 600, 800, and 1000 tokens at 25h of fine-tuning, SEAM degrades only marginally, whereas random and average initialization deteriorate sharply beyond 500 tokens.
• Best against order-aware baselines: On Russian at 5h, SEAM reaches 14.88 CER / 34.41 WER, versus Learned Attention (15.30 / 42.86), Attention Pooling (20.14 / 50.03), Position-Aware Pooling (23.37 / 54.70), Mean (17.53 / 36.73), and Random (20.80 / 42.49). Learned Attention reportedly achieves roughly 10% lower cosine similarity to ground-truth embeddings and higher MSE than SEAM's closed-form solution.
• Order matters, and shuffling hurts more than reversing: Reversing the subtoken sequence raises Russian 5h CER from 14.88 to 19.67 (+4.79 points), while shuffling raises it to 22.91 (+8.03 points) and nearly doubles WER from 34.41% to 53.66%.
• Bandlimited embedding trajectories: Empirical spectral analysis shows cumulative energy exceeds 99% by frequency bin k=3, with Gaussian gains of G(0)=1.0, G(1)≈0.88, G(2)≈0.61, G(3)≈0.32, and G(4)≈0.14, so high-frequency components are suppressed by more than 85% by k=4.
• Cross-architecture generalization: On VibeVoice fine-tuned on 5 hours of Russian, SEAM achieves 6.14% CER versus average initialization at 8.34% and random at 9.26%.
• Zero-audio beats audio-guided refinement: An audio-guided variant starts from a lower training loss (7.2 vs 10.2) but spikes to 15.0 before recovering to about 10.0, versus zero-audio SEAM's smooth convergence to roughly 4.0.
• Robustness to data splits: Across three random Kazakh splits, WER ranges 14.97%–16.51% (mean 15.61%, σ=0.66) and CER ranges 4.79%–5.67% (mean 5.27%, σ=0.36).
• A non-monotonic Kazakh anomaly is explained, not hidden: CER rises from 5h to 25h for Random (11.62 → 32.60) and Mean (8.55 → 15.89); SEAM rises only slightly (4.99 → 5.26). The authors attribute this to word-level lexical overlap between the small training subset and the 100-utterance test set, and note the pattern appears across all methods.

Methodology in Plain English

The setup. A pretrained TTS model has an embedding lookup table for its known vocabulary. A new word — say a Kazakh Cyrillic word — is broken into pieces the model already knows. The question is how to combine those pieces into one new embedding.

Step 1 — Build the feature. Take the sequence of subtoken embeddings and treat each embedding dimension as its own short signal over time. Apply a discrete Fourier transform, then a Gaussian low-pass filter, keeping only the first K_eff = min(K, floor(m/2)+1) frequency bins with K=4 and σ=2.0. Flatten the real and imaginary parts into a vector. Keeping the complex coefficients (rather than just magnitudes) is what retains phase information, and phase encodes the order of the subtokens — which is why reversing the input changes the output, unlike averaging.

Step 2 — Learn the projector without audio. For every existing vocabulary token that can be decomposed into smaller known subtokens, build a training pair: the spectral feature of its decomposition, and its actual pretrained embedding as the target. Stack these into matrices and solve a ridge regression in closed form with λ=1e-6. The bias column is excluded from L2 regularization. No gradient descent, no target-language audio.

Step 3 — Embed a new token. Transliterate non-Latin scripts to English IPA (using the transliterate library for Russian, qazaq_transliterator for Kazakh, and standard normalization for French), segment the IPA string with the original BPE tokenizer, apply the spectral feature, apply the learned projector, and normalize the result to match the average norm of the pretrained embeddings.

The intuition for why it should work. The authors assume the TTS generator's conditional expectation is Lipschitz continuous, so small perturbations in the conditioning space produce bounded acoustic changes. Smooth, order-preserving embeddings of phonetically similar tokens therefore stay close to reference embeddings and keep fine-tuning stable from the first step.

Evaluation. CER via pretrained ASR models (nvidia/stt_kk_ru_fastconformer_hybrid_large and nvidia/stt_fr_fastconformer_hybrid_large_pc) on 100 held-out samples from rulibrispeech, KazakhTTS, and AdrienB134/french-unique-speaker-tts, averaged over three seeds, plus the human preference study. ASR error floors measured on original human audio were 4.88 CER / 14.68 WER for Russian, 0.63 / 3.30 for French, and 0.98 / 5.68 for Kazakh.

Why This Matters

Impact on research. Vocabulary expansion has been treated as a trivial preprocessing step, with methods borrowed from language modeling. This paper reframes it as a signal-processing and manifold-alignment problem and shows empirically that the choice of initialization materially changes how much data adaptation requires. The ablation results also make a sharper point: order-awareness alone is not enough — naive order-aware pooling (position-aware, fixed-query attention) performs worse than plain mean pooling, so the low-pass filtering is doing real work.

Real-world applications:

• Voice interfaces for under-documented and low-resource languages, where collecting 100 hours of transcribed speech is infeasible but 25 hours may be achievable.
• Localization and dubbing pipelines that need to add a new language to an existing TTS product quickly.
• Language documentation and revitalization projects where audio is scarce and expensive to produce.
• Multilingual agentic systems, which the paper frames as the motivating deployment context for rapid language adaptation.

Industry relevance. The method is cheap — a closed-form solve over existing embeddings, run on a single NVIDIA RTX 3090 with 24 GB VRAM in these experiments — and drop-in compatible with any architecture that conditions on token embeddings. It reduces the data-collection cost that is often the deciding line item in shipping a new TTS language.

Future Directions

• Robustness of IPA transliteration. The authors flag subtoken and IPA dependency as a limitation: for scripts or languages lacking reliable IPA mapping tables, transliteration errors propagate directly into the conditioning space. Better or learned grapheme-to-IPA front ends would broaden applicability.
• Closing the remaining gap to baselines at high resources. At 100h, SEAM does not lead in every language (Russian and French at 100h are within, but not better than, the baseline range), so whether a hybrid objective could retain the low-resource advantage while improving the high-resource ceiling is open.
• Making audio-guided refinement stable. The audio-guided ablation failed because embedding norms inflated during refinement and distorted the geometric alignment, causing a loss spike from 7.2 to 15.0. A constrained or norm-regularized refinement step is an obvious next experiment.
• Larger vocabularies and inference speed. The paper constrained |V_new| to 200 for competitive comparison while noting that larger vocabularies yield shorter token sequences and faster inference. Pushing to substantially larger expansion sizes and measuring latency and quality jointly is unexplored.

Target Audience

Speech synthesis researchers and engineers working on cross-lingual or low-resource TTS; practitioners deploying multilingual voice products who need to add languages cheaply; and machine learning readers interested in a concrete case where classical signal processing (DFT-based low-pass filtering) and a closed-form linear solve substitute for gradient-based training and large datasets. Readers should be comfortable with embedding geometry and basic Fourier analysis to get the most from the method section, though the experimental results are readable without it.

Authors’ abstract

Expanding the vocabulary of conditional text-to-speech (TTS) models typically requires extensive fine-tuning with paired audio, creating a major bottleneck for low-resource and cross-lingual adaptation. To address this, we introduce SEAM, a zero-audio initialization method that constructs embeddings for unseen tokens from known subtokens. Unlike standard pooling strategies that ignore sequential structure, SEAM applies a spectral filter to preserve temporal phonetic order and uses closed-form ridge regression to align the new embeddings with the pretrained conditioning manifold. Across three languages and two TTS architectures, SEAM approaches or matches baseline performance using up to four times less training data, reducing sample complexity by 75\% compared to random initialization and mean pooling. Our approach provides a lightweight, reproducible, and theoretically grounded solution for vocabulary expansion in low-resource speech synthesis.