CVSS-X: A Multilingual Speech-to-Speech Translation Corpus for 28 Languages

AI summary

Overview

Research area: Natural Language Processing — multilingual speech-to-speech translation (S2ST), speech corpora construction, and text-to-speech synthesis.

Technical level: Intermediate. Readers need basic familiarity with speech translation pipelines (ASR, MT, TTS), but the paper's structure and metrics are explained clearly enough for newcomers to corpus-building work.

Scope: The paper introduces and evaluates CVSS-X, a ~16,000-hour synthetic English-to-28-languages speech-to-speech translation corpus that reverses the direction of the widely used CVSS corpus.

What This Paper Is About

Speech-to-speech translation research has been bottlenecked by a shortage of large, publicly available parallel speech data. The main existing resource, CVSS, only translates from 21 source languages into English, leaving researchers who want to translate out of English or between non-English pairs without suitable training data. This paper builds CVSS-X, a synthetic corpus that translates from English into 28 target languages, so that combined with CVSS it becomes possible to study bidirectional and pivot-based multilingual speech translation at scale.

Key Contributions

1. A direction-reversed multilingual S2ST corpus: CVSS-X contains approximately 240,000 parallel speech pairs per language across 28 target languages spanning 12 language families (grouped into 7 macro-categories), totaling over 16,000 hours of parallel speech.

2. Two complementary corpus variants: CVSS-X-C uses two fixed canonical voices per language (one male, one female), while CVSS-X-T applies cross-lingual zero-shot voice cloning to preserve the original English speaker's vocal characteristics in the target language.

3. Rigorous quality evaluation across typologically diverse languages: The authors re-evaluate the original CVSS using an identical pipeline and sample size, enabling a controlled comparison, and report ASR-BLEU, WER/CER, UTMOS naturalness, and speaker similarity broken down by language family.

4. A publicly released resource with balanced splits: 222,349 training, 10,000 dev, and 7,843 test samples per language, with code and data released under open licenses (dataset CC-BY-NC 4.0, inherited from NLLB-200).

Main Findings

• Scale advantage: CVSS-X is roughly eight times larger than CVSS in total hours (16,070 vs. 1,900 hours), with 6,725,376 total translated pairs and full cross-language symmetry in source utterances because all target speech derives from the same 240,192 verified English recordings.

• Translation and synthesis quality is comparable to CVSS despite eight times the coverage: CVSS-X-C averages ASR-BLEU 82.4, WER 12.1 percent, and UTMOS 3.55, versus CVSS-C's 94.2, 3.5 percent, and 4.43 on the re-evaluated baseline. The authors attribute the gap to the difficulty of synthesizing 28 languages with a single multilingual model rather than to a flaw in the corpus design.

• Quality is remarkably even across language families: UTMOS for CVSS-X-C stays within a narrow band (3.48 to 3.62) across Romance, Germanic, Slavic, CJK, Uralic, Indo-Iranian, and Other families, suggesting the multilingual TTS model does not trade off some languages for others.

• High-fidelity results for well-resourced families: Romance and Slavic languages reach ASR-BLEU of 88 to 90 with word error rates under 8 percent, indicating near-perfect round-trip intelligibility.

• Hard cases remain hard: Indo-Iranian and the "Other" family show lower scores (BLEU 60 to 65), driven by non-Latin scripts and tonal complexity. Hebrew reaches only ASR-BLEU 46.1, and Thai shows 77.0 percent WER when Whisper fails to handle tone diacritics.

• Unsegmented languages perform far better under proper tokenization: With language-specific tokenizers (jieba, fugashi, pythainlp), Chinese and Japanese reach tokenized BLEU of 87.6 and 88.9 respectively for CVSS-X-C, and Thai reaches 49.1.

• Voice cloning comes at a modest cost: CVSS-X-T scores 3.21 UTMOS (down from 3.55) with speaker similarity of 0.607 via ECAPA-TDNN. Germanic languages preserve speaker identity better (0.648) than Slavic (0.567), plausibly due to phonetic proximity to English.

• Roughly 3.4 percent of samples could not be voice-cloned because the source English audio lacked sufficient signal quality, a natural consequence of using crowdsourced recordings.

Methodology in Plain English

The construction pipeline has three stages. First, the researchers take English source recordings from Common Voice version 17 and match them by normalized text against the Common Voice version 4 utterances that the original CVSS/CoVoST 2 pipeline used, recovering 91 percent of the original English utterances with verified human audio. Because the matched development set was too small, they topped it up by sampling from the training split.

Second, they machine-translate the English transcripts into 28 target languages using the distilled NLLB-200 model. This model was chosen after benchmarking seven candidate translation systems; selection used an LLM-as-a-judge protocol over 100 English-to-Portuguese samples, rating each output from 1 to 10 on accuracy, fluency, and terminology preservation.

Third, they synthesize target speech with OmniVoice, a multilingual TTS system capable of zero-shot voice cloning across languages. For the canonical variant, two fixed synthetic voices per language are created with ElevenLabs' voice design feature, and the voice assigned to each sample depends on the source speaker's gender metadata. For the transferred variant, the system clones the timbre of the original English speaker into the target language.

For evaluation, they draw a stratified random sample of 200 utterances per language from the dev set, a size chosen through power analysis to yield 95 percent confidence intervals of about plus or minus 0.07 on UTMOS. They then run the synthesized target speech back through Whisper large-v3 and compare the transcription against the translated text prompt, which measures TTS-to-ASR round-trip fidelity rather than translation accuracy in the abstract. Naturalness is scored with UTMOS and speaker similarity with ECAPA-TDNN.

Why This Matters

Impact on research: The corpus removes a structural barrier in S2ST. Before CVSS-X, the community could only train many-to-one systems pointing at English. With CVSS and CVSS-X together, researchers can train bidirectional models and, using English as a pivot, tackle arbitrary language pairs. Because both corpora derive from aligned English sources, the pairing is unusually clean for comparative work. The synthetic construction also sidesteps the imperfect alignment that plagues mined corpora like SpeechMatrix and SeamlessAlign, giving perfect sentence-level alignment between source and target speech.

Real-world applications:

• Automated dubbing and video localization from English source content into 28 languages while optionally preserving the original speaker's voice.
• Real-time or near-real-time interpretation for meetings, conferences, and travel, particularly for the many-to-one and one-to-many directions now covered.
• Cross-lingual voice assistants and customer-support agents that can respond in a user's language using a consistent branded voice.
• Assistive communication tools for speakers of lower-resource languages who need to interact with English-dominant services.

Industry relevance: Media localization, streaming platforms, telephony and conferencing vendors, and voice-assistant developers all require exactly this kind of parallel speech data to train or fine-tune direct S2ST models. The split into canonical and timbre-transferred variants maps directly onto two distinct product needs: consistent brand voices versus personalized voice preservation. The non-commercial license limits deployment in commercial products, which the authors explicitly flag.

Future Directions

• Improving translation fidelity and licensing: Regenerating translations with TranslateGemma-12B would address NLLB-200's weaknesses on low-resource languages and would allow relicensing from CC-BY-NC to Apache 2.0, unlocking commercial use.

• Expanding and diversifying the source data: Incorporating Common Voice v26, which has substantially more speakers, and Common Voice Spontaneous Speech 4.0, which contains natural conversational recordings, would increase scale and provide more realistic voice-cloning conditions.

• Training and evaluating baseline S2ST models: The paper releases data but no trained models; establishing baselines on CVSS-X would validate the corpus quantitatively and give the community reference points.

• Human perceptual validation and cross-checking against human references: The authors note that automated MOS predictors like UTMOS may behave inconsistently across non-Western phonologies and that cross-validation against human-authored references such as CoVoST 2 English-to-15 subsets remains an open benchmark. The inherited asymmetry between evaluating X-to-EN (CVSS) and EN-to-X (CVSS-X) also remains unresolved.

Target Audience

This paper is most useful to speech translation researchers and engineers who need multilingual training or evaluation data, particularly those working on direct S2ST, voice cloning, or multilingual TTS. It also benefits practitioners in media localization and voice-application development who are evaluating whether synthetic corpora are viable substitutes for scarce real parallel recordings. Researchers specifically interested in low-resource and typologically diverse languages will find the per-family breakdown and the discussion of failure modes (Hebrew, Thai, non-Latin scripts) directly actionable. Those seeking a fully trained translation model will need to look elsewhere, since this release is a data contribution with quality analysis rather than a modeling paper.

Authors’ abstract

We introduce CVSS-X, a large-scale synthetic speech-to-speech translation corpus that extends CVSS by reversing the translation direction. While CVSS translates from 21 languages into English, CVSS-X enables translation from English into 28 target languages spanning 12 language families. The corpus comprises approximately 240,000 parallel speech pairs per language, totaling over 16,000 hours, eight times larger than CVSS. We provide two variants: CVSS-X-C with two canonical voices per language, and CVSS-X-T with cross-lingual voice cloning, both fully generated. Evaluation shows comparable translation quality to CVSS with consistent performance across typologically diverse languages. Combined with CVSS, this enables research on bidirectional and multilingual speech-to-speech translation. The code is available at https://github.com/ErmisAI/XVSS-X and the dataset under CC-BY-NC 4.0 license at https://huggingface.co/datasets/lgris/XVSS-X.