The Pulse
Hugging Face Launches an Open TTS Leaderboard for Multilingual Models
Hugging Face has launched an open text-to-speech leaderboard that compares model intelligibility, speed and speaker similarity. Its evaluation covers multilingual generation, voice cloning and streaming performance, while letting users list
AI.info Team · Written with AI ·
Hugging Face launched its Open TTS Leaderboard on September 30, 2026, to compare open text-to-speech systems across intelligibility, speed and speaker similarity. The company says the Hub already hosts more than 8,000 TTS models, while existing comparisons often rely on human votes that take longer to gather and can be difficult to standardize. The new leaderboard pairs automated measurements with a listening interface where users can compare outputs and submit preferences.
Three metrics, not one winner
The leaderboard measures intelligibility through word error rate (WER) and character error rate (CER), comparing the input text with a transcript generated by Qwen3 ASR. It measures batched, offline generation speed as inverse real-time factor, or RTFx, and evaluates streaming latency through time-to-first-audio, or TTFA. A third measure, speaker similarity, compares WavLM speaker embeddings from generated speech and a reference clip.
Those measures answer different questions, rather than collapsing quality into a single score. WER and CER estimate whether a model speaks the requested words; RTFx and TTFA describe how quickly it produces audio; speaker similarity estimates how closely a cloned voice matches its reference. Hugging Face says an automated evaluation can take a few hours, compared with the weeks an arena may need to collect votes.
“The Open TTS Leaderboard does not replace human preference ranking.”
Eric Bezzam, Steven Zheng, Eustache Le Bihan and mrfakename, Hugging Face
The team cautions that automatic speech-recognition scores only act as a proxy for intelligibility. The measures do not directly capture naturalness, expressiveness or what listeners prefer, so the leaderboard includes a “Listen” tab for side-by-side comparisons and voting. Users must sign in with a Hugging Face account to vote, a measure the team says helps filter spam and bots.
English scores do not stand in for other languages
The default ranking uses macro-average WER on English test splits from Seed TTS Eval and CV3 Eval. Users can select languages to examine multilingual performance, but coverage differs by benchmark: Seed TTS Eval supplies audio for English and Chinese, while scores for other languages come from CV3 Eval’s zero-shot tests. For Chinese, Japanese and Korean, the leaderboard reports character error rate because those languages use character-based writing systems.
Hugging Face names hexgrad/Kokoro-82M, Supertone/supertonic-3 and fishaudio/s2-pro among the leading models on the combined English WER measure. For multilingual performance, it points to k2-fsa/OmniVoice, fishaudio/s2-pro and FunAudioLLM/Fun-CosyVoice3-0.5B-2512 as strong performers. Those examples illustrate why a model’s English score alone cannot establish how it will perform in another language.
A voice-cloning option filters comparisons to models that support cloning in the selected languages. The leaderboard then adds speaker similarity to its results and displays trade-offs among similarity, inference speed and model size. Hugging Face also notes that some models’ average WER improves when supplied with a reference audio clip.
Streaming tests measure when speech begins
A separate Streaming tab ranks models by TTFA, the wait between sending a prompt and receiving playable audio. For a model with a streaming API, the test records the arrival of its first audio chunk; for a model without streaming, it measures the time to generate the complete utterance. Each model runs one audio request at a time against the same 50 English prompts from CV3 Eval, using its default voice.
The tests run on an H200 GPU, with CPU results available for a smaller, growing group of models. Hugging Face drops the first three runs as warm-up and reports the median TTFA from the remaining runs. The company points to kyutai/pocket-tts as a strong streaming option on both GPU and CPU.
Human listening remains part of the comparison
The project is aimed at open-source models, which Hugging Face says appear less often in arena-style rankings. As one illustration, the post counts 16 open-weight models among 92 on Artificial Analysis as of September 30, 2026, and says Voice Arena shows a similar skew. The team argues that open systems can be harder for arena operators to host than API-accessible commercial models.
The leaderboard is presented as a community project: users can listen, compare, and vote, while the team invites suggestions for models, datasets and metrics. Hugging Face says it plans to open-source the evaluation scripts, but the September 30 announcement does not yet provide them. For now, the release combines standardized measurements with human listening rather than treating either as a complete account of TTS quality.