Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual and Long-Form Speech Recognition Evaluation

AI summary

Overview

• Research area: Automatic speech recognition (ASR) evaluation and benchmarking, with a focus on reproducible, multilingual, and long-form transcription comparisons.
• Technical level: Intermediate. The paper assumes familiarity with ASR concepts such as word error rate, Conformer and Whisper encoders, and CTC/TDT decoders, but the benchmarking methodology is described accessibly.
• Scope: The paper describes and reports results from the ASR Leaderboard, a community-contributed platform that evaluates 86 open-source and proprietary speech recognition systems across 12 datasets on English short-form, multilingual short-form, and English long-form tracks.

What This Paper Is About

As speech recognition models proliferate, it has become hard for developers to know which baselines to compare against and hard for users to know which model, open or proprietary, fits their accuracy and efficiency needs. Existing benchmarks also lean heavily on English and short audio. The paper presents the ASR Leaderboard, a platform that standardizes accuracy (word error rate) and speed (inverse real-time factor) evaluation across many models, toolkits, and datasets so results are directly comparable.

Key Contributions

1. An interactive leaderboard covering 86 systems from 26 organizations over 12 datasets, spanning 74 open-source models plus proprietary systems, with evaluation scripts open-sourced so new models and datasets can be added.
2. Standardized cross-toolkit evaluation: the paper harmonizes how models are run across four open-source toolkits (ESPNet, NeMo, SpeechBrain, Transformers), seven commercial APIs (AssemblyAI, Aqua Voice, Google, ElevenLabs, Rev AI, Speechmatics, Zoom), and model-specific repositories, with unified text normalization before WER is computed.
3. A multilingual benchmark currently covering German, French, Italian, Spanish, and Portuguese, built on CoVoST-2, FLEURS, and MLS.
4. A dedicated English long-form track for audio longer than 30 seconds, motivated by the fact that 32 percent of open models in the leaderboard derive from Whisper's pretrained encoder or architecture, and by differing chunking and context-window strategies.

Main Findings

• Accuracy leaders use Conformer encoders with transformer or LLM decoders. On short-form English, models pairing a Conformer-based encoder with a transformer-based decoder achieve the best average WER. The top entry, Cohere Labs Transcribe, reaches 5.42 average WER with an RTFx of 525; Zoom Scribe v1 follows at 5.47.
• CTC and TDT decoders trade accuracy for speed. The highest-ranking TDT model, NVIDIA Parakeet TDT 0.6B v2, places 10th with 6.05 average WER but an RTFx of 3390. The best CTC model, NVIDIA Parakeet CTC 1.1B, ranks 34th at 7.40 average WER with an RTFx of 2730. The highest RTFx in the displayed short-form table is NVIDIA FastConformer CTC Large at 6400 with 8.96 average WER.
• Conformer-based systems are 3.77 times faster than Whisper-based ones at comparable decoder type. Restricting to models with transformer/LLM-based decoders (10 Conformer-based and 22 Whisper-based), Conformer systems average an RTFx of 758 versus 201 for Whisper-based models.
• Whisper encoders remain popular for language coverage. Whisper-based encoders are pretrained on large-scale multilingual data and support up to 99 languages. Fine-tunes such as Nyra Health CrisperWhisper (6.67 WER) and Mistral AI Voxtral Small 24B (6.62 WER) beat OpenAI Whisper Large v3 (7.44 WER) on the English short-form track.
• Self-supervised encoders lag on accuracy but cover the most languages. They make up 19 percent of the approaches benchmarked; the best, Meta Omnilingual ASR LLM 7B v2, ranks only 53rd on short-form English (8.14 average WER), while the family enables systems for 1600+ languages.
• Closed-source models lead on multilingual. ElevenLabs Scribe v2 posts the best average WER at 2.67 across German, French, Italian, Spanish, and Portuguese; Assembly AI Universal 3 Pro follows at 3.23. The best open-source model shown is Mistral AI Voxtral Small 24B at 3.70 (RTFx 42.0).
• Broader language coverage costs English accuracy. NVIDIA's Parakeet TDT 0.6B v3 adds multilingual support over v2, and Canary 1B v2 expands from 4 to 25 languages, but in both cases broader coverage comes at the cost of English transcription accuracy.
• Closed-source models also lead long-form English by a distinct gap. ElevenLabs Scribe v2 is best at 7.32 average WER, ahead of AssemblyAI Universal 3 Pro (8.34) and Speechmatics Enhanced (8.80). The best open-source results are Cohere Labs Transcribe at 9.73 (RTFx 418) and NVIDIA Parakeet TDT 0.6B v3 at 10.7 (RTFx 1000).
• Fast long-form options exist among open models. NVIDIA Parakeet CTC 1.1B reports 12.9 average WER at RTFx 2790, and NVIDIA Parakeet CTC 0.6B reports 13.7 at RTFx 4383.
• Architecture distribution among the 78 open-source models counted in the encoder/decoder table: Conformer-based encoders total 28, Whisper 25, self-supervised 15, and custom 10; decoder counts are 32 transformer, 26 CTC, 8 RNN-T/TDT, and 12 LLM. Some models use hybrid architectures and are counted twice.

Methodology in Plain English

The team defines three evaluation tracks: short-form English (audio under 30 seconds, matching Whisper's receptive field), multilingual (German, French, Italian, Spanish, Portuguese), and long-form English (audio over 30 seconds). Audio for the short-form track is segmented into chunks of at most 30 seconds, with a small number of exceptions. Each model is run through standard evaluation scripts so that toolkits, commercial APIs, and model-specific repositories are measured the same way.

Before scoring, all transcripts are normalized to remove punctuation and casing and to apply an English text normalization pipeline closely following Whisper's, including number normalization, spelling standardization, and removal of filler words. Two metrics are reported: word error rate for accuracy and inverse real-time factor (RTFx = total audio duration divided by transcription time) for speed, with higher RTFx meaning faster inference. Measurements were run on an NVIDIA A100-SXM4-80GB GPU (driver 560.28.03, CUDA 12.6) with batch size 64 where memory allowed, reduced adaptively (48, 32, 16, and so on) when needed. The authors note that absolute RTFx depends on hardware but that their measurements are comparable to one another because they were collected under the same setup.

To limit test-set contamination risk, each track includes at least one evaluation dataset under a non-commercial license: TED-LIUM v3 for short-form English, CoVoST-2 for multilingual, and TED-LIUM v3 and CORAAL for long-form. Contributors add new models by opening a pull request with an evaluation Python script, a Bash script that calls it per dataset and model, and self-reported metrics, which the maintainers then verify in their own environment. To date 29 pull requests have been merged adding 56 models, plus 34 more pull requests for other aspects and fixes.

Why This Matters

Impact on research. A single metric and a single dataset are not enough to characterize an ASR system, and there is no catch-all model, but before this work it was unclear which baselines were comparable and how they had been run. By standardizing normalization, metrics, and execution across toolkits and APIs, the leaderboard makes cross-paper and cross-vendor comparisons meaningful, and it extends evaluation beyond English and short utterances.

Real-world applications:

• Selecting a production ASR model where latency matters, using RTFx alongside WER to choose CTC or TDT decoders for batched or streaming workloads.
• Transcribing long recordings such as meetings, earnings calls, presentations, and sociolinguistic interviews, where the long-form track and datasets like AMI, Earnings21/22, CORAAL, and TED-LIUM v3 are directly relevant.
• Deploying multilingual transcription in German, French, Italian, Spanish, or Portuguese, where the multilingual table shows per-language WER rather than a single averaged number.
• Comparing a commercial API against open-source alternatives before committing to a vendor or to self-hosting.

Industry relevance. The evaluated systems come from 26 organizations spanning both industry and academia, including NVIDIA, Meta/Facebook, OpenAI, Hugging Face, IBM, Google, Microsoft, Mistral AI, Cohere Labs, and multiple commercial API vendors. The findings that Conformer-plus-transformer systems lead on accuracy while CTC/TDT systems lead on speed give engineering teams a concrete basis for architecture and deployment decisions.

Future Directions

• Expand languages and domains, including far-field speech, since the current multilingual track covers five languages and the leaderboard has not yet addressed all deployment conditions.
• Add metrics beyond WER, such as token error rate, to capture errors that word-level scoring misses.
• Explore underrepresented encoder-decoder combinations in the architecture table, where some pairings (for example Whisper encoders with RNN-T/TDT decoders, or self-supervised encoders with non-CTC decoders) currently have no entries.
• Use private evaluation sets to further reduce the risk of test-set contamination, and consider splitting tasks based on whether models explicitly model disfluencies or are expected to produce verbatim transcripts.

Target Audience

This paper is most useful to ASR practitioners choosing or benchmarking models, including ML engineers deploying transcription systems, researchers who need standardized baselines for new architectures, and product teams comparing open-source and commercial APIs. Speech researchers interested in multilingual or long-form evaluation methodology, and contributors who want to add their own models to a community leaderboard, will also find the described contribution process directly applicable.

Authors’ abstract

We present the Open ASR Leaderboard, a reproducible benchmarking platform with community contributions from academia and industry. It compares 86 open-source and proprietary systems across 12 datasets, with English short- and long-form and multilingual short-form tracks. We standardize word error rate (WER) and inverse real-time factor (RTFx) evaluation for consistent accuracy-efficiency comparisons across model architectures and toolkits (e.g., ESPNet, NeMo, SpeechBrain, Transformers). We observe that Conformer-based encoders paired with transformer-based decoders achieve the best average WER, while connectionist temporal classification (CTC) and token-and-duration transducer (TDT) decoders offer superior RTFx, making them better suited for long-form and batched processing. All code and dataset loaders are open-sourced to support transparent, extensible evaluation. We present our evaluation methodology to facilitate community-driven benchmarking in ASR and other tasks.