Building a Production Greek-English Speech Recognizer

AI summary

Overview

Research area: Automatic speech recognition (eess.AS) — bilingual (Greek-English) production ASR engineering, data-pipeline calibration, ensembling, and evaluation reliability.

Technical level: Intermediate. The vocabulary (WER, language identification, beam search, ROVER) is approachable, but the central result is a quantitative training-step-budget frontier that assumes some familiarity with ASR training and evaluation practice.

Scope: A multi-month industrial report on building and shipping the Sophea bilingual Greek-English ASR system, measured against nine production gates across twenty-three training iterations, two model architectures, a three-model ensemble, and a two-model learned-arbiter system listed on the public Open ASR Leaderboard.

What This Paper Is About

Commercial ASR is close to solved for high-resource languages, but Greek (roughly eleven million speakers), mixed unpredictably with English inside single sentences and recorded mostly over telephone lines and crowded meeting rooms, is not. The authors describe a multi-month engineering program to build a bilingual Greek-English recognizer, Sophea, that must pass nine independently measured production gates (seven word-error-rate ceilings, a 95% language-identification floor, and zero hallucinated output on non-speech audio). The core finding is negative and quantitative: at the tested model scale and configuration, no training-data composition satisfies all nine gates from one model, so the team shipped a family of models, a routing layer, and decode-time fixes instead.

Key Contributions

1. A measured impossibility result at one model scale. Across the first seven versions of a 1.7B-parameter bilingual model, a controlled experiment holding the base checkpoint fixed traces a trade-off between Greek noisy-environment exposure and English language identification: the Greek gate requires at least approximately 1,500 training steps of dense noisy-environment exposure, while the English LID floor tolerates at most approximately 250 steps of that exposure (approximately 1,250 under a rebalanced mix). Because 1500 > 1250 > 250, the required and tolerated intervals never overlap, and the joint operating point sits roughly one WER point outside the frontier.

2. A six-stage data-quality pipeline with anchor-calibrated filters. The pipeline runs an audio-quality score, a words-per-minute plausibility filter, a transcription-confidence filter, a multi-teacher cross-check, a forced-alignment check, and a final assembly stage enforcing language parity. The most consequential fix is recalibrating the UTMOS audio-quality filter against known-good in-domain references rather than a textbook absolute threshold, which changes the discarded share of scored Greek training audio from 98.7% (threshold 3.0) to 10.6% (threshold 1.30).

3. A pre-registered ablation that isolates a hallucination bug to one training-data package. Holding every hyperparameter fixed and varying only which of three candidate ingredients was present, the ablation identified a short-utterance-and-negatives package as the cause of a boilerplate-phrase hallucination mode, rather than the additive noise-augmentation package or the base checkpoint lineage.

4. Two shipped combination strategies with distinct mechanisms. A three-model ROVER confusion-network vote (production bilingual model, a Greek fine-tune of NVIDIA Canary, and a LoRA-adapted Whisper large-v3) raises gate coverage from 4 to 7 of 9 for single models up to 9 of 9, and cuts overlapping-speech WER from 53.35% to 37.87%. Separately, Sophea ASR K1 combines two models via a gradient-boosted per-clip arbiter rather than word-level voting, and is listed on the public Open ASR Leaderboard as sophea/asr-k1 (preview) at 4.26 average WER over the board's eight public English sets.

Main Findings

• No single composition passes all nine gates. Across the first seven versions of the compact model, no training-data composition satisfied all nine production gates; the Greek noisy-environment gate (WER ≤ 25.25) needed at least approximately 1,500 steps of dense exposure, improving from 26.8 to 25.21 by checkpoint 1,750.

• The LID floor caps that same exposure. The English language-identification gate (accuracy ≥ 95) tolerated at most approximately 250 steps of noisy-environment exposure, degrading at roughly 3.5 points per 250 steps; a rebalanced mix pushed tolerance to approximately 1,250 steps at a slower 0.9 points per 250 steps but stalled Greek accuracy near 26, short of its own gate. Both trajectories are single runs with no repeated seeds.

• Acoustic neighborhood, not raw language volume, drives the competition. Adding 835 hours of clean studio-quality English to defend against Greek crowding it out protected nothing; adding 577 hours of noisy, overlapping, meeting-style English (roughly a third as much audio) held accuracy over the same number of training steps.

• An uncalibrated quality filter would have destroyed most of the Greek data. UTMOS at its conventional threshold of 3.0 would have discarded 98.7% of scored Greek audio, including the clean FLEURS benchmark itself. In-domain anchor medians were 2.72 for clean Greek read speech, 2.43 for clean English read speech, 3.35 for community-sourced English, and 1.83 for real Greek business meetings; recalibrating the drop threshold to 1.30 reduced the discarded share to 10.6%, a roughly ninefold difference.

• A specific data defect caused hallucination. An earlier training pool contained roughly 40,000 rows whose paired transcript did not match the audio closely enough, teaching an early model version to invent fluent continuations and pushing one internal hallucination benchmark over 100. The words-per-minute band plus forced-alignment check has since held hallucination rates at zero on every subsequent evaluation battery.

• The three-model ROVER vote reaches nine of nine gates. Individually the three models passed seven, six, and four of nine gates. Voting between the two strongest reached eight of nine, missing only the hallucination gate on a handful of clips out of 500; a serving rule suppressing output when the strongest model detects silence closed the gap to nine of nine, recorded as conditional because it was found after the fact rather than pre-registered.

• Overlapping speech is deleted, not mistranscribed. On real overlapping speech from the AMI Meeting Corpus, the production model deletes close to 40% of words in an overlapping span, and for whichever speaker it locks onto second, recall falls to roughly 18%. Two-model voting reduced overlap-only WER from 53.35% to 37.87%, a 29% relative reduction with a bootstrap 95% confidence interval of

Authors’ abstract

We report a multi-month engineering program to build Sophea, a production bilingual Greek-English automatic speech recognition system. We evaluate the system against nine production gates covering Greek and English word error rate, language identification, and hallucinations on non-speech audio. Across twenty-three training iterations and two model architectures, no training-data composition passed all nine gates simultaneously. Meeting the Greek noisy-environment target required about 1,500 steps of dense domain exposure, while preserving English language identification tolerated only about 250 steps, or about 1,250 with a rebalanced mix that reduced Greek accuracy. We describe a six-stage data pipeline in which calibrating an audio-quality filter against in-domain anchors reduced the discarded share of scored Greek audio from 98.7 percent to 10.6 percent. A pre-registered ablation isolated a hallucination defect to one training-data package. A three-model ROVER ensemble increased gate coverage from 4-7 of 9 for individual models to 9 of 9 and reduced overlapping-speech WER from 53.35 percent to 37.87 percent, a 29 percent relative improvement. A separate learned per-clip arbiter over two models is listed as sophea/asr-k1 (preview) on the public Open ASR Leaderboard, with 4.26 percent average WER across eight public English test sets, and reaches 25.88 percent WER on live Greek noisy-environment traffic. We also document five cases in which a measurement tool produced a plausible but incorrect result and seven substantial approaches that were evaluated but not shipped. No model weights or training data are released; we report methodology and quantitative results only.