Pisets: A Robust Speech Recognition System for Lectures and Interviews

AI summary

Overview

Research area: Automatic speech recognition (ASR) for long-form Russian audio — specifically lectures, interviews, and meeting recordings.

Technical level: Advanced. The paper assumes familiarity with Wav2Vec2, Whisper, the Audio Spectrogram Transformer (AST), curriculum learning, BIRM fine-tuning, and test-time augmentation.

Scope in one sentence: The paper describes "Pisets," a three-stage offline Russian speech-to-text pipeline (Wav2Vec2 segmentation → AST false-positive filtering → Whisper transcription) with uncertainty modeling, evaluated against WhisperX and plain Whisper on 20–40 minute lecture recordings.

What This Paper Is About

Autoregressive ASR models like Whisper transcribe long audio well but sometimes hallucinate whole phrases that were never spoken, and they decode slowly because tokens are generated one at a time. The authors build a pipeline for Russian scientific speech that segments audio more intelligently before Whisper ever runs, filters out segments that probably contain no real speech, and flags uncertain words for human review — with the goal of cutting errors and hallucinations while staying practical for lectures and interviews.

Key Contributions

1. A three-component architecture: Wav2Vec2 for speech-segment detection ("VAD on steroids"), the Audio Spectrogram Transformer (AST) trained on the Audioset ontology for filtering false positives from that detector, and Whisper for final transcription.
2. Domain adaptation for Russian: curriculum learning over open Russian corpora (Golos, Russian Librispeech, RuDevices), and BIRM (Bayesian Invariant Risk Minimization) fine-tuning of Whisper over three corpora (Russian Librispeech, Taiga Speech, Podlodka Speech) using a "speech environment" concept.
3. Practical uncertainty modeling: a comparison of three straightforward uncertainty sources — Whisper token scores, disagreement between Whisper and Wav2Vec2 predictions, and disagreement between Whisper on original versus stretched audio (audio stretched by 33% via polyphase filtering, upsampling factor 3, downsampling factor 4).
4. Public release: source code at https://github.com/bond005/pisets, and the resulting Whisper model released under the Apache 2.0 license on HuggingFace.

Main Findings

• Pisets beats WhisperX on long lectures: WER 0.1065 versus 0.1683, and BERT-score 0.9652 versus 0.9479 (Table 2, long audio lectures dataset).
• Smart chunking is the biggest single win: On quiet audio, Whisper with uniform chunking scored WER 0.1995 / BERT-F1 0.9102, while Whisper with Wav2Vec2 "smart" chunking scored WER 0.1065 / BERT-F1 0.9652. Adding AST gave WER 0.1109 / BERT-F1 0.9588.
• Noise changes the ranking: On loud audio (SNR = 1 dB), uniform chunking scored WER 0.3825 / BERT-F1 0.8508; smart chunking 0.2119 / 0.9169; smart chunking plus AST 0.2133 / 0.9160.
• Whisper-Podlodka-V3 trades quiet-audio accuracy for noise robustness: Whisper-Large-V3 scored WER 0.0931 / BERT-F1 0.9661 on quiet noises but degraded to WER 0.2409 / BERT-F1 0.9151 on loud noises. Whisper-Podlodka-V3 scored WER 0.1199 / BERT-F1 0.9644 quiet, and held up better at WER 0.2119 / BERT-F1 0.9169 under noise (Table 1).
• Speed: On noised audio, Wav2Vec2 "smart" chunking reduced maximum inference time from 192.045 s (uniform chunking) to 152.524 s, and with AST to 151.923 s. Average time went from 136.377 s to 133.219 s (132.495... reported as 131.495 with AST); median time was 121.090 s for uniform chunking versus 134.918 s for smart chunking and 130.809 s with AST.
• Uncertainty is practically useful: Marking only about 5% of predicted words as uncertain captures 35% of all errors (excluding errors caused by missed words). No model-disagreement or ensembling method reliably outperformed Whisper scores as an uncertainty source; Whisper scores are shown as a line for different score thresholds in Figure 2.
• Word-score aggregation: Reducing Whisper token log-probabilities to word scores with min and sum performed on par and better than mean.

Methodology in Plain English

Instead of handing a long recording straight to Whisper, the system first runs Wav2Vec2 over the audio. Wav2Vec2 is normally a recognizer, but here it doubles as a sensitive speech detector that decides where real speech begins and ends, so the audio can be chopped into sensible chunks and Whisper can be run on all chunks in parallel rather than decoding the whole file sequentially. Russian recognition was strengthened by training with curriculum learning — starting on clean, well-annotated data and gradually adding accents, background noise, and harder acoustic conditions — using the Golos, Russian Librispeech, and RuDevices corpora.

Next, the Audio Spectrogram Transformer (AST), trained on the Audioset ontology, looks at each detected segment and throws out ones that are probably not speech at all. This is the step most responsible for suppressing Whisper hallucinations, because Whisper is only asked to transcribe segments that survived the filter. Whisper then produces the final transcript, with fine-tuning driven by the BIRM algorithm and a curated, low-error, acoustically diverse "speech environment."

For uncertainty, the authors align Whisper's and Wav2Vec2's transcriptions with sequence matching, then split or merge the differences so they correspond to linguistically sensible edits (for example, merging a deletion of "no" and a replacement "thing" → "nothing"). Heuristics drop differences that look like transliterations between English and Russian letters, and an optional language-model check plus a look-ahead algorithm reduces noise from the weaker model. Separately, Whisper's own token probabilities are mapped back to whole words, and a threshold marks each word as certain or uncertain. Uncertainty quality is measured as a binary classification problem with two metrics forming a Pareto frontier: the uncertainty ratio (share of words flagged) and error-detection recall (share of incorrect words flagged).

Why This Matters

The paper argues that hallucinated or wrong transcriptions can disrupt downstream LLM-based summarization and potentially harm an individual's reputation — so knowing which words to distrust is as valuable as the raw transcript.

Real-world applications:

• Transcribing university lectures and scientific talks in Russian, the exact setting of the 20–40 minute test set spanning philology, mathematics, and history.
• Journalistic interviews and meeting recordings, the use case named in the introduction.
• Live dictation-style events: Pisets took part in the "Total Dictation" event on April 20, 2024, in a 200-person classroom, missing eight words out of 276 total, making two spelling errors and three punctuation errors, and receiving a grade of B ("good") from professional philologist and linguist experts.
• Feeding uncertain-word markup into LLM post-processing so the model can resolve disagreements using linguistic knowledge and common sense.

Industry relevance: All components are offline and open — the code is on GitHub and the fine-tuned model is released under Apache 2.0 on HuggingFace — making the pipeline directly reusable for production transcription services that need Russian support and hallucination control.

Future Directions

• Multilingual expansion: Adapting the system to multilingual datasets, including recognizing English speech from non-native speakers, plus Bengali and Spanish.
• Better uncertainty handling: The authors report only preliminary experiments on feeding uncertainty-marked text into an LLM; they say this may reduce WER but leave it outside the current work.
• Homophones and near-homophones: The stated limitation is insufficient performance on words and phrases with similar phonetic characteristics; the authors want to add pragmatic understanding via instructions/prompts, and plan to incorporate large multimodal models such as Qwen-Audio, because Whisper-like architectures follow such instructions poorly.
• Tokenization accounting: The paper notes that word probability is split across different Whisper token sequences (e.g., " cat", " Cat", "Cat", " C"+"at") and that this was not accounted for, leaving it for future work.

Target Audience

Researchers and engineers building production ASR systems for long-form, noisy, or non-English audio; practitioners who need a concrete recipe for reducing Whisper hallucinations; and anyone studying uncertainty estimation in speech recognition. The paper is also relevant to linguists and journalists interested in the practical accuracy ceiling of Russian transcription, though the heavy use of model names and pipeline terminology makes it most accessible to readers already comfortable with ASR architectures.

Authors’ abstract

This work presents a speech-to-text system "Pisets" for scientists and journalists which is based on a three-component architecture aimed at improving speech recognition accuracy while minimizing errors and hallucinations associated with the Whisper model. The architecture comprises primary recognition using Wav2Vec2, false positive filtering via the Audio Spectrogram Transformer (AST), and final speech recognition through Whisper. The implementation of curriculum learning methods and the utilization of diverse Russian-language speech corpora significantly enhanced the system's effectiveness. Additionally, advanced uncertainty modeling techniques were introduced, contributing to further improvements in transcription quality. The proposed approaches ensure robust transcribing of long audio data across various acoustic conditions compared to WhisperX and the usual Whisper model. The source code of "Pisets" system is publicly available at GitHub: https://github.com/bond005/pisets.