Transcription Policy as a Latent Variable: Activating Controllable Verbatim ASR with Word-Level Timing

AI summary

Overview

Research area: Automatic speech recognition (ASR), specifically transcription-style control (verbatim vs. intended output), disfluency detection, word-level timestamping, and cross-lingual transfer. Keywords listed by the authors: speech recognition, human-computer interaction, computational paralinguistics, verbatim transcription, transcription policy, disfluency detection, word-level timing, cross-lingual transfer.

Technical level: Intermediate to Advanced. The paper assumes familiarity with encoder-decoder ASR, cross-attention alignment, beam search, word error rate, and forced alignment.

Scope: The paper argues that transcription style is an uncontrolled latent variable in modern ASR, and shows that explicit mode tags, supervised cross-attention, and a new "verbatimize" task can activate controllable verbatim output and reliable word-level timing using English-only supervision that transfers to German.

Note on completeness: The provided content is truncated mid-way through the timing results (Section 7.2). The verbatimize results and any later sections are not included in the excerpt, so they are reported here only as stated in the abstract and introduction.

What This Paper Is About

Speech transcription requires an implicit policy decision: should the system write down everything that was said, including fillers, repetitions, cut-offs and sounds ("I i [uh] li- like..."), or only the speaker's fluent intended content ("I like...")? Large ASR models such as Whisper, trained on 680,000 hours of heterogeneously annotated data, mix both conventions with no control signal, so the same hesitation may be emitted, omitted, or written in non-canonical forms depending on context. The authors' goal is to turn this implicit policy into an explicit, controllable interface, and to show that fixing the policy also fixes word-level timing and enables a new "verbatimize" annotation task.

Key Contributions

1. Explicit transcription policy via coverage-aware mode tags. Discrete decoder-prefix tokens ([verbatim_1..3], [sound_1,2], [intended_1..5]) parameterize the policy as a binary choice, trained on paired verbatim/intended transcripts for the same audio and partitioned so that tags match each sample's annotation coverage. This disentangles what was said from what to emit.

2. Supervised cross-attention for word-level timing. The ten cross-attention heads that best correlate with ground-truth alignments on TIMIT are trained with word-span targets using averaged-attention cosine loss, turning emergent alignment into a trainable capability without external aligners or tokenizer changes.

3. Verbatimize: a new transcript-conditioned disfluency recovery task. Given audio plus any intended transcript, the model reconstructs a canonical verbatim version by inserting acoustically grounded disfluencies, described as enabling cost-efficient verbatim annotation at scale.

4. A language-agnostic evaluation framework. An alignment-based protocol automatically extracts typed disfluency labels (repetition, filler, sound, cutoff, other) from verbatim transcripts, disambiguates repetitions via modified edit-distance costs, and decomposes WER into content loss rate (CLR) and style mismatch.

Main Findings

• Transcription style is a measurable failure mode, not a theoretical one. Without mode tags, training on 15–85% verbatim data produces an "ambiguous zone" where disfluencies are sometimes transcribed and sometimes omitted; with mode tags, event F1 stays around 80% across all mixing ratios, including at 10% verbatim, where the untagged model reaches only 2%.

• Decoding instability is real and reproducible. Median maximum pairwise character error rate across 10 beam hypotheses (temperature 0.7) on disfluent AMI speech is 15.1% for Whisper and 14.6% for Canary-1B, falling to 8.1% with verbatim mode tags (a 46% reduction) and 11.2% with intended tags. On clean LibriSpeech, divergence is near zero for Whisper-based models; Canary remains at 5.0%.

• WER conflates content errors with style. Verbatim and intended references of the same audio disagree by 3.7% WER on TED-LIUM and 12.1% on AMI. Whisper's reported TED-LIUM WER shifts from 5.7% to 7.3% purely by reference choice while content loss stays identical (3.8%). On AMI, approximately 60% of reported WER reflects style mismatch rather than information loss.

• Timing is ill-defined under policy ambiguity. Base Whisper cross-attention yields 203 ms mean absolute boundary error on TIMIT (read speech) and 568 ms on FluencyBank (disfluent speech).

• The capability is latent, not absent. Freezing the whole model and training only mode-tag embeddings (stage S1) raises event F1 from 12.0% to 53.0% on English and from 10.3% to 78.9% on German, with zero updates to encoder or decoder weights. The progression reported is 12% → 53% → 91% (English) and 10% → 79% → 94% (German).

• Tags beat target-language data. With zero German verbatim training data, tagged fine-tuning reaches 93.8% German event F1 versus 60.1% for the identical model without tags — a 34 percentage point gap. Adding 10k German verbatim samples gives only marginal gains (93.8% to 95.2% German event F1). The final system, FT_all, is reported as the fully trained model; the introduction cites 94% disfluency event F1 for German zero-shot.

• Intended-mode quality is preserved. Intended-mode errors are dominated by insertion rate (iIR), i.e. disfluency leakage. Untagged models leak heavily (iIR up to 15.9% English and 15.9% German in FT_0 without tags), while mode tags suppress leakage to 3–4%. Substitution and deletion rates stay below 3% and 1% respectively.

• Supervised attention outperforms forced alignment on disfluent speech. With averaged-head loss plus inference-time sharpening (τ = 3), the method reaches 36 ms MAE on TIMIT, 102 ms on FluencyBank, and 55 ms on Thorsten (German, read speech, with no German timing supervision). MAE is reduced 5–6× versus base Whisper. MFA is best on read speech (19 ms on TIMIT) but degrades to 142 ms on FluencyBank; WhisperX goes from 66 ms on TIMIT to 200 ms on FluencyBank. On German Thorsten, the English-trained model beats WhisperX (89 ms) and Canary-1B (103 ms), while CrisperWhisper+s reaches 57 ms despite extensive German training.

• Averaging heads before the loss matters. Averaged-head loss beats per-head loss (36 vs. 45 ms on TIMIT; 102 vs. 119 ms on FluencyBank), and MAE decreases as heads are added from 1 to 10 (78 → 39 ms on TIMIT) with diminishing returns beyond 8.

• Sharpening without supervision backfires. Applying attention sharpening to base Whisper helps slightly on read speech (203 → 197 ms on TIMIT) but worsens disfluent speech dramatically (568 → 944 ms on FluencyBank).

• Verbatimize claims (from abstract and introduction). Rare-word recall is reported to rise from 6.8% to 96.1%. The corresponding results table is not included in the truncated content.

Methodology in Plain English

The authors start from the official OpenAI Whisper-medium checkpoint and add new atomic tokens to its vocabulary: two filled-pause tokens ([uh], [um]), twelve paralinguistic sound tokens ([laughter], [cough], [sigh], [breath], [lipsmack], [sniff], [throat-clearing], [yawn], [noise], [crying], [scream], [sneeze]), ten mode tags, and two delimiters (<sot>, <eot>) for the verbatimize prompt.

To make style controllable, they prepend mode tags to the decoder input. Because real corpora are annotated inconsistently — some mark disfluencies but not sounds, some the reverse — they partition the tags by annotation coverage, so a sample only receives tags for the phenomena its labels actually cover. This stops the model from being penalized for "predicting a sound in one sample and omitting the identical event in another" purely because annotations are incomplete.

Training uses paired supervision: for the same audio, the mode is randomly chosen and the corresponding transcript is the target. This teaches a division of labour in which the encoder representations hold all available content (words, disfluencies, sounds) and the tags decide the emission policy. Training has two stages: Stage 1 freezes all pretrained parameters and trains only the new token embeddings for 1 epoch at learning rate 5×10⁻⁴; Stage 2 unfreezes the decoder for 3 epochs at 1×10⁻⁵. Batch size is 160 with 4-step gradient accumulation, fp16, on a single NVIDIA A100.

For timing, they choose the k = 10 cross-attention heads whose unsupervised patterns best match ground-truth alignments on TIMIT (added greedily by marginal improvement), average their attention, and minimize the mean cosine distance between that averaged attention and a binary target that is 1 inside a word's time span and 0 elsewhere. Frames are 20 ms (Δ = 0.02 s; F = 1500 for a 30 s input). At inference they avoid tokenizer changes by decoupling pause detection from attention: an energy-based virtual blank model uses the 10th

Authors’ abstract

Modern ASR models trained on heterogeneously annotated data treat transcription style (verbatim vs. intended) as an uncontrolled latent variable, causing measurable decoding instability, evaluation confounding (up to 60% of reported WER attributable to style mismatch), and unreliable word-level timing. We show that models already encode both styles; the challenge is controlled activation. Using coverage-aware decoder task tokens trained on parallel verbatim/intended transcript pairs, we raise German disfluency F1 from 10% to 79% zero-shot, despite English-only training. Full English-only fine-tuning surpasses all baselines in verbatim accuracy, disfluency detection, and intended-mode quality across both languages. We further introduce supervised cross-attention fine-tuning that improves word-level timestamps on disfluent speech beyond forced-alignment baselines. Finally, we propose verbatimize, a new task enabling scalable creation and enrichment of speech corpora with high-quality canonical verbatim transcriptions.