PROFASR-BENCH: A Benchmark for Context-Conditioned ASR in High-Stakes Professional Speech

AI summary

Overview

• Research area: Automatic speech recognition (ASR) evaluation, specifically context-conditioned/prompt-conditioned ASR in high-stakes professional domains (finance, medicine, legal, technology).
• Technical level: Intermediate — the paper assumes familiarity with WER/SER, encoder–decoder ASR, audio language models, and contextual biasing, but its core argument is conceptual rather than mathematical.
• Scope: The paper introduces ProfASR-Bench, a prompt-conditioned ASR evaluation suite of entity-dense professional utterances paired with natural-language prompts, and uses it to show that current ASR systems largely ignore lightweight textual context.

What This Paper Is About

ASR systems do well on general benchmarks (the paper notes state-of-the-art WER below 5% on LibriSpeech), but they still fail on rare, domain-specific terms in professional settings — the paper's motivating example is a model confusing the antihypertensive hydralazine with the antihistamine hydroxyzine, turning a near-homophone error into a different-medication directive. The goal is to build a benchmark where every test utterance is paired with usable side information (a domain cue, a speaker profile, or the preceding sentence) so that the benefit of context can be measured in a matched, within-utterance way. The headline claim is that despite being nominally "promptable," today's systems underuse that context — a phenomenon the authors name the context-utilization gap (CUG).

Key Contributions

1. A public prompt-conditioned ASR evaluation suite for professional talk spanning four domains (Finance, Medicine, Legal, Technology), distributed on Hugging Face with code on GitHub.
2. A standardized "context ladder" with matched no-context versus with-context conditions: no-prompt, profile, domain+profile, oracle (gold transcript as prompt), and adversarial (mismatched prompt), evaluated on identical utterances.
3. Entity-aware and slice-aware reporting — NE-WER and Entity-F1 over typed entity spans, plus accent and gender slice analysis with paired confidence intervals — alongside conventional WER and SER.
4. A reproducible testbed for comparing fusion strategies across model families (encoder–decoder ASR versus audio language models), designed so any subset can be deterministically regenerated.

Main Findings

• Whisper-Small wins on WER in every slice. Overall WER was 10.0% for Whisper-Small versus 12.1% (Whisper Base), 14.3% (Whisper Tiny), and 24.3% (Qwen 2.5 Omni 3B). By domain, Whisper-Small scored 13.3% (Financial), 8.5% (Legal), 15.8% (Medical), and 2.3% (Technical).
• Qwen 2.5 Omni 3B wins on SER despite higher WER. Overall SER was 37.9% for Qwen versus 52.4% (Whisper-Small), 62.8% (Whisper Base), and 69.2% (Whisper Tiny). The paper attributes this to Qwen producing more fully correct sentences but making heavier edits when it errs, whereas Whisper distributes smaller errors more evenly.
• Domain difficulty tracks entity density and terminology. Technical is comparatively easy across models and Medical is hardest, with Legal between Medical and Finance. Medical SER reached 85.3% for Whisper Tiny and 54.3% for Qwen.
• Lightweight textual context barely moves average WER for Whisper-small. Overall WER was 9.98% (no-prompt), 9.95% (profile), 9.95% (domain+profile), 9.92% (oracle), and 9.95% (adversarial) — deltas of −0.03, −0.03, −0.06, and −0.03 percentage points respectively.
• Even the oracle ceiling yields almost nothing. Providing the gold normalized transcript as the prompt produced only about a −0.06 percentage point directional WER change, which the paper describes as a marginal directional decrease.
• Adversarial prompts do not reliably degrade performance. Despite intentionally wrong, domain-mismatched prompts (for example, telling the model a financial utterance is about cooking recipes), adversarial overall WER was 9.95%, a −0.03 pp change from no-prompt.
• Domain-wise deltas echo the same pattern. For Whisper-small, Medical showed the most favorable direction under informative prompts (up to −0.18 pp under oracle, −0.15 pp under domain+profile), Technical up to −0.06 pp, Financial was flat or slightly worse under domain+profile (+0.09 pp), and adversarial remained benign (+0.02 pp Financial, +0.01 pp Legal).
• Accent and gender gaps vary across models. The British-minus-American accent gap was +3.3 pp for Qwen 2.5 Omni 3B versus +0.5 pp (Whisper Tiny), +0.8 pp (Whisper Base), and +0.5 pp (Whisper Small). The female-minus-male gender gap was +2.7 pp for Qwen, −0.4 pp (Tiny), +0.2 pp (Base), and +0.4 pp (Small) — showing slice gaps can move independently of average scores.
• Statistical significance is assumed, not demonstrated. The paper states the context-condition differences fall within overlapping 95% confidence intervals and "by assumption" are not statistically significant; Table 2 notes that 95% paired bootstrap CIs are to be added in parentheses for camera-ready, so detailed interval values are not reported in the paper content.
• Entity-centric gains are modest and model-dependent. The abstract reports only modest, model-dependent improvement on information-bearing tokens, which the authors take as evidence that average WER can stay flat even where targeted units matter most.

Methodology in Plain English

The authors built a synthetic benchmark rather than collecting real professional recordings. For each of four domains, they sampled a professional scenario (earnings update, discharge summary, motion hearing, incident postmortem) together with a persona defined by role, region, and seniority. An instruction-tuned LLM (Claude 3.7) drafted text under soft constraints on which entity types must appear, the discourse structure appropriate to the domain, and lexical phenomena that typically break ASR (acronyms, code names, homophones, numeric expressions). Utterances were then synthesized with Kokoro TTS (82M), giving four voice variants (American/British × male/female) to enable accent and gender slicing, and every record was automatically validated for register, entity realization, prompt–utterance coherence, and acoustic quality.

Each record carries a canonical transcript (truth), an LLM-assisted written-form normalization (normalized_truth), a natural-language prompt, a speaker_profile, a voice identifier with accent and gender, typed named entities as {value, type} pairs, an asr_difficulty scalar, sentiment labels, and error_targets flagging brittle tokens.

For scoring, both references and hypotheses go through a deterministic normalization (spoken-to-written canonicalization of numbers, dates, units, currency and acronyms; punctuation removal; lowercasing; whitespace and hyphen normalization) before whitespace tokenization. WER is Levenshtein-based (S+D+I)/N × 100%, SER is the fraction of utterances with any non-zero edit, NE-WER restricts the alignment to tokens inside annotated entity spans, and Entity-F1 measures span-level precision and recall. Named entities are extracted from reference text with a constrained Claude 4 NER prompt using a closed type inventory, and decoding prompts follow fixed templates that exclude reference text to prevent leakage.

The evaluation protocol then runs Whisper (Tiny, Base, Small) and Qwen 2.5 Omni 3B through the five context conditions, minimizing formatting variance so that any measured difference is attributable to the prompt.

Why This Matters

For research, the paper reframes context-conditioned ASR as an unsolved control problem rather than a solved engineering convenience: the same utterances, same model, and same acoustics across five prompt conditions isolate how much the model actually uses side information, and the oracle–adversarial bracketing provides both a ceiling and a robustness probe. A benchmark where the oracle prompt produces roughly a −0.06 pp WER change is a strong signal that prefix-style prompting alone is not a viable control channel.

Real-world applications:

• Clinical dictation and medication orders: the hydralazine/hydroxyzine confusion shown in the paper is exactly the error class where a wrong drug name changes a directive rather than merely degrading readability.
• Financial earnings calls and analyst workflows: utterances densely packed with tickers, financial institutions, and metrics such as EBITDA margins, where a misheard ticker propagates into downstream data.
• Legal transcription: statutes, legal documents, and legal roles, where a misrecognized citation or modality term changes meaning.
• Technology/incident response: software names, databases, protocols, and version strings, which are rare tokens that ASR systems routinely mangle.

For industry, the benchmark ships with a reporting contract specifying per-context and per-slice metrics with paired confidence intervals, which matters for teams who need to justify paying for context-aware transcription and for vendors who claim prompt-based personalization works.

Future Directions

• Stronger fusion mechanisms. The paper explicitly calls for learned relevance gating, phrase/lexicon encoders, contextual RNN-T joints, and constrained or biased decoding, since prefix-style prompting left average WER essentially unchanged.
• Training objectives that reward context use on entity spans, plus calibration strategies that decide when to trust or ignore a prompt — motivated by the finding that adversarial prompts did not reliably hurt performance, suggesting the model may not be reading prompts either way.
• Robustness to plausible-but-wrong prompts and fairness-aware adaptation that improves critical entities without widening demographic gaps, given that accent and gender gaps for Qwen (+3.3 pp and +2.7 pp) moved independently of average scores.
• Extending beyond the current corpus. The authors list human-collected speech, additional languages and varieties, multi-turn interaction, overlapping talk, and realistic acoustic conditions as important future work, noting the benchmark is synthetic in origin, English-focused (US/UK accents), and presently single-turn.

Target Audience

Researchers and engineers working on ASR robustness, contextual biasing, and audio language models; evaluation and benchmarking specialists who need entity-aware and fairness-sliced metrics rather than average WER alone; and applied teams in clinical, financial, legal, and technical transcription who must decide whether prompt-based context conditioning is worth deploying. Readers looking for a new model architecture will not find one here — the contribution is a measurement instrument and a negative result about prompt utilization.

A note on scope: the paper content provided does not report the total number of utterances or records in ProfASR-Bench, the actual numeric values of the 95% paired bootstrap confidence intervals, or an evaluation using the full context ladder for models other than Whisper-small. The title also appears in two forms — the metadata title emphasizes "Context-Conditioned ASR in High-Stakes Professional Speech," while the in-paper title emphasizes "Exposing the Context-Utilization Gap."

Authors’ abstract

Automatic Speech Recognition (ASR) in professional settings faces challenges that existing benchmarks underplay: dense domain terminology, formal register variation, and near-zero tolerance for critical entity errors. We present ProfASR-Bench, a professional-talk evaluation suite for high-stakes applications across finance, medicine, legal, and technology. Each example pairs a natural-language prompt (domain cue and/or speaker profile) with an entity-rich target utterance, enabling controlled measurement of context-conditioned recognition. The corpus supports conventional ASR metrics alongside entity-aware scores and slice-wise reporting by accent and gender. Using representative families Whisper (encoder-decoder ASR) and Qwen-Omni (audio language models) under matched no-context, profile, domain+profile, oracle, and adversarial conditions, we find a consistent pattern: lightweight textual context produces little to no change in average word error rate (WER), even with oracle prompts, and adversarial prompts do not reliably degrade performance. We term this the context-utilization gap (CUG): current systems are nominally promptable yet underuse readily available side information. ProfASR-Bench provides a standardized context ladder, entity- and slice-aware reporting with confidence intervals, and a reproducible testbed for comparing fusion strategies across model families. Dataset: https://huggingface.co/datasets/prdeepakbabu/ProfASR-Bench Code: https://github.com/prdeepakbabu/ProfASR-Bench