Temporal Taxation Compounds Under Post-Training Compression of Whisper Models

AI summary

Overview

Research area: Fairness auditing of automatic speech recognition (ASR), specifically how post-training model compression changes demographic error burdens in the Whisper model family.

Technical level: Intermediate. Readers should be comfortable with ASR word error rate, quantization and pruning terminology, and basic statistical testing; no deep compression-theory background is required.

Scope: A single-family empirical study (eight Whisper-family checkpoints across three speech benchmarks) measuring how 50% Wanda pruning, INT8/INT4 NF4/INT4 HQQ quantization, and Distil-Whisper distillation redistribute word-error-rate burden across demographic groups, expressed as a "temporal taxation" time cost.

What This Paper Is About

ASR systems are normally audited for demographic fairness at full precision (FP32 or FP16), but the models actually shipped to edge devices have been quantized, pruned, or distilled. This paper asks whether that weight compression — not audio compression, not feature compression — redistributes error burden across demographic groups, converting a fairness gap into a gap in human correction labor. The goal is to measure whether benchmark-time fairness audits predict deployment-time fairness.

Key Contributions

1. First systematic study of post-training pruning and ASR fairness. The authors show that 50% Wanda pruning of whisper-large-v3 more than doubles the Black/AA-versus-Asian temporal-taxation differential on Fair-Speech (+111%), an effect invariant to the assumed per-error cost and robust to an audio-quality control, and only partly mitigated by beam-search decoding (which still leaves a +86% increase).
2. Extension of prior 8-bit fairness analysis to sub-8-bit precision on English dialect and accent. At edge model size, INT4 HQQ multiplies catastrophic-loop rates on West African accents by factors of five to seven, and the authors show WER compounding is capacity-dependent and recipe-specific.
3. Operationalization of the temporal-taxation construct of Choi and Choi (2025), originally a conceptual proposal, as a reporting convention for ASR compression audits. The authors explicitly state the metric is a linear rescaling of per-group WER and do not claim the arithmetic as a contribution; its value is a deployment-interpretable time unit plus a relative compounding measure exactly invariant to the cost-per-error assumption.
4. A counter-expectation result on distillation. Distillation does not compound demographic gaps but narrows them in 21 of 27 tested settings, with one structured exception concentrated on a single teacher-student pair.

Main Findings

• Pruning compounds the worst-versus-best gap at deployment scale. whisper-large-v3 with 50% Wanda unstructured pruning on Fair-Speech raises the Black/AA-versus-Asian temporal-taxation differential from 30.14 to 63.73 seconds of correction time per minute of speech at an assumed cost of C = 5 seconds per edit, a relative increase of 111.4%. The relative change is exactly invariant in C.
• The damage is concentrated on one group. Under whisper-large-v3 + Wanda, Black/AA WER rises by +7.89 percentage points (BH-adjusted p approximately 0, n = 7782) while Asian WER rises by only +0.43 points (p_BH = 0.006, n = 3853). Black/AA stands alone at +7.89 pp while Asian, Native American, Pacific Islander, and Middle Eastern speakers cluster below +1.5 pp.
• Pruning compounding is monotonic in model size, in the opposite direction from quantization. The Fair-Speech Wanda compounding index moves from -0.56 at whisper-tiny through -0.34, -0.02, +0.35 to +0.58 at whisper-large-v3. The authors note this is the opposite of the pattern reported in vision compression work.
• Nine of the top ten compounding cells are Wanda. Among them: whisper-medium Wanda on Fair-Speech Black/AA versus Asian (+88%), whisper-base Wanda on Common Voice 25 indian versus canada (+70%), whisper-large-v3 Wanda on Common Voice 25 indian versus canada (+69%), whisper-small Wanda on Fair-Speech Black/AA versus Asian (+52%), and whisper-small Wanda on Common Voice 25 african versus canada (+35.1%).
• The audio-quality confound does not explain the result. The ethnicity effect survives an SNR control in 34 of 34 (model, precision) Fair-Speech cells. After SNR adjustment, the demographic MMR is larger than the raw MMR in every cell, by 15.0% on average (median 13.9%) — the control widens the gap rather than narrowing it.
• Quantization compounding is capacity-bounded. At whisper-tiny on Common Voice 25, INT4 HQQ produces 7 of 7 BH-significant per-accent deltas, all positive, ranging from +5.45 pp (Canadian) to +13.19 pp (African). At whisper-large-v3 on Common Voice 25, the MMR shift is a smaller +0.216 and only British English reaches BH significance (+0.73 pp). Four mid-size Common Voice 25 quantization cells show narrowing, with compounding index between -0.13 and -0.06.
• 4-bit recipe choice is itself a fairness decision. At whisper-tiny, HQQ yields 7 of 7 BH-significant per-accent Common Voice 25 deltas against 5 of 7 for NF4, while NF4 flattens MMR more on that dataset (-0.117 against +0.010 for HQQ). At whisper-large-v3 the picture inverts on AfriSpeech (NF4 significant on 3 of 63 accents, HQQ on none) while on Common Voice 25 HQQ shifts MMR further than NF4 (+0.216 against +0.138).
• Catastrophic transcript loops compound at edge size. Under whisper-tiny + INT4 HQQ on AfriSpeech, seven of 62 accents become BH-significantly worse in paired permutation. Kanuri (n = 66) moves from 4.55% to 30.30% loop rate (6.67-fold, p_BH = 2.68 × 10⁻³), Hausa (n = 196) from 1.53% to 9.18% (6.0-fold, p_BH = 1.43 × 10⁻²), and Yoruba (n = 648) from 1.23% to 6.33% (5.12-fold, p_BH = 8.62 × 10⁻⁵). Fair-Speech replicates in the same direction: Black/AA loop rate rises from 0.51% to 1.28% (2.5-fold, n = 7807, p_BH = 2.46 × 10⁻⁶). At whisper-large-v3 + INT4 HQQ on AfriSpeech, no accent reaches BH significance.
• Model rankings survive compression even when absolute gaps do not. Kendall's tau between the FP16 ranking by MMR and compressed-precision rankings is 1.000 in 11 of 12 (dataset, precision) cells, with one adjacent-pair swap at Fair-Speech + INT4 HQQ (tau = 0.929).
• Beam search is not a fairness fix. In six of eight quantization ablation cells the beam-5 MMR moves by less than 0.2 from greedy; in whisper-large-v3 + INT4 NF4 on Common Voice 25, beam-5 amplifies the compounding index (+0.296 against +0.138). Loop rate increases under beam-5 in eight of eight cells. On the pruning arm, beam-5 lowers the compounding index from +0.584 to +0.468 and moves the headline from +111% to +86%, with Black/AA still carrying +5.63 pp against +7.89 pp under greedy.
• Distillation narrows gaps, contrary to the long-tail prediction. Over 27 (teacher, student, precision, dataset) cells, distillation narrows the gap in 21, widens it in 4, and shows near-zero shifts in 2. The widening is concentrated on the whisper-medium to distil-medium.en pair on Fair-Speech ethnicity at every precision (gap-widening index +0.011 to +0.151). On AfriSpeech under INT4 HQQ, 57 of 63 accents narrow with BH significance. This is a relative-parity result: aggregate WER rises uniformly across all 27 cells, with relative student-versus-teacher deltas from +0.07% to +31.4%.
• INT8 is close to, but not exactly, fairness-neutral. Across all three datasets, INT8 produces 7 significant per-group deltas out of 427. Four of the seven fall on AfriSpeech accents with n ≤ 101 (bini +3.35 pp, khana +3.05 pp, etche +2.64 pp, mwaghavul +1.80 pp); on the two larger benchmarks the significant deltas are at most +0.40 pp and +0.22 pp.

Methodology in Plain English

The researchers took eight Whisper-family models — five Whisper backbones (whisper-tiny, whisper-base, whisper-small, whisper-medium, whisper-large-v3, spanning 39M to 1.55B parameters) and three Distil-Whisper students paired with their teachers (distil-small.en, distil-medium.en, distil-large-v3) — and applied three families of post-training weight compression to each. Restricting to one family is deliberate: it prevents architecture, tokenizer, training data, and audio encoder from confounding the compression effect. Whisper-large-v3-Turbo was excluded because it is itself a layer-pruned version of large-v3.

Compression covered four precisions per model: FP16 as the deployment-realistic reference, INT8, INT4 NF4, and INT4 HQQ. NF4 and HQQ are both data-free, so including both isolates the optimization-recipe axis while holding bit-width and calibration-data status constant. Pruning used 50% Wanda unstructured pruning extended to the linear layers of Whisper's encoder, decoder, and decoder cross-attention, with calibration on 128 utterances from LibriSpeech train-clean-100 held out from every test set. Distil-Whisper checkpoints were paired with their teachers and evaluated at each precision.

Three benchmarks carry the fairness claims: Fair-Speech (26,471 utterances, 593 speakers, with ethnicity, age, gender, geography, and native-English axes), Common Voice 25 (16,398 English test clips with accent labels for seven groups at n ≥ 50), and AfriSpeech-200 (6,318 test utterances). LibriSpeech test-clean was held in reserve as an aggregate-WER reference only.

Per-utterance metrics were WER, deletion rate, insertion rate, and a loop flag set when a hypothesis contains more than five 5-gram repetitions or when the hypothesis-to-reference length ratio exceeds three, with thresholds fixed before the full sweep. Per-cell fairness is summarized by the max-min ratio (MMR) and a compounding index relative to FP16. Statistical testing used paired permutation tests with B = 2,000 permutations per cell, two-sided, adjusted within each cell with the Benjamini-Hochberg procedure, and percentile bootstrap confidence intervals with B = 200. The headline metric is the temporal-taxation differential T_g = WER_g × C × wpm, anchored at C = 5 seconds per edit with sensitivity over C in {2, 5, 8}. Because T_g is linear in C, the relative change under compression cancels C exactly.

Why This Matters

Impact on research. Prior ASR fairness work treats the model as fixed, auditing which model fails on whom at FP32 or FP16. This paper shows that failure profile is not preserved through deployment, and that the direction of redistribution is not predictable from the compression literature: pruning compounds with increasing capacity (opposite to the vision pattern), distillation narrows rather than compounds, and NF4 and HQQ at the same bit-width redistribute error to different groups. It also supplies an operational metric that separates the cost-dependent part of a fairness claim from the cost-independent part.

Real-world applications:

• Edge transcription, captioning, and dictation workflows, where recognition errors are surfaced to human reviewers and a WER gap becomes a gap in repair labor.
• Model selection for on-device ASR: choosing between 50% Wanda pruning and INT4 HQQ to fit whisper-large-v3 on a mobile device is a fairness decision — Wanda more than doubles the Black/AA-versus-Asian Fair-Speech differential, while sub-8-bit quantization at the same model size shifts MMR by less than 0.25 and reaches BH significance on only one accent group (England, +0.73 pp).
• Regulatory and procurement audits of ASR systems, which the authors argue should evaluate at least one sub-8-bit precision and one pruned condition rather than a single full-precision snapshot.
• Deployment to West African accent populations, where INT4 HQQ at edge scale multiplies catastrophic-loop rates by factors of five to seven, forcing disproportionate correction effort on

Authors’ abstract

Automatic speech recognition models are audited for demographic fairness at full precision, yet the models that ship to production have been quantized, pruned, and distilled. We ask whether post-training weight compression, which alters model weights rather than the audio signal or its feature representation, redistributes error burden across demographic groups. Across the Whisper family on Fair-Speech, Common Voice 25, and AfriSpeech-200, 50% Wanda pruning of Whisper-large-v3 sharply widens the Black/AA-vs-Asian temporal-taxation differential on Fair-Speech: the absolute word-error-rate gap between the worst- and best-served groups more than doubles; at an assumed cost of five seconds of correction effort per transcription error this is a rise from 30 to 64 seconds of correction time per minute of speech. This +111% relative increase is invariant to the assumed per-error cost, survives an audio-quality control, and is only partly mitigated by beam-search decoding, which still leaves an +86% increase. At edge model size, INT4 HQQ quantization compounds catastrophic transcript loops on West African accents by factors of five to seven. Distillation, by contrast, narrows demographic gaps in 21 of 27 evaluated settings (teacher-student pair, precision, and dataset), with the exceptions concentrated on a single model pair. We cast the temporal-taxation construct of Choi and Choi (2025) as a quantitative metric, and show that single-snapshot fairness audits on full-precision models do not capture the deployment-time burden that compression places on already-marginalized speakers.