Accent-Invariant Automatic Speech Recognition via Saliency-Driven Spectrogram Masking

AI summary

Overview

Research area: Automatic speech recognition (ASR), specifically accent robustness, data augmentation, and multilingual speech processing.

Technical level: Intermediate. The paper assumes familiarity with spectrograms, convolutional neural networks, Grad-CAM saliency maps, and word/character error rate metrics, though the core idea is explained conceptually.

One-sentence scope: The paper proposes an accent-invariant ASR pipeline that uses a spectrogram-based accent classifier and Grad-CAM to locate and mask accent-specific regions, then fine-tunes Whisper on the augmented data, and it introduces PDID, a newly collected 10-accent Persian benchmark.

What This Paper Is About

Transformer-based ASR models such as Whisper perform well on standard speech but produce elevated word error rates when speakers have regional or non-native accents. The goal of this work is to reduce that sensitivity without redesigning the model architecture or collecting large amounts of new accent-labeled speech, by identifying which spectrogram regions drive accent classification and suppressing those regions during training. A secondary goal is to fill a gap in Persian speech research by releasing the first systematic multi-accent benchmark for Persian ASR.

Key Contributions

1. A saliency-driven spectrogram masking framework that uses Grad-CAM on an accent classifier to identify accent-sensitive spectrogram regions and suppress them, so the ASR model is pushed toward accent-neutral linguistic features.
2. A lightweight, model-agnostic training strategy that improves robustness to both known and unseen accents without architectural modifications or full model retraining.
3. PDID (Persian Dialect Identification), a newly collected multi-accent corpus covering 10 regional Persian accents, presented as the first systematic benchmark for Persian accent robustness.
4. Extensive experiments on English (LibriSpeech, EdAcc, CommonAccent) and Persian (CommonVoice-fa, PDID) showing consistent WER/CER reductions over SpecAugment baselines on accented speech.

Main Findings

• English accented speech improves over SpecAugment: On the accented column, Whisper_tiny drops from 42.0/37.7 (WER/CER) to 23.4/15.1 with the proposed method, versus 27.0/17.8 for the SpecAugment baseline.
• English unseen accents improve most: On unseen accents, Whisper_tiny goes from 34.7/26.7 to 26.7/17.9, and WhisperLS_tiny from 29.3/19.4 to 26.7/17.9, while the SpecAugment baseline reaches 30.1/20.3.
• CommonAccent also improves: Whisper_tiny moves from 62.2/38.5 to 34.8/18.2, and WhisperLS_tiny from 36.3/18.5 to 34.8/18.2.
• Standard English is not degraded: On LibriSpeech, the proposed method reaches 6.8/2.7 with Whisper_tiny, slightly better than Whisper_tiny's 8.0/3.2 and SpecAugment's 7.3/2.9.
• Persian accented results improve with both base and medium models: On accented Persian, the base model reaches 88.8/40.6 versus 92.8/51.6 for SpecAugment and 97.2/61.7 for the CommonVoice-fine-tuned base model; the medium model reaches 67.5/36.5 versus 70.4/42.5 for the CommonVoice-fine-tuned medium model.
• Grad-CAM++ variant is competitive: The Grad-CAM++ variant (ARWsPr++_b) reaches 62.4/22.1 on standard Persian and 90.3/41.5 on accented Persian, slightly behind the paper's own base configuration on accented speech.
• Classifier accuracy: The spectrogram CNN accent classifier reaches 74.6% accuracy on English accents and 95% on the Persian accented dataset with the same settings.
• Reported relative gain: The conclusion states relative WER/CER gains up to 14% on accented speech compared to SpecAugment baselines.
• Persian baseline instability: The unfine-tuned Whisper base model reports very high Persian figures (186.4/209.4 on standard, 128.6/93.6 on accented), which fine-tuning on CommonVoice substantially reduces.

Methodology in Plain English

The pipeline has three stages. First, a convolutional neural network is trained to classify the accent of an utterance from its spectrogram, using normalized inputs of size 80 × 3000 (80 frequency bins by 3000 time frames) and four convolutional layers with 32, 64, 128 and 256 filters of size 3 × 3, followed by a 128-neuron fully connected layer. English training data mixes the Edinburgh dataset (Southern British, Irish, Egyptian, Italian) with LibriSpeech for Standard English, totaling 3790 samples; Persian data comes from the newly collected PDID corpus.

Second, Grad-CAM is run on that classifier for each spectrogram. The gradients of the predicted accent class with respect to the last convolutional layer's feature maps are globally average-pooled into weights, combined with the feature maps, and turned into a heatmap that is normalized to the range [0, 1], giving a score C(i, j) per pixel. A probabilistic mask then decides what to erase: pixels with C(i, j) ≥ 0.7 are always masked; pixels with 0.5 ≤ C(i, j) < 0.7 are masked with probability drawn uniformly from U(0.7, 0.9); pixels with C(i, j) < 0.5 are masked with probability drawn from U(0, 0.05); and pixels where the binary threshold mask (C(i, j) > 0.3) is zero are never masked. The low-probability case exists to catch accent-related regions the classifier may have scored incorrectly.

Third, the mask is multiplied element-wise with the original spectrogram. The masked spectrograms are combined with the primary dataset to form an augmented training set, and a pre-trained transformer ASR model (Whisper) is fine-tuned on it. Training used NVIDIA RTX 3090 GPUs with the AdamW optimizer, learning rates of 1 × 10⁻⁵ for tiny and 3 × 10⁻⁶ for base/medium models, batch sizes of 32, 16 and 4 respectively, and 10 epochs. Evaluation used WER and CER.

For the PDID corpus, audio was collected from local TV and radio plus online platforms such as Aparat and YouTube for 10 regional accents (Isfahani, Yazdi, Lori, Kurdish, Balochi, Southern, Northern, Tajiki, Mashhadi, Shirazi). Preprocessing followed a pipeline similar to EMILIA: voice activity detection, speaker diarization, silence-based segmentation, and speech–music separation. Audio was standardized to 16 kHz, mono-channel, 16-bit WAV with normalized loudness and cut into 3–30 second clips. Roughly 23 hours of clean accent-labeled data remained from more than 200 hours of raw speech, with Tajiki, Shirazi and Balochi held out for the test set. The training split contains 12613 samples across seven accents, the largest being Shomali at 7632 samples (~10.9 h) and the smallest Kurdish at 125 samples (~1.0 h).

Why This Matters

Research impact: The work reframes accent robustness as a targeted data-augmentation problem rather than a model-architecture problem, and it shows that gradient-based saliency can steer which spectrogram regions an ASR model learns from. It also contributes a new Persian benchmark, which the authors describe as filling a gap for low-resource, linguistically diverse languages.

Real-world applications:

• Voice interfaces and virtual assistants used by speakers with regional or non-native accents, where elevated error rates currently limit usability.
• Healthcare transcription and other clinical documentation, given the paper's stated link between ASR and accessibility and efficiency in healthcare.
• Call centers and customer-service transcription in linguistically diverse markets such as Iran or multilingual regions.
• Transcription and accessibility tools for media, since the PDID corpus itself was built from broadcast and online video sources.

Industry relevance: The method is model-agnostic and does not require architectural changes or full retraining, which matters for teams that want to adapt an existing production ASR model to accented traffic at low compute cost. The paper notes that LLM-based robustness approaches greatly increase computational and memory costs and degrade in low-resource languages, positioning this masking approach as a cheaper alternative. Pretrained Whisper checkpoints, fine-tuning, and accent classification are all standard, widely available components.

Future Directions

• Improving the accent classifier, whose accuracy is 74.6% on English accents; better saliency maps could sharpen the masking policy.
• Extending PDID beyond its current scope, for example adding more speakers and hours for the accents that are currently represented by small splits (Kurdish at 125 samples and Jonubi at 147 samples) or by held-out test accents.
• Testing the framework on ASR architectures beyond Whisper and on more languages, given the claim of being model-agnostic.
• Comparing against parameter-efficient accent adaptation methods such as MAS-LoRA and fusion-based approaches like Qifusion-Net under matched conditions, and standardizing how much of the gain comes from masking versus fine-tuning on the new corpus.
• The paper does not report per-accent breakdowns beyond the aggregate standard and accented columns, nor does it report results for Tajiki, Shirazi and Balochi individually, which would clarify generalization to held-out accents.

Target Audience

Speech and NLP researchers working on ASR robustness, accent and dialect adaptation, and data augmentation. Also relevant to engineers deploying multilingual or Persian-language ASR systems, to teams building speech datasets for low-resource languages, and to readers interested in applying saliency methods like Grad-CAM to audio rather than images. Beginners can follow the conceptual pipeline but will need background on spectrograms, CNNs, and error-rate metrics to interpret the results tables.

Authors’ abstract

Pre-trained transformer-based models have significantly advanced automatic speech recognition (ASR), yet they remain sensitive to accent and dialectal variations, resulting in elevated word error rates (WER) in linguistically diverse languages such as English and Persian. To address this challenge, we propose an accent-invariant ASR framework that integrates accent and dialect classification into the recognition pipeline. Our approach involves training a spectrogram-based classifier to capture accent-specific cues, masking the regions most influential to its predictions, and using the masked spectrograms for data augmentation. This enhances the robustness of ASR models against accent variability. We evaluate the method using both English and Persian speech. For Persian, we introduce a newly collected dataset spanning multiple regional accents, establishing the first systematic benchmark for accent variation in Persian ASR that fills a critical gap in multilingual speech research and provides a foundation for future studies on low-resource, linguistically diverse languages. Experimental results with the Whisper model demonstrate that our masking and augmentation strategy yields substantial WER reductions in both English and Persian settings, confirming the effectiveness of the approach. This research advances the development of multilingual ASR systems that are resilient to accent and dialect diversity. Code and dataset are publicly available at: https://github.com/MH-Sameti/Accent_invariant_ASR