ADI-20: Arabic Dialect Identification dataset and models

AI summary

Overview

• Research area: Speech processing and natural language processing, specifically Arabic Dialect Identification (ADI) from speech, with a focus on datasets and classification models.
• Technical level: Intermediate. The paper assumes familiarity with self-supervised speech representations, speaker-embedding architectures, and weighted F1 evaluation, but it explains its datasets and experimental setup in accessible terms.
• Scope: The paper introduces ADI-20, a 3,556.76-hour Arabic speech dataset covering 19 dialects plus Modern Standard Arabic (MSA), and benchmarks ECAPA-TDNN and Whisper-based systems on it, studying how training data quantity and model size affect dialect identification.

What This Paper Is About

Identifying which Arabic dialect a person is speaking is difficult because Arabic varieties differ not only in accent and vocabulary but also in grammar, and speakers frequently code-switch between local dialects, MSA, and foreign languages. The previous ADI-17 dataset left out MSA and some dialects entirely (Tunisian and Bahraini), and its training data was heavily imbalanced, with 31 times more Iraqi data than Jordanian. This paper builds ADI-20 to broaden that coverage and systematically tests how much training data and how many model parameters are actually needed for strong dialect identification.

Key Contributions

1. The ADI-20 dataset. An extension of ADI-17 covering all Arabic-speaking countries' dialects, comprising 3,556.76 hours of MSA and dialectal Arabic speech across 19 dialects plus MSA (the paper also refers to covering 20 Arabic dialects), with at least 53 hours of training data per dialect. It adds Tunisian (the full TunSwitch dataset including its code-switched subset, 142.7 hours), Bahraini (YouTube content, approximately 271 hours for training and 2 hours each for validation and testing), and MSA (68 hours, built from ADI-5 audio plus scraped YouTube broadcast news, using the ADI-5 MSA dev and test sets for evaluation). It also expands the two most underrepresented ADI-17 dialects, Sudanese and Jordanian, up to the 53-hour level of Yemeni.

2. A controlled study of training data quantity. Four size-based variations of ADI-17 were created — ADI-17-10h (10 hours per dialect, 5.6% of the full ADI-17 train set), ADI-17-25h (25 hours per dialect, 14%), ADI-17-53h (53 hours per dialect, about 30%), and ADI-17-full — plus ADI-20-53h (ADI-17-53h extended with 53 additional hours of MSA, Bahraini, and Tunisian speech) and ADI-20-full.

3. A controlled study of model complexity. Whisper models of four sizes were compared: Whisper-base (20.6 million parameters), Whisper-small (88.2 million), Whisper-medium (307.2 million), and Whisper-large (637 million), alongside an ECAPA-TDNN system with 20.8 million parameters.

4. Open-source release of data, models, and recipes. Pre-trained models, dataset manifests, and complete recipes for the ADI-17 and ADI-20 systems are released at github.com/elyadata/ADI-20. The authors state that, to their knowledge, this is the first publicly available ADI model.

Main Findings

• ECAPA-TDNN beats Whisper at equal size. With comparable parameter counts (20.8 million for ECAPA-TDNN, 20.6 million for Whisper-base), ECAPA-TDNN reached a weighted F1-score of 93.16% on the ADI-17 test sets versus 91.34% for Whisper-base. The authors attribute this to ECAPA-TDNN being pre-trained on a speaker recognition task, which is somewhat similar to dialect identification.

• Larger models perform better. On the full ADI-17 training set, Whisper-large reached 95.66% F1, compared with 93.80% for Whisper-small and 95.46% for Whisper-medium.

• A small amount of data goes a long way. Using only 30% of the original ADI-17 training data (ADI-17-53h) caused only a small decrease in F1. Whisper-small actually improved, reaching 94.42% with ADI-17-53h versus 93.80% with the full ADI-17 set.

• Cutting data to 10 hours hurts substantially. For Whisper-large, ADI-17-10h yielded 92.96% F1 versus 95.66% with ADI-17-full, a drop that persisted across model sizes.

• Mid-size models can match large ones at reduced data. Whisper-medium reached 95.29% F1 trained on ADI-17-53h, close to Whisper-large's 95.66% on the full dataset, despite having half the parameters and using only 30% of the training data.

• ADI-20 data helps the largest model. Training Whisper-large on ADI-20-53h gave 95.41% F1 on the ADI-17 test set, slightly above the 95.16% obtained with ADI-17-53h. This gain appeared only with Whisper-large; Whisper-small and Whisper-medium trained on ADI-20-53h did not outperform their ADI-17-53h counterparts.

• Freezing and augmentation gave the best overall ADI-20 system. The strongest configuration was Whisper-large with the first half of its encoder layers frozen plus data augmentation as implemented in the SpeechBrain toolkit.

• ECAPA-TDNN on ADI-20-53h. Trained on ADI-20-53h, ECAPA-TDNN produced F1-scores of 92.89% and 90.49% on the ADI-17 and ADI-20 test sets respectively.

• Zero-shot evaluation on Casablanca. In a zero-shot setting on the Casablanca dataset, Whisper-large trained on ADI-20-53h with layer freezing and augmentation reached 62.74% F1, far above the 39.24% reported for the HuBERT-17 system in prior work evaluated the same way. Whisper-medium and Whisper-large trained on ADI-17-full reached 53.84% and 58.89% respectively, both exceeding the 92.12% HuBERT-17 number reported on ADI-17 itself while clearly outperforming HuBERT-17's 39.24% on Casablanca. The authors hypothesize that augmentation improves robustness to the unseen TV-drama conditions of Casablanca, which differ from their YouTube-sourced training data.

• Errors cluster between close dialects. Frequent confusions include Jordanian misclassified as Egyptian or Syrian (and as Lebanese, Palestinian, or Syrian in the detailed error analysis), Bahraini confused with Emirati and Qatari, and confusion among the Maghrebi dialects Algerian, Moroccan, and Libyan. MSA was most often confused with North African varieties (Algerian, Libyan, Egyptian).

Methodology in Plain English

The authors start from the existing ADI-17 corpus of 3,033 hours of YouTube programs split into 17 dialect categories, with roughly 2 hours per dialect for the test and validation splits. They note its imbalance — Iraqi, Egyptian, and Mauritanian (Hassaniya) together account for over half the training set — and its missing varieties, and then fill the gaps with new audio. Tunisian comes from TunSwitch, Bahraini entirely from YouTube, and MSA from ADI-5 files plus scraped broadcast news. To make the data study fair, they resample ADI-17 down to fixed budgets of 10, 25, and 53 hours per dialect using the same random seed, stratifying by speech duration rather than segment count. All segments longer than 30 seconds are split into segments of 30 seconds or shorter, and segments shorter than 3 seconds are discarded across all training sets; scraped content is segmented using YouTube caption timestamps.

Two model families are compared. The first is ECAPA-TDNN, a time-delay neural network with residual connections, squeeze-and-excitation modules, channel-dependent attentive statistical pooling, and Res2Blocks, implemented via the SpeechBrain CommonLanguage recipe and initialized from a VoxCeleb speaker-recognition checkpoint, with 80-dimensional Mel filterbanks replacing the original MFCC features. The second keeps only the encoder of Whisper, a weakly supervised encoder-decoder transformer trained on 680,000 hours of internet audio, and replaces the decoder with attention pooling followed by a classification dense layer. This design makes it easy to vary model size, since Whisper ships in tiny, base, small, medium, and large variants. All models were trained on 4 GPUs with a batch size of 16. Three experimental setups were run: ECAPA-TDNN versus Whisper-base at matched size, data quantity versus model complexity across the ADI-17 subsets, and the final ADI-20 systems with augmentation and layer-freezing experiments.

Why This Matters

• Research impact. The paper provides a larger, more balanced, and publicly released benchmark for Arabic dialect identification, including MSA for the first time in this line of work, and offers controlled evidence on the data-versus-model-size trade-off that other low-resource speech tasks can reuse. Its zero-shot result on Casablanca shows that the released models transfer outside their training domain far better than the previously reported HuBERT-17 baseline.

• Real-world applications:

  • Automatic speech recognition: routing audio to a dialect-specific ASR system, since the paper notes ADI supports ASR, machine translation, sentiment analysis, text-to-speech, and text normalization.
  • Media and broadcast monitoring: the original ADI-5 corpus was built from Al Jazeera TV, and dialect ID can label or route large volumes of Arabic broadcast and YouTube content.
  • Customer service and call analytics: identifying a caller's dialect to direct them to appropriate support or to adapt downstream language models.
  • Content moderation and search: tagging dialectal Arabic content, which differs substantially from MSA in vocabulary and grammar.

• Industry relevance. The finding that competitive models can be trained with as little as 53 hours per dialect, and that a mid-sized model can rival a large one at 30% of the data, lowers the cost of building production dialect identification. The public release of models and recipes makes adoption and reproduction straightforward.

Future Directions

• Deeper exploration of ECAPA-TDNN. The authors state it showed promising results but was not explored in depth due to time constraints and the availability of Whisper models in various sizes.
• City-level dialect identification. Extending beyond country-level labels to capture finer variation within countries, which the introduction describes as shaped by historical, social, and geographical factors.
• Coverage of remaining dialects. Data for dialects from countries such as Somalia and Eritrea remained challenging to obtain and is explicitly left for future exploration.
• Domain robustness. The Casablanca zero-shot results, where the best model reached 62.74% F1, suggest that generalization to unseen domains such as TV dramas remains an open problem.

Target Audience

Researchers and engineers working on speech processing for Arabic and other low-resource language varieties, particularly those building dialect identification, dialect-aware ASR, or speaker-embedding-based classification systems. It is also useful for practitioners who need a ready-made benchmark and pre-trained ADI models, and for corpus builders interested in how to rebalance and extend an imbalanced speech dataset.

Authors’ abstract

We present ADI-20, an extension of the previously published ADI-17 Arabic Dialect Identification (ADI) dataset. ADI-20 covers all Arabic-speaking countries' dialects. It comprises 3,556 hours from 19 Arabic dialects in addition to Modern Standard Arabic (MSA). We used this dataset to train and evaluate various state-of-the-art ADI systems. We explored fine-tuning pre-trained ECAPA-TDNN-based models, as well as Whisper encoder blocks coupled with an attention pooling layer and a classification dense layer. We investigated the effect of (i) training data size and (ii) the model's number of parameters on identification performance. Our results show a small decrease in F1 score while using only 30% of the original training data. We open-source our collected data and trained models to enable the reproduction of our work, as well as support further research in ADI.